Improved Q Network Auto-Scaling in Microservice Architecture

Abstract

Microservice architecture has emerged as a powerful paradigm for cloud computing due to its high efficiency in infrastructure management as well as its capability of largescale user service. A cloud provider requires flexible resource management to meet the continually changing demands, such as auto-scaling and provisioning. A common approach used in both commercial and open-source computing platforms is workload-based automatic scaling, which expands instances by increasing the number of incoming requests. Concurrency is a request-based policy that has recently been proposed in the evolving microservice framework; in this policy, the algorithm can expand its resources to the maximum number of configured requests to be processed in parallel per instance. However, it has proven difficult to identify the concurrency configuration that provides the best possible service quality, as various factors can affect the throughput and latency based on the workloads and complexity of the infrastructure characteristics. Therefore, this study aimed to investigate the applicability of an artificial intelligence approach to request-based auto-scaling in the microservice framework. Our results showed that the proposed model could learn an effective expansion policy within a limited number of pods, thereby showing an improved performance over the underlying auto expansion configuration.

1. Introduction

With the advancement and spread of Virtual Machine and container technology, the adoption of microservice computing models has increased substantially [1]. Microservice computing provides two main benefits to users: First, it is easy to scale when in use, as it is not difficult to develop idle VMs or containers. Second, because it is delegated to a cloud provider, there is no overhead related to infrastructure maintenance for users. Microservice also brings other advantages such as flexibility as well as the capability of immediate scalability to automatically add or remove resources depending on the incoming load. Some microservice frameworks use a resource-based Kubernetes Horizontal Pod Autoscaler (HPA) to drive expansion through CPU or memory utilization thresholds per instance. As a result, the Auto-Scaling function depends on each system component delivering fast and accurate calculations [2]. Commercially available microservice platforms are often characterized by workload-based expansion, where they provide additional resources as the incoming traffic increases. For example, AWS lambda initializes for each new request that comes in until the limit is reached [3]. Continually creating new requests in this way leads to a speed delay problem, because it is processed using the HPA method. To minimize this problem, one may use an open-source framework that supports parallel processing up to a predefined number of simultaneous requests per instance [4]. When so-called concurrency is reached, the Knative Pod Autoscaler (KPA) deploys additional pods to handle the extra loads. Simultaneous parameters can also be manually adjusted to use resources more efficiently and adjust automatic scaling systems to best suit individual workloads.

Existing research shows that the use of different workloads can affect microservice performance and cause latency differences up to a few seconds. Since this can have a significant impact on user experience, we proposed a reinforcement learning RL-based model that dynamically determines the optimal concurrency for individual workloads. In general, RL operates under the idea that agents can learn effective decision-making policies by experiencing a series of trial-and-error interactions with the environment, then evaluating the current state of the system dynamics in each iteration and determining the operation of opening the CPU core. After learning, the agent receives positive or negative rewards and consequently learns the advantages of each reward-status combination. Since reinforcement learning can adapt to changes in runtime without requiring prior knowledge of the incoming workloads, the RL algorithm has been adopted as a valid method in the field of VM auto expansion technology. Therefore, we evaluated the applicability of the established RL algorithm Q-learning to determine the level of concurrency with optimized performance. Specifically, we implemented a cloud-based framework and performed three consecutive experiments.

We analyzed the performance changes of various workload profiles in Auto-Scaling configurations. The results demonstrated the dependence of throughput as well as delay time during overload work; they also showed the possibility of improvement through adaptive expansion setting. Informed by these results, we reinforced the framework with intelligent RL-based logic to evaluate the ability of self-learning algorithms to make effective decisions in the microservice framework. The results showed that the proposed model can learn appropriate expansion policies within a limited time without prior knowledge of incoming workloads, thereby showing improved performance over basic automatic CPU expansion settings of the framework. Lastly, we proved that the proposed RL algorithm achieves better performance than all considered alternatives.

The rest of this work is organized as follows: Section 2 introduces the microservice platform, Q-learning, and the new Q-Learning theory, and this section reviews related work into both the intelligent microservice framework and cloud-based Auto-Scaling technology (Table 1). Section 3 proposes a Q-learning model to utilize these findings to improve the auto-scaling functionality of HPA as well as the general performance. Section 4 compares the performance of the existing HPA with the HPA applying the query algorithm and the performance of the HPA applying the proposed Quad Q Network. Finally, Section 5 provides the conclusions and limitations of this study, along with possible future research directions.

Table 1. A Overview of Cloud computing & Reinforcement Learning Discussed in Section 2.

Name	Reference
Microservice Computing	Auto-Scaling Function (The Framework without An Open-Source Server) [5,6,7] User Control for Limited Custom Service Quality Requirements [8] Intelligent Automatic Expansion Kubernetes HPA [9]
Auto-Scaling	Auto-scaling of Cloud Resources [10] Flexible and Skillful Resource Allocation, The Core Technology of Cloud Computing [11,12] Development of Appropriate Threshold-Based Rules [13]
	Threshold-based	Investigate the automatic expansion system [14] Elastic Docker [15] Horizontal automatic scaling [16]
	Reinforcement Learning-Based	Adaptive Fuzzy Logic Controller [17] Threshold-based Solution for Automatic Expansion of Horizontal Containers [18] Reinforce Learning Solution (i.e., Q Learning, Dyna-Q and Model-Based) [19] Dynamic Number of Servers Using A Threshold-Based Expansion Policy [20]
Cloud Computing	Various Middleware Frameworks of Containers [21] Programming Sensors and IoT Devices (Small VMs Supported by Python Runtime Environments such as Spring and NodeJS) [22,23,24,25] Application Efficiency (Resource Virtualization Technology for Hardware Flexibility) [26] Lightweight Virtualization Solution [27] Virtualization Application to Lower Overhead [28] Definition of Container [29] Strengths of Containers, Lightening Up [30] The Connection between Lightweight Virtualization and Microservice [31] Strategies to Strengthen The Fuzzy Distribution of Microphone Services [32] Cluster Intelligence-Based Strategies [33]
Reinforcement Learning	Deep Q Network That Performs Learning by Updating the Q Value [34,35,36] Deep Q Network Proceeds with Effective Learning by Resetting the Q Value [37,38,39,40] Double Deep Q Network’s Idea [41,42] Double Deep Q Network’s Improvements [43,44]

2. Literature Review: Related Work

2.1. Microservice Computing

As microservice computing continues to spread, scalability represents one of the key elements being researched, and there is high interest in providing various solutions and comparing their performance [4]. We benchmarked a variety of microservice platforms, including extensions focused on Amazon Lambda, Microsoft Azure Functions, Google Cloud Functions, and IBM Cloud Functions [5,6,7], and found that the open source serverless framework paid more attention to auto-scaling functions. Another comparison of qualitative and quantitative features of Kubless, OpenFaas, Apache Openwhisk, and Knative came to the same conclusion, although user control of custom service quality requirements was generally limited [8]. One study proposed an intelligent automatic expansion Kubernetes HPA to solve the computing resource management problems listed above [9]. This study used the Kubernetes concurrency level adjustment function to compare the performance of various workload scenarios and investigate the applicability of auto-scaling functions.

2.2. Auto-Scaling

The auto-scaling of cloud resources has emerged as a subject of intensive research in recent years due to the fact that customized resource allocation is one of the key characteristics of increased adaptation to cloud computing [10]. Other taxonomies have been proposed to classify numerous techniques at the algorithm level, with Threshold-based Rules, Queuing Theory, and RL being the main categories. Various studies have also been conducted with a focus on public cloud providers such as Amazon ECS using Auto-Scaling [9]. Even with simplified implementations, developing appropriate threshold-based rules requires expert knowledge and precise application understanding [9,10,11,12,13].

In practice, we use the Kubernetes scaling engine, which makes appropriate automatic scaling decisions to handle the actual variations in incoming requests. It can also be set up for resource over-provisioning by introducing certain management parameters for cloud application providers. Providing adequate application expansion represents one of the most important elements for cloud providers, as flexible and skillful resource distribution is a key concept of cloud computing [11,12]. Auto-scaling has been classified and studied using basic theoretical models or methods.

Threshold-based: Studies focusing on threshold-based expansion rules have improved vertical and horizontal elasticity performance in cloud systems of lightweight virtualization technology [14,15,16]. Specifically, one study examined a resource utilization-based automatic expansion system that demonstrates Kubernetes’ VPA through its ability to dynamically adjust container allocation in the Kubernetes cluster without interruption [14]. Further, based on IBM’s principle of autonomous computing MAPE-K, an Elastic Docker study was conducted that autonomously supports the vertical elasticity of docker containers [15]. The above two papers integrated container movement and reviewed the possibility of vertical auto-scaling. Another study aimed to improve horizontal auto-scaling [16].

Reinforcement learning-based: One study combined two reinforcement learning (RL) approaches—Q-learning and state-compensation-state-behavior (SARSA) algorithms—with self-adaptive fuzzy logic controllers that lead to dynamic resource allocation to the virtual machine (VM) [17]. Another work provided a threshold-based solution for the automatic scaling of horizontal containers that uses Q-learning to adjust scaling thresholds [18]. In a different study, a group of dockers proposed different RL solutions (i.e., Q learning, Dyna-Q, and model-based) that utilize various levels of knowledge about system dynamics [19,20]. They conducted simulations and compared the behavior of their models with those of typical M/M/c queues.

RL provides an interesting approach to agent learning about the most suitable extension measures without prior knowledge [7]. In recent years, various studies have investigated the applicability of modeless RL algorithms such as Q-learning [10]. With the advent of container-based applications, this field has attracted greater attention. However, few studies have focused on areas that can mitigate the general auto-expansion problem of VM or container configurations. Therefore, the objective of the present study was to investigate the applicability of newly introduced Q learning to request-based automatic expansion in a server-free environment. Unlike previous studies on direct vertical or horizontal scaling using RL [13], we proposed a model that could learn effective scaling policies by adapting simultaneous request levels per container instance to specific workloads.

2.3. Container Computing

Over the past few years, various middleware frameworks of containers have been proposed in attempts to reduce the difference between the modern high computing needs with simultaneous interest in multiple applications and the hardware capabilities by IoT node/end user [21]. For example, small VMs supported in Spring and NodeJS, Python runtime environments, are designed to allow for programming and code mobility of sensors and other IoT devices [22,23,24,25]. The efficiency of applications depends heavily on resource virtualization technologies that limit hardware flexibility and depend on code [26]. As a result, there has recently been increased interest in lightweight virtualization solutions such as docker containers [27]. Although virtualized applications/services can be implemented more effectively with respect to hypervisor or VM-based virtualization technology, there is a need for lower overhead [28].

Containers can be defined as a collection of processes that are separated from the rest of the system encapsulating the associated dependencies [29]. Containers do not require a complete guest operating system (OS), so they are much lighter than VMs [30]. For example, a container can boot faster than a VM, in just a few seconds, and involves a resource set of less than 2GB, thus allowing it to scale to meet requirements as necessary [27].

Lightweight virtualization is highly related to microservice networks [31]. This is because it allows for the fast creation and booting of virtualized instances, the possibility of having many applications simultaneously running on the same host, and separation between instances running on the same host while reducing overhead costs. Further, container-based services do not imply strong dependence on a given platform, programming language, or specific application domain, so they have the flexibility to be developed once and deployed everywhere.

Research related to the implementation of smart containers, dockers, Apache, and Kubernetes is currently being actively conducted. For example, studies have proposed fuzzy enhancement learning strategies for microservice allocation [32]. Research has also been conducted to suggest cluster intelligence-based strategies that can contribute to scheduling in big data applications [33]. In both cases, the proposals are limited to the scheduling of smart containers in cloud computing, and no proposals have been presented for the scheduling of smart containers in intelligent auto-scaling. Therefore, previous studies have not been able to propose an essential solution for building an intelligent microphone service.

2.4. Basic Microservice Platform

The open source Microservice platform provides a set of Kubernetes-based middleware components that can support the deployment and service of serverless applications, including the ability to automatically scale resources as necessary [6]. When the service revision expands, the receiving gateway is structured to first forward the receiving request to the activator. The activator then reports the information to the auto regulator, which then instructs the distribution of the revised version to be expanded appropriately.

The fact that the request is buffered until the user pod of the revision is available might have negative effects in terms of waiting time, as the request is blocked during that time. By contrast, if one or more replicas remain active, the active program can be ignored, and the traffic can flow directly to the user’s pod to increase the efficiency.

When the request reaches the pod, it is channeled by the queue proxy container and then processed in the user container. The queue proxy only allows a specific number of requests to enter the user container at the same time, and queues requests exceeding this number if necessary. The number of parallel requests processed is specified by the concurrency parameters that have been configured for a particular revision. Depending on the concurrency set in the revision, the cue proxy can only simultaneously process that number of requests by the user container, thus queueing requests as necessary.

Each cue proxy measures the incoming load and then reports the average concurrency per second and the request on an individual port. The metrics for all queue proxy containers are scraped by the Autoscaler component, which determines the number of new pods to be added or removed to maintain the desired level of concurrency.

2.5. Reinforcement Learning

Reinforcement Learning (RL) refers to a collection of trials and errors in which agents are trained to interact with their environment and make good decisions by being given positive or negative feedback in the form of rewards for each action. The widely used RL algorithm is Q-learning without a model. Q-learning is a stepwise learning that can train the approximation of the optimal behavior value. The approximation of the Q value specifies the cumulative reward that can be expected when the agent starts with state (s), takes action (a), and then forever acts on the optimal policy afterward. By observing the actual reward in each iteration, the optimization of the Q-function is performed step by step.

RL describes how much newly observed information redefines old information, along with discount factors used to balance present and future rewards. Since RL is a trial-and-error method, the agent must choose between exploring new measures and using the current best option during training [9]. With a certain probability, the agent selects the task that is anticipated to maximize the expected return based on the optimal policy; i.e., the task with the highest Q value starting with s.

2.6. Introduction of Deep Q Network, Double DQN

The Deep Q Network (DQN), a representative reinforcement learning algorithm, is based on the mathematical model definition of the Markov Decision Process (MDP), wherein an agent exploring an environment recognizes the current state and takes an action to obtain an optimal policy for a series of actions that maximizes the cumulative rewards. Figure 1 illustrates the application of the idea of the target network to DQN, which is expressed by updating the Q value in Q-Learning [34,35,36].

$$Q\left(s,a\right)=Q\left(s,a\right)+alpha \left(R+gamma maxQ\left(s prime ,a prime \right)-Q\left(s,a\right)\right)$$

Figure 1. Structure of learning using the target network in DQN & Double DQN.

The target network maintains a fixed value while the original Q Network learns, but it periodically resets to the original Q Network value. This can lead to effective learning because it can be close to the Q Network with such a fixed target network [37,38,39,40].

$$Q\left(s,a\right)\to R+gamma maxQ\left(s prime ,a prime\right)$$

As shown in Figure 1, Double DQN is an algorithm that involves the use of two Q Networks to improve the accuracy of Q-learning before DQN. However, the fact that Q values tend to be overestimated is a limitation, as the accuracy of Q-learning is reduced. Double DQN is an algorithm developed by combining the above ideas in the existing DQN algorithm with two Q Networks. When the state value s′ is given in maxQ(s′,a′) in the existing Q-learning formula, a′ with the highest Q value in the Q Network is selected, after which the Q value is multiplied by γ to derive the target value that Q(s,a) should be close to [41,42]. The aspect that needs to be improved is that the same Q network is used in both the selection process and the process of obtaining the Q value. The Q value selected to use the maximum operation is generally a large value, and as learning progresses, the Q value increases more than necessary. As a result, when the Q value increases, the performance of the Q Network is greatly degraded. To prevent this, two Q Networks should be created and learned [43,44].

3. Introducing Intelligent Kubernetes Framework Design Methods and Workloads

For customized HPA-based auto-scaling technology, we designed an Intelligent Kubernetes framework that is based on Machine Learning algorithms and which can be expanded. This section describes the first experiment conducted to evaluate the performance of the HPA-based auto-scaling and methods used in the experiment to evaluate AI based auto-scaling. This section also introduces the Microservice Architecture monitoring system environment used to evaluate the auto-scaling performance.

3.1. Proposed Microservice Architecture for Auto-Scaling Performance Evaluation

Microservice computing is used for customized applications that have a variety of resource requirements. For example, high traffic loads require significant memory and computing power high parallel analysis operations to process video materials such as Netflix and YouTube videos. Meanwhile, applications like chatbots tend to be low in computing intensity. However, as the number of users increases, the number of requests also increases, thereby requiring more responses. To investigate the auto-scaling impact of various workloads, a synthesized and reliable workload profile that can simulate microservice applications needs to be created. In this study, an API YAML (API Version: v1bera1, Kind: Deployment) file was created to specify a target port. A load operation was then run on the PHP-Apache server. Figure 2 illustrates the Intelligence Microservice Architecture proposed in this study.

Figure 2. Proposed Intelligence Microservice Architecture for Auto-Scaling Performance Evaluation.

Two separate clusters were set up using Kubernetes to test the auto-scaling capabilities in a high-performance computing environment (Intel i9 X processor, 128GB RAM, NVIDIA GPU RTX 3090 * 2). The sample services used in the experiment were distributed to the service cluster. Each cluster is designed to provide sufficient capacity to host all components as well as avoid performance limitations. Based on the collected metrics, the agent manages the activities of the two clusters, such as updating the configuration of the sample service and adjusting the process flow of the experiment by acting as a Kubernetes user. The existing HPA controls the statuses of the deployed Kubernetes services and enables automatic scaling of additional pods.

ML’s Reinforcement Learning method was used to update the auto-scaling configuration of the service in each iteration to create a new output each time. To comprehensively test the auto-scaling function, we increased the number of copies of the receiving gateways to process load balancing, while focusing only on the auto-scaling function. We also experimentally investigated the Auto-Scaling method using the existing reinforcement learning algorithm and the newly proposed Q Network algorithm.

3.2. Customized Kubernetes Environmental Monitoring Architecture

The application performance within the Kubernetes cluster was examined as depicted in Figure 3 below with an architecture designed to monitor the designated Kubernetes environment.

Figure 3. Customized Kubernetes Environmental Monitoring Architecture.

Using the Open-Source Service (Basic Kubernetes monitoring program), we examined the application performance within the Kubernetes cluster as an architecture to monitor the designated Kubernetes environment. The basic process flow is shown in Figure 3. These metrics (Kubelet, Resource Estimator, and Metrics Server) are used in the core system components, such as the scheduling logic and simple basic UI components. The monitoring pipelines used to collect and expose various metrics in the system to end users via adapters are the Horizontal Pod Autoscaler and the Infrastructure. Two types of metrics are used: system metrics and service metrics. System metrics can be used by all monitored entities, such as CPU and memory usage per container and node. Meanwhile, service metrics are explicitly defined in the application code and exported. Both metrics can be used in the user’s container or system infrastructure components (master components such as API servers, add-on pods running on the master, and add-on pods running on the user nodes).

3.3. Customized New Q Network Algorithm Applied to the Experiment

This study introduced a Quad Q Network algorithm that performs Compare Optimizer work on algorithms combining the ideas of Dueling DQN algorithm with existing double DQN algorithms. The existing method of combining Dueling DQN and Double DQN derives the Double q_s value by substituting the argmax equation based on the calculated q_θ’s value of the Q1 Network in the Dueling DQN with the q_θ’s’ value through the new Q2 Network to go through the Double DQN process. Although this algorithm—which operates through the fusion of the Dueling method and the Double method—is structurally complex, it is expected to lead to high performance; however, there is one problem: It is impossible to determine the performance when combining the argmax value based on the Q value calculated by the Q1 Network with the Q2 Network, and it is better to combine the argmax value based on the Q2 Network with the Q1 Network. Therefore, after obtaining the value opposite to the existing value, we strengthened the Q network through the comparison process, thus reducing the loss value and increasing the compensation. The proposed algorithm provides a basis for selecting a Q value that derives excellent performance through comparison and selection of each loss value. This process can describe the structure of a Quad Q Network using four Q networks, then represent it as shown in Figure 4.

Figure 4. Quad Q Network algorithm structure that derives Target Q values with the structures of the Quad Q Network with four Q Networks.

4. Experiment and Results

To determine the performance of intelligent microservices, we first conducted a baseline experiment comparing different workloads in terms of relative performance. This section simulates the application’s existing Horizontal Pod Autoscaler, the Autoscaler applying reinforcement learning to HPA, and the Autoscaler incorporating the proposed reinforcement learning algorithm. In the existing HPA work, the CPU allocation gradually increased for each new experiment, starting with the number of cores. To maximize each core, the response to the request was simulated by applying a load. The time using each core was measured while the maximum number of cores was limited to 10. The detailed experimental results are summarized in Table 2.

Table 2. HPA-based auto-scaling performance test results.

Name	Reference	Targets	Minpods	Maxpods	Replicas	Time
HPA	Deployment/HPA	458%/50%	1	10	1	1 min 14 s
HPA	Deployment/HPA	411%/50%	1	10	2	1 min 32 s
HPA	Deployment/HPA	364%/50%	1	10	3	1 min 47 s
HPA	Deployment/HPA	331%/50%	1	10	4	2 min 2 s
HPA	Deployment/HPA	280%/50%	1	10	5	2 min 35 s
HPA	Deployment/HPA	241%/50%	1	10	6	2 min 44 s
HPA	Deployment/HPA	200%/50%	1	10	7	2 min 59 s
HPA	Deployment/HPA	164%/50%	1	10	8	3 min 17 s
HPA	Deployment/HPA	140%/50%	1	10	9	3 min 31 s
HPA	Deployment/HPA	122%/50%	1	10	10	4 min 24 s

Since there is no intelligent algorithm in the existing HPA work that is suitable for predicting overload with relation to the target share, the number of cores is opened one by one with the existing method. The work was confirmed to be gradually carried out. The number of cores started with one. All cores were opened in 4 min and 24 s. This method requires improvement because it is ineffective in responding to more traffic requests.

To solve this problem, in Autoscaler work equipped with AI algorithms, the DQN algorithm has been embedded in the existing HPA method to make the most of each core. The learning procedure is as follows. The input data value was set as the percentage loaded on the CPU, which is the target value. And the compensation system aimed to reduce overload by expanding the Pod as quickly as possible. The target value was reduced to a minimum, and reaching the target quickly was set as the best compensation value. To prevent the use of the core without limitation, the CPU load was continuously checked. In addition, penalties were set up to prevent unnecessary use of pods. If CPU resources are needed when opening the core, +1 point is given. If CPU resources were optimized when opening the core, −1 point is given. If it was difficult to judge, the learning was conducted by setting it to +0 point. The learning results are shown in Figure 5.

Figure 5. Deep Q Network-based intelligent autoscaler compensation value result.

The highest value was derived from the 41st out of the 50 total number of learning, with a compensation value of 7, and overall, the compensation value is upward. The same load was given to the intelligent Autoscaler to perform simulations in a similar manner to the previous environment. The maximum number of cores was limited to 10. The time used for each core was measured. The detailed experimental results are listed in Table 3.

Table 3. DQN-based auto-scaling performance test results.

Name	Reference	Targets	Minpods	Maxpods	Replicas	Time
i1-MSA	Deployment/i1-MSA	438%/50%	1	10	1	25 s
i1-MSA	Deployment/i1-MSA	419%/50%	1	10	2	31 s
i1-MSA	Deployment/i1-MSA	359%/50%	1	10	3	34 s
i1-MSA	Deployment/i1-MSA	330%/50%	1	10	4	39 s
i1-MSA	Deployment/i1-MSA	276%/50%	1	10	5	46 s
i1-MSA	Deployment/i1-MSA	222%/50%	1	10	6	48 s
i1-MSA	Deployment/i1-MSA	198%/50%	1	10	7	53 s
i1-MSA	Deployment/i1-MSA	172%/50%	1	10	8	57 s
i1-MSA	Deployment/i1-MSA	148%/50%	1	10	9	1 min 1 s
i1-MSA	Deployment/i1-MSA	119%/50%	1	10	10	1 min 10 s

The Autoscaler equipped with AI algorithms began with one core according to the target share. CPU allocation was rapidly increased for each new experiment. Since there was an intelligent algorithm suitable for predicting overload, the Autoscaler quickly opened the cores, rather than proceeding according to the one-by-one system. It was also confirmed that the work proceeded radically. The number of cores began with one. All cores were opened in one minute and one second. This method helps stabilize the system by predicting a response to a future request according to the value loaded on the target.

Using this process, we proved that the AI Autoscaler had remarkable performance compared to the existing HPA. We also compared its performance with that of the Autoscaler equipped with concepts aiming to strengthen the Q Network process in the DQN structure.

As shown in Figure 6 at the top, the highest value was derived with the compensation value 9 at the 26, 39, and 40th out of the total number of learning, and better performance was derived than before. In Autoscaler work equipped with new AI algorithms, CPU allocation increased faster than it previously did for each new experiment, starting with one core. To make the most of the cores, the response to each request was loaded to perform simulations in a manner similar to the previous environment. The maximum number of cores was also limited to 10. The time used for each core was then measured. The detailed experimental results are summarized in Table 4.

Figure 6. Quad Q Network-based intelligent autoscaler compensation value result.

Table 4. Improved DQN-based auto-scaling performance test results.

Name	Reference	Targets	Minpods	Maxpods	Replicas	Time
i2-MSA	Deployment/i2-MSA	432%/50%	1	10	1	19 s
i2-MSA	Deployment/i2-MSA	415%/50%	1	10	2	22 s
i2-MSA	Deployment/i2-MSA	364%/50%	1	10	3	26 s
i2-MSA	Deployment/i2-MSA	329%/50%	1	10	4	33 s
i2-MSA	Deployment/i2-MSA	269%/50%	1	10	5	37 s
i2-MSA	Deployment/i2-MSA	224%/50%	1	10	6	42 s
i2-MSA	Deployment/i2-MSA	198%/50%	1	10	7	46 s
i2-MSA	Deployment/i2-MSA	168%/50%	1	10	8	51 s
i2-MSA	Deployment/i2-MSA	129%/50%	1	10	9	55 s
i2-MSA	Deployment/i2-MSA	114%/50%	1	10	10	59 s

The Autoscaler equipped with new AI algorithms has an intelligent algorithm that is more suitable for predicting overloads. It can perform tasks by opening the cores very effectively due to its excellent prediction of performance loads. It was also confirmed that the work proceeded radically, starting with one core. All cores were opened in 59 s. This method helps stabilize the system by making more effective predictions based on the value loaded on the target.

Figure 7 above depicts the Auto-Scaling Performance Measurement Results by Pod Scaler Type. The results for HPA, i1-MSA (Intelligent Autoscaler Mounted), and i2-MSA (Improved DQN Algorithm) are also shown. The performances of the Autoscalers equipped with AI algorithms were found to be significantly higher than that of HPA, with the Autoscaler equipped with the newly developed DQN algorithm showing the best performance.

Figure 7. Auto-Scaling Performance Measurement Results by Pod Scaler Type.

5. Conclusions

With the advent of the microservice framework, auto-scaling capabilities that meet various demands have emerged as a major area of interest. Numerous expansion mechanisms have been developed. This study focused on request-based expansion, where we first investigated the performance and efficiency of core expansion for requests by mounting AI algorithms. The results of the experiments showed significant differences in throughput as well as deviations in maximum moisture from average latency. Therefore, the presence or absence of AI algorithms can affect performance. To flexibly adjust the auto-scaling settings to meet the particular requirements of a given situation, we designed RL models based on Q-learning and evaluated their applicability to learn effective scaling policies during runtime. Further, based on other algorithms with an enhanced Q Network, we showed that the proposed model could effectively and appropriately adjust the number of cores within a limited time, thereby outperforming the average throughput of the default HPA. Given these results, the presented research makes a significant contribution to existing work in the fields of both microservice frameworks and RL-based auto-scaling applications.

Based on the experimental results, we identified the following limitations of this approach: First, the maximum number of cores was limited to 10. There was no experiment investigating unexpected variables depending on the increasing load and whether it could perform better in a real-life environment that required a larger amount of traffic work. We developed the RL approach to learn core expansion policies mainly by testing the load on the target. However, it is necessary to analyze the extent to which the resource use ratio of individual components can affect performance. Therefore, there is a need for a comprehensive study to find the combination of usage levels that is capable of achieving the best possible performance in all microservices.