Model-Based Reinforcement Learning for Containing Malware Propagation in Wireless Radar Sensor Networks

Abstract

To address malware containment challenges in WRSNs—where traditional integer-order models neglect propagation memory effects and standard reinforcement learning (RL) suffers from slow trial-and-error limitations—we propose the following: (1) a fractional-order VCISQ epidemic model capturing temporal dependencies for higher accuracy, and (2) a model-based Soft Actor–Critic (MBSAC) method, which integrates a learned transition model into an actor–critic architecture to predict future states from limited data, accelerating learning. Experiments confirm MBSAC outperforms RL baselines by reducing control overhead, hastening convergence, and enhancing robustness. It alleviates the rigidity of the traditional method and establishes a reward-driven safeguard for WRSNs.

1. Introduction

Wireless radar sensor networks (WRSNs) have powerful detection, monitoring, and information dissemination capabilities [1]. They play an essential role in intelligent transportation systems, target identification and tracking, and environmental monitoring [2,3]. Ensuring the long-term stability of sensor networks hinges on the security of the network environment [4]. However, as shown in Figure 1, the intrusion and spread of malware in WRSNs are significant and pressing issues [5]. This directly endangers the normal communication functions of internal network devices. To effectively address this threat, many studies have concluded that proactively deploying control mechanisms is the most promising way to optimize and curb the spread of malware [6,7]. However, it is important to note that excessive control measures may be ineffective and costly. Therefore, the key challenge lies in determining and applying the optimal level of control to mitigate the spread of malware in WRSNs.

In order to generate a control object, it is essential to establish a valuable mathematical model of the spread of malware. Currently, some scholars are using integer-valued differential equations to model the spread of malware in a complex manner [5,8,9,10,11]. However, the role of integer-order models is inherently limited. Their core mechanism is to treat the system evolution as a process in which the system instantaneously transitions from one state to the next at each time step [12]. Notably, the actual propagation of malware is driven by the cumulative effects of a prolonged period of time. This includes, but is not limited to, infection scale and propagation rate, which means that integer-order systems cannot accurately model the spread of malware. In contrast, fractional-order differential equations provide a superior framework. Their core mechanism lies in the fact that they incorporate the cumulative effect of the entire evolutionary history of the system through weighted integration [13]. This non-local dependence on historical trajectories makes the model more effective in fitting the long-term dynamics of malware spread.

In-depth research by many scholars has led to the use of many excellent traditional control methods to overcome the spread of malware [14,15,16]. These methods include the use of game theory [14], fuzzy control [15], and optimal control [16] to implement intermittent control of malware. The method that is most widely recognized by the research community is the optimal control method [4,10,16,17]. This is because optimal control provides a rigorous mathematical framework through variational methods or dynamic programming [4,17], enabling it to optimize the system’s performance indicators in the time domain [10] while satisfying complex dynamic constraints. While these traditional methods have been proven effective in controlling the spread of malware, they are highly dependent on precise mathematical models for complex deductive reasoning. However, due to the uncertainty of sensor network communication channels and the dynamic changes in network topology, the spread of malware is usually not represented by a precise mathematical model [18,19]. As malware self-updates and evolves [20], these traditional methods will gradually become less effective in controlling its spread. Therefore, it is critical to find a control method that relies solely on environmental observation, control input, and control effect evaluation.

Fortunately, a recent breakthrough in this area has been achieved through an artificial intelligence method that relies solely on rewards. Many researchers have used reinforcement learning (RL) methods to solve a variety of control problems [21,22,23,24]. Several studies have also used RL methods to address the issue of malware propagation [4,25,26]. Previous RL algorithms used to control the spread of malware include the Soft Actor–Critic (SAC) [4], Proximal Policy Optimization (PPO) [5], and Deep Q-Network (DQN) [27]. These MFRL methods do not require precise mathematical models to output effective control and successfully overcome the limitations of traditional methods. However, these MFRL algorithms all aim to build an effective intelligent agent network using a large amount of interaction experience. This means that these RL algorithms have poor sample efficiency. Unfortunately, the high-speed diffusion properties of malware in WRSNs are fundamentally incompatible with RL’s long-term training [28]. This results in a delay in strategy updates, which can prevent the system from accurately controlling the spread of malware. Therefore, improving RL’s learning efficiency is a key area for further research.

Recently, researchers have used model predictive control (MPC) to improve the timeliness of control [29,30,31]. MPC can simulate the control results before making control decisions, allowing for the selection of a more optimal control sequence, which can effectively reduce unnecessary sample trials in the actual operation [32]. At the same time, it optimizes each time according to the latest system state, making full use of the sample information to accelerate the process of exploring strategies. However, these existing MPC methods depend on precise mathematical models and do not have the advantage of RL, which relies solely on self-interaction. Therefore, using MPC to generate an MFRL-based neural network predictive model will effectively improve RL’s sample efficiency. To the best of our knowledge, this is the first study to use a fractional-order propagation model to control the spread of malware in WRSNs. The proposed model-based RL (MBRL) method demonstrates superior learning efficiency in controlling the spread of malware.

Specifically, the main contributions of this paper are as follows:

(1)

Based on previous research [5], this paper constructs a fractional-order VCISQ propagation model to more accurately describe the spread of malware in WRSNs. At the same time, this paper derives a fractional-order optimal control method for the fractional-order VCISQ propagation model. Through fractional-order optimal control, the theoretical optimal control effect under treatment and quarantine measures is obtained. At the same time, the final control cost generated becomes the standard by which the control effect of RL methods is judged.

(2)

This study proposes a method called the Model-Based Soft Actor–Critic (MBSAC) MBRL algorithm, which is based on the general RL method SAC and has an additional prediction network that can use existing interaction data to predict future interaction environments. MBSAC can also use interaction and predictive data to optimize the agent network simultaneously, reducing the number of interactions required for RL methods and improving their learning efficiency. Compared to the current SOTA general RL methods, MBSAC has lower control costs, faster convergence, and more stable control processes.

The other sections of the text are organized as follows: Section 2 introduces related work; Section 3 builds the fractional-order VCISQ model and derives the fractional-order optimal control method; Section 4 proposes the MBRL algorithm MBSAC; Section 5 carries out multi-dimensional verification of the model and method; and Section 6 summarizes the contributions of this paper and suggests future improvements. Figure 2 outlines each step of the research process.

2. Related Work

In many existing studies, scholars have used integer-order models to model the spread of malware [5,9,10,11]. In [5], the authors proposed an integer-order VCISQ epidemic model to describe malware propagation in WRSNs and combined it with control methods to effectively suppress malware propagation. In [9], an integer-order hybrid epidemic propagation model was proposed to study the dynamic behavior of malware spreading via external storage devices. In [10], an integer-order epidemic-based malware propagation model was established to precisely describe the propagation patterns and random interference characteristics of malware in unmanned aerial vehicle–wireless sensor network (UAV-WSN) systems. In [11], an integer-order epidemic model was constructed by scholars to analyze the propagation characteristics of malware in multi-layer networks, and a hybrid maintenance strategy was introduced to suppress malware propagation. These propagation models effectively simulate malware propagation to a certain extent. However, the non-memory characteristics of integer-order systems do not align with the actual propagation effects of malware [12,33]. Therefore, they cannot accurately fit the propagation of malware. It is worth noting that the global memory property of fractional-order differential equations partially overcomes this limitation, making the propagation model more aligned with reality [13].

Figure 2. Research Flowchart.

Figure 2. Research Flowchart.

Some excellent traditional control methods have been used by researchers to control malware [5,14,15,16]. In [5], researchers built a hybrid control strategy and proposed an integer-order optimal control method to suppress the spread of malware in WRSNs. In [14], scholars have proposed a patching strategy based on the differential game theory method to effectively suppress the spread of malware in the Industrial Internet of Things (IIoT). In [15], a self-adaptive integer-order SIR model is proposed, combining fuzzy control and gradient descent optimization to control the spread of malware in IIoT networks. In [16], researchers effectively reduce the spread of malware in IoT networks and enhance system security using an integer-order optimal control strategy. These traditional control methods have proven effective in controlling the spread of malware. However, they require a precise mathematical model of the malware propagation process, which is difficult to obtain due to the high degree of uncertainty in sensor network communication channels and network topology [18,19], thus reducing the adaptability of traditional control algorithms.

To overcome the limitations of traditional control methods in terms of adaptability, different deep learning (DL) methods have been developed to control the spread of malware [34,35,36,37]. In [34], a DL-based fine-grained hierarchical learning method was proposed, which effectively controlled the spread of malware in IoT by extracting its behavioral patterns. In [35], a novel network forensics framework based on multi-layer perceptrons was proposed to enhance the network’s ability to suppress the spread of malware. In [36], a DL method based on multi-layer convolutional neural networks was proposed to effectively prevent the spread of malware in IIoT. In [37], a self-supervised learning method based on a masked autoencoder framework was proposed, which identified malware network traffic to achieve effective control of malware. Due to the fact that DL methods only depend on the network model obtained during training, they are more adaptable than traditional methods. However, DL methods often require a large amount of high-quality manually labeled data for network training. In practice, obtaining large amounts of data for WRSNs is costly [5,10], which means that using DL methods alone to suppress the spread in WRSNs is not realistic.

In environments with a strong demand for adaptability, the study of controlling malware using MFRL methods has emerged [4,25,26,38].

In [4], the Multi-Agent SAC (MASAC) method is used to effectively control the spread of malware in the IoV. In [25], a framework called GAME-RL is proposed to generate adversarial malware samples to effectively prevent the spread of malware. In [26], a method combining federal learning and the Q-RL framework was proposed to effectively control the spread of malware on IoT devices. In [38], researchers proposed an RL-based functional verification attack framework that effectively increased the efficiency of generating adversarial examples and controlled the spread of malware. These MFRL methods largely overcome the need for precise models and high-quality data. However, an excellent policy network often requires a long training period to be established [13]. Nevertheless, the long training period of RL is in fundamental conflict with the fast spread of malware in WRSNs [28].

Fortunately, new work on MPC is constantly emerging [30,39,40]. In [30], an MPC method based on a feedback algorithm was proposed to solve the optimization problem of the cost function and constraints in MPC. In [39], scholars proposed a framework based on DL and MPC methods to control and mitigate the security threats posed by tampering attacks on the power grid. In [40], a self-adaptive control framework combining an unknown input observer and MPC methods was designed to effectively control the safety and stability of the power system. These studies all use the predictive optimization idea of MPC to determine the optimal control sequence in advance. However, MPC methods alone cannot do without precise mathematical models [41], and lack the self-optimizing ability of RL. Conversely, if MPC’s predictive control sequence can be used to assist RL in exploration, it will accelerate RL training. With this approach, MFRL will become MBRL and achieve higher sample efficiency [42,43,44].

Table 1. Comparison of related works.

Advantage	Fractional-Order Modeling	Traditional Control Method or DL Control Method	RL Control Method	Utilizing MPC Thinking to Accelerate Policy Network Learning
Study
Essouifi et al. [9]	×	√	×	×
Peng et al. [11]	×	√	×	×
Zheng et al. [15]	×	√	×	×
Liu et al. [10]	×	×	√	×
Abusnaina et al. [34]	×	√	×	×
Mei et al. [35]	×	√	×	×
Ahmed et al. [36]	×	√	×	×
Tian et al. [38]	×	×	√	×
Liu et al. [4]	√	×	√	×
Liu et al. [5]	×	×	√	×
Shen et al. [14]	×	√	×	×
Jafar et al. [16]	×	√	×	×
Zhan et al. [25]	×	×	√	×
Feng et al. [26]	×	×	√	×
Lin et al. (ours)	√	×	√	√

presents a comparison of the relevant works.

3. Fractional-Order Model and Optimal Control Method

3.1. Fractional-Order Malware Propagation Model

In previous work [5], a model for the propagation of the VCISQ malware was proposed. As shown in Figure 3, this model contains five state nodes, as follows: susceptible node $V\left(t\right)$, carrier node $C\left(t\right)$, infected node $I\left(t\right)$, secured node $S\left(t\right)$, and quarantined node $Q\left(t\right)$.

(1)

Susceptible node $V\left(t\right)$: $V\left(t\right)$ is a sensor node with vulnerabilities but not yet infected with the malware. It may carry the malware and transform into the carrier node $C\left(t\right)$ by interacting with the infected node $I\left(t\right)$. Alternatively, it may transform into the quarantined node $Q\left(t\right)$ due to quarantine measures.

(2)

Carrier node $C\left(t\right)$: After ${\tau }_i$ time, the malware is activated and the carrier node $C\left(t\right)$ is transformed into the infected node $I\left(t\right)$.

(3)

Infected node $I\left(t\right)$: $I\left(t\right)$ is a node that has activated and is spreading the malware. It will change to the secured node $S\left(t\right)$ after ${\tau }_p$ time, due to treatment measures such as installing a patch.

(4)

Secured node $S\left(t\right)$: $S\left(t\right)$ is a node that has been patched and restored, and is temporarily immune to malware. However, the patch effect may weaken after H time, and it may revert to a susceptible node $V\left(t\right)$.

(5)

Quarantined node $Q\left(t\right)$: $Q\left(t\right)$ is a node that has been quarantined and is no longer participating in network interactions. It may revert to susceptible node $V\left(t\right)$ after h time if the quarantine fails. Alternatively, it may be transformed into the secured node $S\left(t\right)$ if treatment measures such as installing a patch are implemented.

In the model, the WRSN system considers the delay factors in the real system. These include the delay ${\tau }_i$ in installing malware, which describes the time it takes for the malware to be carried and activated. The delay ${\tau }_p$ in installing patches describes the time it takes for the patch to be deployed and take effect. The delay ${\tau }_s$ in patch expiration describes the time it takes for the patch to gradually expire on the secured node $S\left(t\right)$. The delay h in quarantine expiration describes the time it takes for the quarantine measures to gradually expire on the quarantined node $Q\left(t\right)$.

The VCISQ model can partially simulate the spread of malware in WRSNs by integrating multiple factors. However, integer-order models have inherent limitations. Their current state is only determined by the previous moment. This is unable to match the continuous influence of past states on current states in the actual spread of malware [33]. To effectively describe such phenomena with genetic characteristics and cumulative effects, Caputo fractional-order models have proven to be highly applicable [13]. Considering the need for a more accurate description of the effects of malware spread, this study proposes the VCISQ malware Caputo fractional-order spread model:

$$\begin{array}{cc}\hfill \stackrel{C}{t_0}{D}_{t_d}^{q}V\left(t\right)& =-2\varphi \left(t\right)\sqrt{I\left(t\right)}\mu \left(t\right)+\Lambda S(t-{\tau }_s)-{\alpha }_2\left(t\right)V\left(t\right)+(1-k){\alpha }_2(t-h)V(t-h)e^{-h\xi },\hfill \\ \hfill \stackrel{C}{t_0}{D}_{t_d}^{q}C\left(t\right)& =2\varphi \left(t\right)\sqrt{I\left(t\right)}\mu \left(t\right)-2\Omega \varphi (t-{\tau }_i)\sqrt{I(t-{\tau }_i)}\mu (t-{\tau }_i),\hfill \\ \hfill \stackrel{C}{t_0}{D}_{t_d}^{q}I\left(t\right)& =2\Omega \varphi (t-{\tau }_i)\sqrt{I(t-{\tau }_i)}\mu (t-{\tau }_i)-{\alpha }_1(t-{\tau }_p)I(t-{\tau }_p),\hfill \\ \hfill \stackrel{C}{t_0}{D}_{t_d}^{q}S\left(t\right)& ={\alpha }_1(t-{\tau }_p)I(t-{\tau }_p)-\Lambda S(t-{\tau }_s)+k{\alpha }_2(t-h)V(t-h)e^{-h\xi },\hfill \\ \hfill \stackrel{C}{t_0}{D}_{t_d}^{q}Q\left(t\right)& ={\alpha }_2\left(t\right)V\left(t\right)-k{\alpha }_2(t-h)V(t-h)e^{-h\xi }-(1-k){\alpha }_2(t-h)V(t-h)e^{-h\xi }.\hfill \end{array}$$

(1)

Among them, $\stackrel{C}{t_0}{D}_{t_d}^q$ is called the q-order Caputo fractional derivative, where $t_0$ is the initial time and $t_d$ is the target time. Unlike integer-order differential equations, if $Y\left(t\right)$ requires a Caputo fractional derivative expansion, it will follow the following expression [4]. It should be noted that the Gamma function is represented by $\Gamma \left(z\right)$, whose expansion is $\Gamma \left(z\right)={\int }_0^{\infty }{e}^{-s}{s}^{z-1}ds$

$$\stackrel{C}{t_0}{D}_{t_d}^{q}Y\left(t\right)={\displaystyle \frac{1}{\Gamma (1-q)}}{\int }_{t_0}^t{\displaystyle \frac{Y\left(s\right)}{{(t-s)}^q}}ds,0<q\le 1.$$

(2)

The initial value $G_l\in [0,\infty ),l=1,2,\dots ,5$ of each state equation is non-negative and invariant. According to the above settings, the initial conditions of the VCISQ model (1) follow the following expressions:

$${(V\left(t\right),C\left(t\right),I\left(t\right),S\left(t\right),Q\left(t\right))}^T={(G_1,G_2,G_3,G_4,G_5)}^T$$

(3)

In (1), $\varphi \left(t\right)$ represents the number of neighboring nodes of susceptible node $V\left(t\right)$ in the VCISQ model. $\mu \left(t\right)$ represents the contact rate between neighboring nodes in the VCISQ model. In [5], $\varphi \left(t\right)$ is established based on the interaction range of radar nodes, Rayleigh fading, path loss, SNR threshold, node spatial distribution, and the degree of difference between ideal and actual communication. In [5], $\mu \left(t\right)$ is established based on the time-varying effects of node distribution, contact opportunities, and device transmission power.

Specifically, $\varphi \left(t\right)$ is given by the following expression:

$$\varphi \left(t\right)=\sqrt{{\displaystyle \frac{2q_c\pi {\chi }^{\frac{2}{a}}}{\alpha {\sigma }_n^{\frac{4}{a}}{\gamma }_{th}^{\frac{2}{a}}{a}^2}}\Gamma \left({\displaystyle \frac{2}{a}}\right)V\left(t\right)}$$

(4)

Additionally, $\mu \left(t\right)$ follows the following expression:

$$\left\{\begin{array}{c}\mu \left(t\right)={\left(1-e^{-{\displaystyle \frac{1}{2}}\eta }\right)}^t,\hfill \\ \eta ={\displaystyle \frac{2\rho \pi {\chi }^{\frac{2}{a}}}{\alpha {\sigma }_n^{\frac{4}{a}}{\gamma }_{th}^{\frac{2}{a}}}}\Gamma \left({\displaystyle \frac{2}{a}}\right).\hfill \end{array}\right.$$

(5)

To optimize the VCISQ model (1) for the suppression of malware propagation, it is necessary to introduce treatment and quarantine control quantities (${\alpha }_1\left(t\right)$,${\alpha }_2\left(t\right)$) to implement intermittent control with varying intensity. However, the inherent resource limitations of WRSNs restrict the control intensity (${\alpha }_1$,${\alpha }_2$) and the WRSN’s inherent resource constraints. To balance control effectiveness and resource consumption within these constraints, we must perform a real-time evaluation of the system state at each time point t and decide on the optimal control action (${\alpha }_1\left(t\right)$,${\alpha }_2\left(t\right)$). We consider the allowable control function space k on the time interval T, which is defined as the set of all Lebesgue measurable admissible control functions:

$$k=\{({\alpha }_1(·),{\alpha }_2(·))\in L^{\infty }(0,t_d):0\le {\alpha }_1(·),{\alpha }_2(·)\le 1\}.$$

(6)

Among them, $t_d$ represents the termination control moment.

The core objective of implementing control for the VCISQ model (1) is to minimize control costs while suppressing the spread of malware. Susceptible nodes $V\left(t\right)$ are at risk of being attacked by malware and gradually becoming unable to work. The number of infected nodes $I\left(t\right)$ directly affects the security of the WRSN system. Excessive quarantined nodes $Q\left(t\right)$ also prevent WRSN systems from operating efficiently. Therefore, reducing these nodes is an effective way to mitigate the spread of malware. Control measures ${\alpha }_1\left(t\right)$ and ${\alpha }_2\left(t\right)$ are introduced to quickly suppress the spread of malware, but their implementation consumes energy and bandwidth resources. Weighting coefficients $B_1,B_2,B_3,B_4$, and $B_5$ are used to quantify the impact of different parts in the cost function. Specifically, they include the damage caused by different numbers of sensor nodes and the resource consumption of different control measures. The cost function is defined as follows:

$$J(V\left(t\right),I\left(t\right),Q\left(t\right),{\alpha }_1\left(t\right),{\alpha }_2\left(t\right))={\int }_0^T\left(B_{1}V\left(t\right)+{\left(B_{2}I\left(t\right)\right)}^{\beta }+B_{3}Q\left(t\right)+B_4{\alpha }_1^2\left(t\right)+B_5{\alpha }_2^2\left(t\right)\right)dt.$$

(7)

The parameters ${\alpha }_1\left(t\right)$ and ${\alpha }_2\left(t\right)$ in the equation represent control measures to suppress the spread of malware, while the weighting coefficients $B_1,B_2,B_3,B_4$, and $B_5$ represent the damage caused by different numbers of sensor nodes and the resource consumption of different control measures, respectively.

All parameters involved in the VCISQ model (1) are defined in

Table 2. Model parameters.

	Parameters	Values
Model Parameters	Initial number of susceptible nodes $G_1$	270
	Initial number of carrier nodes $G_2$	0
	Initial number of infected nodes $G_3$	30
	Initial number of secured nodes $G_4$	0
	Initial number of QUARANTINED nodes $G_5$	0
	Cost weight of susceptible nodes $B_1$	300
	Cost weight of infected nodes $B_2$	900
	Cost weight of quarantined nodes $B_3$	300
	Cost weight of treatment measure $B_4$	12
	Cost weight of quarantine measure $B_5$	12
	Attack severity of the malware $\beta$	0.85
	Malware installation delay ${\tau }_i$	1
	Patch installation delay ${\tau }_p$	0.5
	Patch failure delay ${\tau }_s$	2
	Quarantine failure delay h	1
	Failure rate of treatment measures $\Lambda$	0.05
	The rate of quarantine k	0.5
	Successful installation rate of malware $\Omega$	0.8
	The communication difference coefficient between ideal and reality $q_c$	$2ln2/\pi$
	Transmit power coefficient $\chi$	$5\times {10}^{-10}$
	Sliding length of the target square area a	500 m
	The exponent of path loss $\alpha$	4
	Intensity of noise ${\sigma }_n^2$	−60 dBm
	Signal-to-noise threshold ${\gamma }_{th}$	3 dB
	The density of total nodes $\rho$	0.6
	Fractional order q	0.9

.

3.2. Fractional Optimal Control Method

In this section, we will derive the fractional-order optimal control method for the fractional-order model in Section 3.1. For a known control model, the state vector $Z\left(t\right)$, the delay vector $D\left(t\right)$, and the control vector $L\left(t\right)$ are defined as follows:

$$\begin{array}{cc}\hfill Z\left(t\right)& =\left(V\right(t),C(t),I(t),S(t),Q(t\left)\right),\hfill \\ \hfill D\left(t\right)& =(Z(t-{\tau }_s),Z(t-{\tau }_i),Z(t-{\tau }_p)),\hfill \\ \hfill L\left(t\right)& =({\alpha }_1\left(t\right),{\alpha }_2\left(t\right)).\hfill \end{array}$$

(8)

According to the formula, the fractional-order VCISQ model (1) can be converted into the following equation: $\stackrel{C}{t_0}{D}_t^{q}Z\left(t\right)=f(Z\left(t\right),D\left(t\right),L\left(t\right))$. The corresponding Hamilton function is then as follows:

$$\begin{array}{cc}\hfill & H\left(Z\right(t),D(t),L(t),\lambda (t\left)\right)\hfill \\ & =B_{1}V\left(t\right)+{\left(B_{2}I\left(t\right)\right)}^{\beta }+B_{3}Q\left(t\right)+B_4{\alpha }_1^2\left(t\right)+B_5{\alpha }_2^2\left(t\right)+\lambda {\left(t\right)}_{t_0}^C{D}_{t_d}^{q}Z\left(t\right)\hfill \\ \hfill & =B_{1}V\left(t\right)+{\left(B_{2}I\left(t\right)\right)}^{\beta }+B_{3}Q\left(t\right)+B_4{\alpha }_1^2\left(t\right)+B_5{\alpha }_2^2\left(t\right)+\lambda \left(t\right)f(Z\left(t\right),D\left(t\right),L\left(t\right))\hfill \\ \hfill & =B_{1}V\left(t\right)+{\left(B_{2}I\left(t\right)\right)}^{\beta }+B_{3}Q\left(t\right)+B_4{\alpha }_1^2\left(t\right)+B_5{\alpha }_2^2\left(t\right)\hfill \\ \hfill & +{\lambda }_1\left(t\right)\left(-2\varphi \left(t\right)\sqrt{I\left(t\right)}\mu \left(t\right)+\Lambda S(t-{\tau }_s)-{\alpha }_2\left(t\right)V\left(t\right)+(1-k){\alpha }_2(t-h)V(t-h)e^{-h\xi }\right)\hfill \\ \hfill & +{\lambda }_2\left(t\right)\left(2\varphi \left(t\right)\sqrt{I\left(t\right)}\mu \left(t\right)-2\Omega \varphi (t-{\tau }_i)\sqrt{I(t-{\tau }_i)}\mu (t-{\tau }_i)\right)\hfill \\ \hfill & +{\lambda }_3\left(t\right)\left(2\Omega \varphi (t-{\tau }_i)\sqrt{I(t-{\tau }_i)}\mu (t-{\tau }_i)-{\alpha }_1(t-{\tau }_p)I(t-{\tau }_p)\right)\hfill \\ \hfill & +{\lambda }_4\left(t\right)({\alpha }_1(t-{\tau }_p)I(t-{\tau }_p)-\Lambda S(t-{\tau }_s)+k{\alpha }_2(t-h)V(t-h)e^{-h\xi })\hfill \\ \hfill & +{\lambda }_5\left(t\right)({\alpha }_2\left(t\right)V\left(t\right)-k{\alpha }_2(t-h)V(t-h)e^{-h\xi }-(1-k){\alpha }_2(t-h)V(t-h)e^{-h\xi })\hfill \end{array}$$

(9)

Among them, $\lambda \left(t\right)={\lambda }_i\left(t\right)$ represents the state equation, $i=1,2,\dots ,5$. They are obtained by taking the partial derivative of the Hamilton function for each state equation.

Due to the fact that ${\alpha }_1\left(t\right)$ and ${\alpha }_2\left(t\right)$ are the control variables corresponding to their respective state variables, if x is defined, then each existing state equation $\lambda \left(t\right)$ should satisfy the following condition:

$$\begin{array}{cc}\hfill \stackrel{C}{t_0}{D}_{t_d}^q\lambda \left(t\right)& =H_Z\left(t\right)+{\chi }_{[0,t_d-{\tau }_s]}{H}_D(t+{\tau }_s)+{\chi }_{[0,t_d-{\tau }_i]}{H}_D(t+{\tau }_i)\hfill \\ \hfill & +{\chi }_{[0,t_d-{\tau }_p]}{H}_D(t+{\tau }_p)+{\chi }_{[0,t_d-h]}{H}_L(t+h),\hfill \\ \hfill {\lambda }_i\left(t_d\right)& =0,i=1,2,\dots ,5,\hfill \\ \hfill H\left(Z\right(t),D(t),L(t),\lambda (t\left)\right)& =\underset{0\le L\left(t\right)\le 1}{min}H\left(Z^{*}\left(t\right),D^{*}\left(t\right),L^{*}\left(t\right),{\lambda }^{*}\left(t\right)\right).\hfill \end{array}$$

(10)

Among them, ${\chi }_{[0,t_d-{\tau }_s]}\left(t\right)=\left\{\begin{array}{cc}1,\hfill & t\in [0,t_d-{\tau }_s]\hfill \\ 0,\hfill & \mathrm{otherwise}\hfill \end{array}\right.,{\chi }_{[0,t_d-{\tau }_i]}\left(t\right)=\left\{\begin{array}{cc}1,\hfill & t\in [0,t_d-{\tau }_i]\hfill \\ 0,\hfill & \mathrm{otherwise}\hfill \end{array}\right.$, ${\chi }_{[0,t_d-{\tau }_p]}\left(t\right)=\left\{\begin{array}{cc}1,\hfill & t\in [0,t_d-{\tau }_p]\hfill \\ 0,\hfill & \mathrm{otherwise}\hfill \end{array}\right.,$ and ${\chi }_{[0,t_d-h]}\left(t\right)=\left\{\begin{array}{cc}1,\hfill & t\in [0,t_d-h]\hfill \\ 0,\hfill & \mathrm{otherwise}\hfill \end{array}\right.$ .

Theorem 1.

For the VCISQ model (1), the state variable can be expressed as follows:

$$\begin{array}{cc}\hfill \stackrel{C}{t_0}{D}_{t_d}^q{\lambda }_1\left(t\right)& =-(B_1+({\lambda }_2\left(t\right)-{\lambda }_1\left(t\right)){\displaystyle \frac{\varphi \left(t\right)}{V\left(t\right)}}\sqrt{I\left(t\right)}\mu \left(t\right)+({\lambda }_5\left(t\right)-{\lambda }_1\left(t\right)){\alpha }_2\left(t\right)\hfill \\ \hfill & +({\lambda }_3(t+{\tau }_i)-{\lambda }_2(t+{\tau }_i))\Omega {\displaystyle \frac{\varphi \left(t\right)}{V\left(t\right)}}\sqrt{I\left(t\right)}\mu \left(t\right)\hfill \\ \hfill & +{\lambda }_1(t+h)(1-k){\alpha }_2\left(t\right)e^{-h\xi }+{\lambda }_4(t+h)k{\alpha }_2\left(t\right)e^{-h\xi }\hfill \\ \hfill & -{\lambda }_5(t+h){\alpha }_2\left(t\right)e^{-h\xi }),\hfill \\ \hfill \stackrel{C}{t_0}{D}_{t_d}^q{\lambda }_2\left(t\right)& =0,\hfill \\ \hfill {t_0}^C{D}_{t_d}^q{\lambda }_3\left(t\right)& =-(\beta B_2{\left(B_{2}I\left(t\right)\right)}^{\beta -1}+({\lambda }_2\left(t\right)-{\lambda }_1\left(t\right)){\displaystyle \frac{\varphi \left(t\right)}{\sqrt{I\left(t\right)}}}\mu \left(t\right)\hfill \\ \hfill & +{\lambda }_4(t+{\tau }_p){\alpha }_1\left(t\right)-{\lambda }_3(t+{\tau }_p)u_1\left(t\right)\hfill \\ \hfill & +({\lambda }_3(t+{\tau }_i)-{\lambda }_2(t+{\tau }_i))\Omega {\displaystyle \frac{\varphi \left(t\right)}{I\left(t\right)}}\mu \left(t\right)),\hfill \\ \hfill \stackrel{C}{t_0}{D}_{t_d}^q{\lambda }_4\left(t\right)& =-({\lambda }_1(t+{\tau }_s)-{\lambda }_4(t+{\tau }_s))\Lambda ,\hfill \\ \hfill \stackrel{C}{t_0}{D}_{t_d}^q{\lambda }_5\left(t\right)& =-B_3.\hfill \end{array}$$

(11)

The optimal control variables (${\alpha }_1^{*}\left(t\right),{\alpha }_2^{*}\left(t\right)$) can then be obtained:

$$\begin{array}{cc}\hfill {\alpha }_1^{*}\left(t\right)& =max\left(min\left(1,{\displaystyle \frac{\left({\lambda }_3(t+{\tau }_p)-{\lambda }_4(t+{\tau }_p)\right)I\left(t\right)}{2B_4}}\right),0\right)\hfill \\ \hfill {\alpha }_2^{*}\left(t\right)& =max(min\left(0,{\displaystyle \frac{\left({\lambda }_1\left(t\right)-{\lambda }_5\left(t\right)\right)V\left(t\right)}{2B_5}}\right.\hfill \\ \hfill & -{\displaystyle \frac{\left({\lambda }_1(t+h)-{\lambda }_5(t+h)\right)V\left(t\right)e^{-h\xi }}{2B_5}}\hfill \\ \hfill & \left.-{\displaystyle \frac{\left({\lambda }_4(t+h)-{\lambda }_1(t+h)\right)kV\left(t\right)e^{-h\xi }}{2B_5}}\right),0)\hfill \end{array}$$

(12)

The optimal control model of VCISQ is obtained by substituting the optimal control variables (${\alpha }_1^{*}\left(t\right),{\alpha }_2^{*}\left(t\right)$) into the VCISQ model (1):

$$\begin{array}{cc}\hfill \stackrel{C}{t_0}{D}_{t_d}^q{V}^{*}\left(t\right)& =-2\varphi \left(t\right)\sqrt{I^{*}\left(t\right)}\mu \left(t\right)+\Lambda S^{*}(t-{\tau }_s)-{\alpha }_2^{*}\left(t\right)V^{*}\left(t\right)\hfill \\ \hfill & +(1-k){\alpha }_2^{*}(t-h)V^{*}(t-h)e^{-h\xi },\hfill \\ \hfill \stackrel{C}{t_0}{D}_{t_d}^q{C}^{*}\left(t\right)& =2\varphi \left(t\right)\sqrt{I^{*}\left(t\right)}\mu \left(t\right)-2\Omega \varphi (t-{\tau }_i)\sqrt{I^{*}(t-{\tau }_i)}\mu (t-{\tau }_i),\hfill \\ \hfill \stackrel{C}{t_0}{D}_{t_d}^q{I}^{*}\left(t\right)& =2\Omega \varphi (t-{\tau }_i)\sqrt{I^{*}(t-{\tau }_i)}\mu (t-{\tau }_i)-{\alpha }_1^{*}(t-{\tau }_p)I^{*}(t-{\tau }_p),\hfill \\ \hfill \stackrel{C}{t_0}{D}_{t_d}^q{S}^{*}\left(t\right)& ={\alpha }_1^{*}(t-{\tau }_p)I^{*}(t-{\tau }_p)-S^{*}(t-{\tau }_s)\hfill \\ \hfill & +k{\alpha }_2^{*}(t-h)V^{*}(t-h)e^{-h\xi },\hfill \\ \hfill \stackrel{C}{t_0}{D}_{t_d}^q{Q}^{*}\left(t\right)& ={\alpha }_2^{*}\left(t\right)V^{*}\left(t\right)-k{\alpha }_2^{*}(t-h)V^{*}(t-h)e^{-h\xi }\hfill \\ \hfill & -(1-k){\alpha }_2^{*}(t-h)V^{*}(t-h)e^{-h\xi }.\hfill \end{array}$$

(13)

Proof. 

The state variable $\lambda \left(t\right)$ is determined by Pontryagin’s maximum principle and (10). Specifically, it is obtained by solving the partial derivatives of the Hamilton function H (9) of the VCISQ system for each state variable:

$$\begin{array}{cc}\hfill \stackrel{C}{t_0}{D}_{t_d}^q{\lambda }_1\left(t\right)& =-{\displaystyle \frac{𝜕H}{𝜕V\left(t\right)}}\hfill \\ \hfill & =-(B_1+({\lambda }_2\left(t\right)-{\lambda }_1\left(t\right)){\displaystyle \frac{\varphi \left(t\right)}{V\left(t\right)}}\sqrt{I\left(t\right)}\mu \left(t\right)\hfill \\ \hfill & +({\lambda }_5\left(t\right)-{\lambda }_1\left(t\right)){\alpha }_2\left(t\right)\hfill \\ \hfill & +({\lambda }_3(t+{\tau }_i)-{\lambda }_2(t+{\tau }_i))\Omega {\displaystyle \frac{\varphi \left(t\right)}{V\left(t\right)}}\sqrt{I\left(t\right)}\mu \left(t\right)\hfill \\ \hfill & +{\lambda }_1(t+h)(1-k){\alpha }_2\left(t\right)e^{-h\xi }\hfill \\ \hfill & +{\lambda }_4(t+h)k{\alpha }_2\left(t\right)e^{-h\xi }\hfill \\ \hfill & -{\lambda }_5(t+h){\alpha }_2\left(t\right)e^{-h\xi }),\hfill \\ \hfill \stackrel{C}{t_0}{D}_{t_d}^q{\lambda }_2\left(t\right)& =-{\displaystyle \frac{𝜕H}{𝜕C\left(t\right)}}=0,\hfill \\ \hfill \stackrel{C}{t_0}{D}_{t_d}^q{\lambda }_3\left(t\right)& =-{\displaystyle \frac{𝜕H}{𝜕I\left(t\right)}}\hfill \\ \hfill & =-(\beta B_2{\left(B_{2}I\left(t\right)\right)}^{\beta -1}+({\lambda }_2\left(t\right)-{\lambda }_1\left(t\right)){\displaystyle \frac{\varphi \left(t\right)}{\sqrt{I\left(t\right)}}}\mu \left(t\right)\hfill \\ \hfill & +({\lambda }_4(t+{\tau }_p)-{\lambda }_3(t+{\tau }_p)){\alpha }_1\left(t\right)\hfill \\ \hfill & +({\lambda }_3(t+{\tau }_i)-{\lambda }_2(t+{\tau }_i))\Omega {\displaystyle \frac{\varphi \left(t\right)}{I\left(t\right)}}\mu \left(t\right)),\hfill \\ \hfill \stackrel{C}{t_0}{D}_{t_d}^q{\lambda }_4\left(t\right)& =-{\displaystyle \frac{𝜕H}{𝜕S\left(t\right)}}=-({\lambda }_1(t+{\tau }_s)-{\lambda }_4(t+{\tau }_s))\Lambda ,\hfill \\ \hfill \stackrel{C}{t_0}{D}_{t_d}^q{\lambda }_5\left(t\right)& =-{\displaystyle \frac{𝜕H}{𝜕Q\left(t\right)}}=-B_3.\hfill \end{array}$$

(14)

Then, use the Hamilton function H to calculate the partial derivatives of the control variables ${\alpha }_1\left(t\right)$ and ${\alpha }_2\left(t\right)$. The optimal conditions can be obtained from this equation: ${\displaystyle \frac{𝜕H}{𝜕{\alpha }_k\left(t\right)}},k=1,2$. Combining this with the optimal cross-section condition (6) gives the following:

$$\begin{array}{cc}\hfill {\left.{\displaystyle \frac{𝜕H}{𝜕{\alpha }_1\left(t\right)}}\right|}_{{\alpha }_1\left(t\right)={\alpha }_1^{*}\left(t\right)}& =2B_4{\alpha }_1\left(t\right)+{\lambda }_4(t+{\tau }_p)I\left(t\right)-{\lambda }_3(t+{\tau }_p)I\left(t\right)=0,\hfill \\ \hfill {\left.{\displaystyle \frac{𝜕H}{𝜕{\alpha }_2\left(t\right)}}\right|}_{{\alpha }_2\left(t\right)={\alpha }_2^{*}\left(t\right)}& =2B_5{\alpha }_2\left(t\right)-({\lambda }_1\left(t\right)-{\lambda }_5\left(t\right))V\left(t\right)\hfill \\ \hfill & +({\lambda }_1(t+h)-{\lambda }_5(t+h))V\left(t\right)e^{-h\xi }\hfill \\ \hfill & +({\lambda }_4(t+h)-{\lambda }_1(t+h))kV\left(t\right)e^{-h\xi }=0.\hfill \end{array}$$

(15)

Applying a transformation to (12) yields the desired optimal control pair (${\alpha }_1\left(t\right)$, ${\alpha }_2\left(t\right)$):

$$\begin{array}{cc}\hfill {\alpha }_1^{*}\left(t\right)& =max\left(min\left(1,{\displaystyle \frac{\left({\lambda }_3(t+{\tau }_p)-{\lambda }_4(t+{\tau }_p)\right)I\left(t\right)}{2B_4}}\right),0\right)\hfill \\ \hfill {\alpha }_2^{*}\left(t\right)& =max\left(min\left(1,{\displaystyle \frac{({\lambda }_1\left(t\right)-{\lambda }_5\left(t\right))V\left(t\right)}{2B_5}}\right.\right.\hfill \\ \hfill & -{\displaystyle \frac{({\lambda }_1(t+h)-{\lambda }_5(t+h))V\left(t\right)e^{-h\xi }}{2B_5}}\hfill \\ \hfill & \left.\left.-{\displaystyle \frac{({\lambda }_4(t+h)-{\lambda }_1(t+h))kV\left(t\right)e^{-h\xi }}{2B_5}}\right),0\right)\hfill \end{array}$$

(16)

Thus, Theorem 1 has been proven.    □

Theorem 2.

The positivity and boundedness of the solution are prerequisites for the existence of an optimal control pair for the fractional-order dynamics model. For the VCISQ model in (1), the solution remains non-negative and bounded at all times t under the initial condition (5).

Proof. 

Based on the fact that ${\mathbb{R}}_5^{+}=\{\varphi \in {\mathbb{R}}_5:\varphi \ge 0\}$, Theorem 3 in [17] shows that the VCISQ model (1) has a unique solution k in $(0,\infty )$. Based on the above conditions, the key to proving that the positive domain ${\mathbb{R}}_5^{+}$ is a VCISQ model is to show that it is positively invariant.

From (1) and (3), we can directly obtain $V(t-h)\in {\mathbb{R}}^{+}$, $I(t-{\tau }_i)\in {\mathbb{R}}^{+}$, $\mu (t-{\tau }_i)\in {\mathbb{R}}^{+}$, $\varphi (t-{\tau }_i)\in {\mathbb{R}}^{+}$, $S(t-{\tau }_s)\in {\mathbb{R}}^{+}$, ${\alpha }_1(t-{\tau }_p)\in {\mathbb{R}}^{+}$, ${\alpha }_2(t-h)\in {\mathbb{R}}^{+}$. Let $\varphi ={(V\left(t\right),C\left(t\right),I\left(t\right),S\left(t\right),Q\left(t\right))}^T=0$. The state values of the VCISQ model (1) at this time are as follows:

$$\begin{array}{cc}\hfill {\left.\stackrel{C}{t_0}{D}_{t_d}^{q}V\left(t\right)\right|}_{V\left(t\right)=0}& =-2\varphi \left(t\right)\sqrt{I\left(t\right)}\mu \left(t\right)+\Lambda S(t-{\tau }_s)\hfill \\ \hfill & +(1-k){\alpha }_2(t-h)V(t-h)e^{-h\xi }\ge 0,\hfill \\ \hfill {\left.\stackrel{C}{t_0}{D}_{t_d}^{q}C\left(t\right)\right|}_{C\left(t\right)=0}& =2\varphi \left(t\right)\sqrt{I\left(t\right)}\mu \left(t\right)\hfill \\ \hfill & -2\Omega \varphi (t-{\tau }_i)\sqrt{I(t-{\tau }_i)}\mu (t-{\tau }_i)\ge 0,\hfill \\ \hfill {\left.\stackrel{C}{t_0}{D}_{t_d}^{q}I\left(t\right)\right|}_{I\left(t\right)=0}& =2\Omega \varphi (t-{\tau }_i)\sqrt{I(t-{\tau }_i)}\mu (t-{\tau }_i)\hfill \\ \hfill & -{\alpha }_1(t-{\tau }_p)I(t-{\tau }_p)\ge 0,\hfill \\ \hfill {\left.\stackrel{C}{t_0}{D}_{t_d}^{q}S\left(t\right)\right|}_{S\left(t\right)=0}& ={\alpha }_1(t-{\tau }_p)I(t-{\tau }_p)-\Lambda S(t-{\tau }_s)\hfill \\ \hfill & +k{\alpha }_2(t-h)V(t-h)e^{-h\xi }\ge 0,\hfill \\ \hfill {\left.\stackrel{C}{t_0}{D}_{t_d}^{q}Q\left(t\right)\right|}_{Q\left(t\right)=0}& ={\alpha }_2\left(t\right)V\left(t\right)-k{\alpha }_2(t-h)V(t-h)e^{-h\xi }\hfill \\ \hfill & -(1-k){\alpha }_2(t-h)V(t-h)e^{-h\xi }\ge 0.\hfill \end{array}$$

(17)

Let $t_0$ be the initial moment and define the vector $\varphi ={(V\left(t_0\right),C\left(t_0\right),I\left(t_0\right),S\left(t_0\right),Q\left(t_0\right))}^T\in {\mathbb{R}}_{10}^{+}$. Then, based on (4),  it follows that the ${\mathbb{R}}_{10}^{+}$ of the control system (1) constitutes a positively invariant set.

Define $Z\left(t\right)=\left(V\right(t),C(t),I(t),S(t),Q(t\left)\right)$, and according to the VCISQ model (1) and the initial conditions (5), it can be shown that:

$$\stackrel{C}{t_0}{D}_t^{q}Z\left(t\right)={\displaystyle \frac{G_1-(1-k)Z}{G_1-(1-k)Z\left(t\right)}}\le 1$$

(18)

The following inequality can be derived from this inequality:

$$\underset{t\to \infty }{lim}supZ\left(t\right)\le {\displaystyle \frac{G_1}{1-k}}$$

(19)

Thus, Theorem 2 has been proven. Therefore, the VCISQ model (1) proposed in this paper is non-negative and bounded, so it has an optimal control pair in the fractional-order optimal control method.    □

3.3. Algorithm Complexity Analysis

Time Complexity

The overall time complexity of MBSAC is dominated by three core operations per training episode E:

• Environment Interaction ($O^2$): Requires $O(N\times T\times t_e)$ per episode, where $N=1$ (single agent), T is episode length, and $t_e$ is per-step interaction time.
• Predictive Model Training ($O^{model}$): Involves $O(E_t\times K_1)$ computations per episode for $E_t$ training iterations over replay buffer $K_1$.
• Network Updates ($O^3$): Demands $O(t_g·T\times U_p)$ per episode, where $t_g$ is the gradient step interval and $U_p$ denotes policy/critic update time.

The aggregate time complexity across E episodes is as follows:

$$O^{time}=O(E\times [T·t_e+E_t·K_1+t_g·T·U_p])$$

(20)

4. Model-Based Soft Actor–Critic

4.1. Soft Actor–Critic

The implementation of MBSAC is based on the SAC algorithm, which is a general reinforcement learning algorithm that usually performs well for continuous state–action space Markov decision processes (MDPs) [4]. It optimizes its strategy to maximize expected returns. As shown in Figure 4, this optimization process involves iterative interaction with the fractional-order VCISQ model (1). The algorithm’s implementation architecture comprises five multilayer perceptron (MLP) networks, each of which has an input layer, hidden layer, and output layer. The policy network $P_{\phi }$, defined by parameters, $\phi$, is responsible for action sampling; the value function, $V_{\phi }$, is computed by two critic networks, $C_{{\theta }_1}$ and $C_{{\theta }_2}$, defined by parameters ${\theta }_1$ and ${\theta }_2$, respectively; and two target critics, $C_{\overline{{\theta }_1}}$ and $C_{\overline{{\theta }_2}}$, defined by parameters $\overline{{\theta }_1}$ and $\overline{{\theta }_2}$, are designed to generate reliable value-target estimates. The core purpose is to overcome the instability caused by frequent updates to the network. Before interacting with VCISQ, the SAC intelligent agent will observe its initial state $s_0$. For the SAC intelligent agent, it is necessary to observe the state of VCISQ $s_t$ (1) at each moment. After the state sequence $s_t$ is obtained, it is input into the policy network to obtain the control action $a_t$. The policy network constitutes the behavior strategy ${\pi }_{\phi }\left(a_t\right|s_t)$, which determines the generation of $a_t$. Specifically, ${\pi }_{\phi }\left(a_t\right|s_t)$ represents the probability that the agent will output action $a_t$ in state $s_t$. As shown in the equation, $a_t$ is quantized by SAC as the control strength of the VCISQ malware:

$$\left\{\begin{array}{c}{s}_t=\{V\left(t\right),C\left(t\right),I\left(t\right),S\left(t\right),Q\left(t\right)\},\hfill \\ a_t=\{{\alpha }_1\left(t\right),{\alpha }_2\left(t\right)\}\hfill \end{array}\right.$$

(21)

The mean ${\mu }_t$ and standard deviation ${\sigma }_t$ are implicitly output as a probability distribution by the MLP policy network after processing the state $s_t$. These two values together determine the action $a_t$. In the VCISQ continuous action space, this sampling mechanism will cause loss of differentiability. However, subsequent gradient updates depend on differentiability. To overcome this defect, reparameterization techniques are necessary [4]:

$$a_t={\delta }_t*e^{log{\sigma }_t}+{\mu }_t.$$

(22)

Among these, the external random source ${\delta }_t$ remains at a constant value at any time t.

As a strategy optimization navigation benchmark, the reward signal $r_t$ is generated synchronously in each interaction. When the agent completes the interaction with the environment, the VCISQ model (1) executes the state transition to $s_{t+1}$. At the same time, the VCISQ model (1) releases the reward signal $r_t$. The agent computes the reward value according to the (7) paradigm at each moment t:

$$r_t=-\left(B_{1}V\left(t\right)+{\left(B_{2}I\left(t\right)\right)}^{\beta }+B_{3}Q\left(t\right)+B_4{\alpha }_1^2\left(t\right)+B_5{\alpha }_2^2\left(t\right)\right)$$

(23)

In the interactive framework with total time T, the SAC optimization objective is to maximize the discounted cumulative reward $E_{{\pi }_{\phi }}\left({\sum }_t^T\gamma r_t\right)$. The discount factor $\gamma$ quantifies the agent’s trade-off between immediate rewards and future benefits. The entropy term $H\left({\pi }_{\phi }(·|s_t)\right)$ is defined as a measure of the strategy’s randomness. It aims to proactively avoid local optimum traps by dynamically expanding the strategy search space to guide the solution towards the global optimum. Specifically, the entropy expression is as follows:

$$\begin{array}{cc}\hfill H\left({\pi }_{\phi }(·|s_t)\right)& =-{\int }_{a_t}{\pi }_{\phi }\left(a_t\right|s_t)log{\pi }_{\phi }\left(a_t\right|s_t)d{a}_t\hfill \\ \hfill & =E_{a_t\sim {\pi }_{\phi }(·|s_t)}\left[-log{\pi }_{{\phi }_i}\left(a_t\right|s_t)\right].\hfill \end{array}$$

(24)

Let l be the entropy regularization coefficient, whose online calibration ensures continuous exploration capability. In each interaction, the intelligent agent receives a real trajectory, which is defined as follows: ${\tau }_1=(o_{i_t},a_{i_t},o_{i_{t+1}},r_t)$ that have the core features of the VCISQ model (1). All of these true trajectories are stored in the true replay buffer $R_1$ to be used for network updates. During the update stage, the true data provides the most important guidance. Throughout the entire interaction stage, the SAC algorithm learns the optimal policy ${\pi }_{\phi }^{*}$ via a trial-and-error reinforcement learning paradigm:

$${\pi }_{\phi }^{*}=\underset{{\pi }_{\phi }}{\underbrace{argmax}}\left[J\left({\pi }_{\phi }\right)\right]=\underset{{\pi }_{\phi }}{\underbrace{argmax}}\left[\sum _t^T{E}_{{\pi }_{\phi }}[\gamma r_t+lH\left({\pi }_{\phi }(·|s_t)\right)]\right]$$

(25)

Equation explanation:

(1) $\gamma r_t$: Discounted immediate reward at time t, where $\gamma \in \left[0,1\right]$ is the discount factor. This prioritizes short-term rewards due to malware’s instantaneous propagation in WRSNs (Section 1).

(2) $H\left({\pi }_{\phi }\left(·\mid s_t\right)\right)$: Policy entropy defined in (23), measuring action distribution randomness.

(3) l: Adaptively tuned entropy coefficient (Algorithm 1, Line 19; updated via (34)) to balance exploration (high entropy) and exploitation (low entropy).

Algorithm 1 Model-Based Soft Actor–Critic
1:	Initialize the parameters of the policy network $\phi$, the predictive model p, the critic networks ${\theta }_1$ and ${\theta }_2$,
	the target critic networks $\overline{{\theta }_1}$ and $\overline{{\theta }_2}$, the entropy regularization coefficient l
	the real replay buffer $K_1$ and the predictive replay buffer$K_2$
2:	for 1 to E do
	Return the initial state $s_0$
3:	for  $t=1$toTdo
4:	Select the action $a_t\sim {\pi }_{\phi }(·|s_t)$ with agent’s policy network
5:	Put $a_{i_t}$ into the VCISQ model (1), and return the next state $s_{t+1}$ and the reward $r_t$
6:	Store real trajectory ${\tau }_1=s_t,a_t,s_{t+1},r_t$ into $K_1$
7:	$s_{t+1}=s_t$
8:	if  $t=truncation$
9:	Return all trajectory data of $K_1$, and output the vector $x_1$ and $y_2$ with the predictive model (28)
10:	Train the predictive model parameter of p with $x_1$ and $y_{tr}$ (29)
11:	for 1 to $P_k$ do
12:	Randomly sampling state from $K_1$ to become the model state $o_t^m$
13:	Output the model action $a_t^m\sim {\pi }_{\phi }(·|s_t^m)$ with agent’s policy network
14:	Output the predictive vector $x_2$ with $s_t^m$ and $a_t^m$ and
	calculate the next model state $s_{t+1}^m$ and the model reward $r_t^m$ (31)
15:	Store Predictive trajectory ${\tau }_2=s_t^m,a_t^m,s_{t+1}^m,r_t^m$ into $K_2$
16:	$o_t^m=o_{t+1}^m$
17:	end for
18:	end for
19:	for each gradient step do
20:	Randomly sample ${\tau }_1$ and ${\tau }_2$ from $K_1$ and $K_2$ respectively
21:	Combine the two datasets in the specified $s_l$ ratio to facilitate update.
22:	Update all network parameters
	$\phi \leftarrow \phi -a_{\phi }{\nabla }_{\phi }L\left(\phi \right)$ (32)
	${\theta }_j\leftarrow {\theta }_j-a_c{\nabla }_{{\theta }_j}{L}_{\mathrm{critic}}$for $j\in \{1,2\}$ (33)
	${\overline{\theta }}_j\leftarrow {\tau }^{\prime }{\theta }_j+{\overline{\theta }}_j-{\tau }^{\prime }{\overline{\theta }}_j$for $j\in \{1,2\}$ (34)
	Update the entropy regularization coefficient parameter $L\left(l\right)$ (35)
23:	end for
24:	end for

Optimization Goal:

Maximizes the trade-off between cumulative rewards and policy stochasticity:

(1) The $\sum \gamma rt$ term ensures effective malware containment (Section 5.4 validates reduced control cost).

(2) The $lH\left(·\right)$ term prevents premature convergence (e.g., avoiding overly deterministic quarantine policies), crucial for dynamic WRSNs.

4.2. Predictive Model

As shown in Figure 4, MBSAC quickly locates the optimal strategy for the VCISQ model (1), which depends on the explicit MLP predictive model G. This predictive model G uses the idea of the MPC method. First, it initializes an adjustable network parameter p and then performs trajectory prediction to support the optimization process of the policy network. MBSAC prioritizes the VCISQ model (1), and then interacts with it for a certain amount of time to obtain a certain amount of real data. MBSAC then truncates the real trajectory to complete the predictive model’s trajectory prediction. Specifically, the real data contains the model’s characteristics. Predicting based on real data can make the model’s parameters more accurately reflect the real situation, thereby improving the accuracy of the predictive model. The core task of the predictive model is to accurately describe the dynamic evolution of the VCISQ model (1) under state–action pairs, and simultaneously, it should also fit the environment’s instantaneous reward generation logic. When the pre-set truncation moment arrives, the model will read all the training data accumulated in the real replay buffer $R_1$ as a learning basis. At each moment t, $o_t$ and $a_t$ are concatenated to form the state–action vector $x_1$:

$$x_1=(s_t,a_t)$$

(26)

Additionally, the reward $r_t$ and the state difference $s_{t+1}-s_t$ are concatenated to form the reward-state difference vector $y_1$:

$$y_1=(r_t,s_{t+1}-s_t)$$

(27)

The training data for the predictive model consists of the input $\lambda x_1$ and the corresponding target output $y_1$. To prevent order bias and improve distribution uniformity, the training data is shuffled randomly at the start of the model training process. The task of model G is to predict the state at the next moment and the instant reward obtained at that moment. To achieve this prediction, model G uses a Gaussian distribution in the output layer of the MLP to represent the uncertainty of the prediction. Specifically, the mean of the predicted state, ${\mu }_{x_1}$, and the log of the variance, $log\left({\sigma }_{x_1}^2\right)$, will be generated. The mean represents the core expectation of the predicted value, and the log variance describes the dispersion of the prediction:

$$G=\mathcal{N}({\mu }_{x_1},log\left({\sigma }_{x_1}^2\right))$$

(28)

Among them, the input dimension of the model is the same as the state–action vector $x_1$. The output dimension is $2*(s_t,a_t)$.

The state–action vector $x_1$(26) is divided into a training set $x_{t_1}$ (with a proportion of ($1-K_R$) and the validation set $x_{v_1}$ (with a proportion of $K_R$). The training set is used to perform $E_t$ model iterations. In each training round, MBSAC randomly samples a training data subset of size $B_c$. This batch of data is then input into the predictive model. (28), which outputs its predicted training mean ${\mu }_{t{x}_1}$ and training log-likelihood $log\left({\sigma }_{t{x}_1}^2\right)$. The log-likelihood $log\left({\sigma }_{t{x}_1}^2\right)$ is then truncated between the pre-set minimum $V_{min}$ and maximum $V_{max}$. In addition, the validation set is used to evaluate the performance of the predictive model G in each training iteration, with the aim of deciding whether to terminate the training early.

The test set input vector $x_{v_1}$ is input into the predictive model (28) to obtain ${\mu }_{v{x}_1}$ and $log\left({\sigma }_{v{x}_1}^2\right)$. This is then combined with $y_1$ to calculate the loss value.

Based on the above, the training set loss function $L_t\left(p\right)$ and the test set loss function $L_v\left(p\right)$ are constructed to optimize the MLP parameters of the predictive model:

$$\left\{\begin{array}{c}{L}_t\left(p\right)=h_m{V}_{max}-h_m{V}_{min}+{\displaystyle \frac{{({\mu }_{t{x}_1}-y_1)}^2+log\left({\sigma }_{t{x}_1}^2\right)*{({\mu }_{t{x}_1}-y_1)}^2}{{\sigma }_{t{x}_1}^2}},\hfill \\ L_v\left(p\right)={({\mu }_{v{x}_1}-y_1)}^2\hfill \end{array}\right.$$

(29)

Among these, $h_m$ represents the model loss weight. The calculation result of the loss function $L_v\left(p\right)$ is only used to monitor the increase in model performance P, and its gradient does not participate in the process of updating the parameters. If the performance improvement ratio P is less than the pre-set target value $P_{target}$, the training process is terminated early to avoid overfitting of the model. $L_v(p,k_v)$ is defined as the loss value calculated on the validation set at the end of the $k_v$ training iteration using Formula (29). The following expression is used:

$$d_r=1-{\displaystyle \frac{L_v(p,k_v)}{L_v(p,k_v+1)}}$$

(30)

Once the predictive model has completed its training, MBSAC will use the predictive model based on the MPC idea to predict the trajectory and generate the prediction data. As shown in Figure 4, from the truncation moment onward, the intelligent agent samples $B_d$ batches of states from the real experience cache $R_1$ to become the model state $s_t^m$. The predictive model then uses the behavior strategy ${\pi }_{\phi }\left(a_t^m\right|s_t^m)$ to obtain the model action $a_t^m$. Next, the prediction input vector $x_2=(s_t^m,a_t^m)$ is constructed. $x_2$ is then input into the predictive model (28) to obtain the predicted mean ${\mu }_{x_2}^2$ and the predicted log-likelihood $log\left({\sigma }_{x_2}^2\right)$. The random sample is defined as $\mathcal{N}(0,1)$ representing the standard normal distribution, and c represents the state space dimension of the VCISQ model (1). Specifically, the predictive model uses the following formula to obtain the next moment model state $s_{t+1}^m$ and the model reward $r_t^m$:

$$\left\{\begin{array}{c}{s}_{t+1}^m={\left({\mu }_{x_2}^2+\mathcal{N}(0,1)·{\sigma }_{x_2}^2\right)}_{1\sim c-1},\hfill \\ r_t^m={\left({\mu }_{x_2}^2+\mathcal{N}(0,1)·{\sigma }_{x_2}^2\right)}_c.\hfill \end{array}\right.$$

(31)

Among them, the subscripts 1 to c represent the sequence $\left({\mu }_{x_2}^2+\mathcal{N}(0,1)·{\sigma }_{x_2}^2\right)$ from the first to the c-1th element, and the subscript c represents the cth element.

It should be noted that the prediction process is repeatable. MBSAC requires $P_k$ predictions to be completed after each truncation. Once a prediction is complete, $s_{t+1}^m$ is used to generate the prediction input vector $x_2$ as the current state for the next prediction. Once the prediction process is complete, $P_k$ prediction trajectories are produced: ${\tau }_2=(s_t^m,a_t^m,s_{t+1}^m,r_t^m)$. These serve as virtual data and are saved in the virtual replay cache $R_2$.

4.3. Network Optimization

MBSAC improves learning efficiency and reduces dependence on large amounts of real-world interaction data. Its core relies on the synchronous updating of real and predictive data. In fact, the predictive mechanism of the predictive model effectively reduces the number of times RL algorithms interact directly with the real environment. The core value of real data lies in its precise depiction of environmental patterns, providing a benchmark for the learning direction of an intelligent agent. At the same time, real data provides a safe space for trial and error for the model-generated virtual data. However, overreliance on real data can hinder the intelligent agent’s ability to fully exploit the predictive potential of the model. At the same time, excessive reliance on virtual data can be misleading.

It is necessary to set an appropriate ratio to balance the use of the two types of data. This is key to the intelligent agent achieving the optimal strategy in a complex environment. If the utilization rate of the real data is $s_l$, then the utilization rate of the virtual data is $(1-s_l)$. For the MBSAC intelligent agent, the policy network is updated using the following loss function $L\left(\phi \right)$:

$$\begin{array}{cc}\hfill L\left(\phi \right)& =s_l*E_{(a_t,s_t)\sim R_1}\left[D_{\mathrm{KL}}\left({\pi }_{\phi }\left(a_t\right|s_t)\parallel {\displaystyle \frac{exp\left(C_{{\theta }_j}(s_t,a_t)/l\right)}{Z\left(s_t\right)}}\right)\right]\hfill \\ \hfill & +(1-s_l)*E_{(a_t^m,s_t^m)\sim R_2}\left[D_{\mathrm{KL}}\left({\pi }_{\phi }\left(a_t^m\right|s_t^m)\parallel {\displaystyle \frac{exp\left(C_{{\theta }_j}(s_t^m,a_t^m)/\alpha \right)}{Z\left(s_t^m\right)}}\right)\right]\hfill \\ \hfill & =s_l*E_{(a_t,s_t)\sim R_1}\left(-\alpha H\left({\pi }_{\phi }\left(a_t\right|s_t)\right)-\left(\underset{j=1,2}{min}{C}_{{\theta }_j}(s_t,a_t)\right)\right)\hfill \\ \hfill & +(1-s_l)*E_{(a_t^m,s_t^m)\sim D_2}\left(-\alpha H\left({\pi }_{\phi }\left(a_t^m\right|s_t^m)\right)-\left(\underset{j=1,2}{min}{C}_{{\theta }_j}(s_t^m,a_t^m)\right)\right)\hfill \end{array}$$

(32)

This equation updates the policy network $\left({\pi }_{\phi }\right)$ by minimizing the Kullback–Leibler (KL) divergence between the current policy and a target distribution derived from the critic’s evaluation. Critically, it computes this loss using a weighted sum of transitions sampled from both the real replay buffer ($R_1$) and the predictive replay buffer ($R_2$). The hyperparameter $s_l$ balances this data usage, enabling the policy to learn from accurate real interactions while greatly improving sample efficiency with model-generated data.

For any critic network $C_{{\theta }_m}$, where $m=1,2$, the loss function $L_{\mathrm{critic}}$ is given by the following:

$$\begin{array}{cc}\hfill L_{\mathrm{critic}}& =s_l*E_{{\tau }_1}\hfill \\ & \left[{\displaystyle \frac{1}{2}}{\left(C_{{\theta }_m}(s_t,a_t)-\left(r_t+\gamma \left(\underset{j=1,2}{min}{C}_{\overline{{\theta }_j}}(s_{t+1},a_{t+1})-\alpha log{\pi }_{\phi }\left(a_{t+1}\right|s_{t+1})\right)\right)\right)}^2\right]\hfill \\ \hfill & +(1-s_l)*E_{{\tau }_2}\hfill \\ & \left[{\displaystyle \frac{1}{2}}{\left(C_{{\theta }_m}(s_t^m,a_t^m)-\left(r_t^m+\gamma \left(\underset{j=1,2}{min}{C}_{\overline{{\theta }_j}}(o_{t+1}^m,a_{t+1}^m)-\alpha log{\pi }_{\phi }\left(a_{t+1}^m\right|o_{t+1}^m)\right)\right)\right)}^2\right].\hfill \end{array}$$

(33)

This equation updates the critic networks $\left(C_{\theta }\right)$ by minimizing the mean squared error (MSE) of the temporal-difference (TD) error. It calculates the TD target using a soft value estimate from the target networks that includes an entropy bonus. Similar to Equation (32), the final loss is a weighted average of the TD error computed on batches from both the real ($R_1$) and predictive $\left(R_2\right)$ buffers. This allows the critic to learn a more accurate and stable value function from a larger and more diverse set of experiences.

The convergence stability of the loss function (33) depends on the soft update mechanism of the target critic network. Specifically, MBSAC introduces parameters $\overline{{\theta }_1}$ and $\overline{{\theta }_2}$ that are updated using a sliding average based on the scaling factor ${\tau }^{\prime }$:

$$\begin{array}{cc}\hfill \overline{{\theta }_1}& ={\tau }^{\prime }{\theta }_1+\overline{{\theta }_1}-{\tau }^{\prime }\overline{{\theta }_1},\hfill \\ \hfill \overline{{\theta }_2}& ={\tau }^{\prime }{\theta }_2+\overline{{\theta }_2}-{\tau }^{\prime }\overline{{\theta }_2}.\hfill \end{array}$$

(34)

This is not a loss function but a stability mechanism. It slowly synchronizes the parameters of the target critic networks with the parameters of the online critics. This soft update, as opposed to a periodic hard copy, ensures the target values change gradually, which is crucial for preventing divergence and stabilizing the training of the critics in Equation (33).

To maintain the exploratory nature of MBSAC, the entropy value network has the following loss function for real-time adjustment of the regularization coefficient:

$$\begin{array}{c}\hfill L\left(l\right)=s_l*E_{(a_t,s_t)\sim R_1}\left(-l·log\left({\pi }_{\phi }\left(a_t\right|s_t)\right)-l·l^{\prime }\right)\\ \hfill +(1-s_l)*E_{(a_t^m,s_t^m)\sim R_2}\left(-l·log\left({\pi }_{\phi }\left(a_t^m\right|s_t^m)\right)-l·l^{\prime }\right).\end{array}$$

(35)

Entropy coefficient loss function: This equation automates the adjustment of the entropy regularization coefficient. It performs a gradient descent step to minimize the difference between the policy’s current average entropy and a target entropy value. This dynamic adjustment is essential for maintaining a sustainable exploration-exploitation balance throughout the training process, allowing the agent to explore effectively without degrading final performance.

The Adam optimizer and the ReLU activation function are applied uniformly to the design of all RL algorithms’ neural networks. The details of the execution framework are fully presented in Algorithm 1. Additionally, the parameters and values of the MBSAC algorithm are provided in

Table 3. Reinforcement learning algorithm parameters.

	Parameter	Description	Value
RL Algorithms	$l^{\prime }$	The target entropy	$-2$
	E	The total number of running rounds	100
	T	The total interaction time	30
	$a_{\phi }$	The learning rate of the policy network	$2\times {10}^{-3}$
	$a_{\theta }$	The learning rate of the critic and the target critic network	$2\times {10}^{-4}$
	$a_p$	The learning rate of the predictive model	$1\times {10}^{-3}$
	$a_l$	The learning rate of entropy	$3\times {10}^{-3}$
	$\gamma$	The discount factor	$0.85$
	${\tau }^{\prime }$	The value of soft update	$4\times {10}^{-3}$
	$h_m$	The loss weight of the predictive model	$0.02$
	$K_R$	The proportion of the validation set for predictive model training	$0.5$
	$P_k$	The number of predictive rounds after each truncation	5
	$V_{max}$	The maximum of logarithmic variance	$0.5$
	$V_{min}$	The minimum of logarithmic variance	$-10$
	$K_1$	The size of the real replay buffer	8000
	$K_2$	The size of the predictive replay buffer	2000
	$\mathrm{Batch}\mathrm{size}$	The number of samples taken by the agent during each training of the network	128
	$E_t$	The maximum number of iterations for training predictive model	15
	$s_l$	Real data utilization rate of network updates	$0.9$

.

5. Experiment Validation

5.1. Experimental Setup

In order to validate the fractional order optimal control with the proposed algorithm MBSAC, experiments are designed in this paper in several dimensions. All the experiments in this paper were conducted under Python 2025.1 and Python 3.13.5. It should be noted that the fractional-order optimal control method provides the theoretically optimal solution for model control. The closer the final control cost of RL is to the optimal control cost produced by the fractional-order optimal control method, the better the RL performance. The RL methods for each condition were averaged over 10 runs with different random seeds. The experimental parameters are configured according to Table 2 and Table 3 if not specified.

Specifically, this paper has the following comparison baseline:

(1)

DQN [27]: A generalized reinforcement learning algorithm based on value functions.

(2)

SAC [4]: A generalized maximum entropy-based reinforcement learning algorithm that performs well on model species in continuous space.

(3)

PPO [5]: A generalized reinforcement learning algorithm based on importance sampling and trust domain.

(4)

MAPPO [5]: A multi-agent reinforcement learning algorithm based on PPO, in which agents can collaborate to optimize.

(5)

Fractional-order optimal control method: The control method proposed in this paper for the VCISQ model, which aims to output an optimal result.

(6)

MBSAC: A high sample efficiency MBRL algorithm proposed in this paper with higher convergence efficiency and learning performance.

To ensure the reproducibility and transparency of our experimental results, this subsection provides a detailed explanation of the configuration parameters, which are categorized into two groups, as follows: (1) the environment and model parameters that define the WRSNs and malware propagation dynamics, and (2) the algorithm hyperparameters that control the learning process of the reinforcement learning agents.

5.1.1. Environment and Model Parameters (Table 2)

The parameters for the fractional-order VCISQ model were carefully selected to simulate a realistic and challenging WRSNs scenario while ensuring the mathematical stability of the model.

Initial states $\left(G_1-G_5\right)$: The initial population was set to 300 nodes in total. We initialized with 30 infected nodes $\left(G_3=30\right)$ and 270 susceptible nodes $\left(G_1=270\right)$ to simulate an ongoing malware outbreak, a critical scenario for testing the containment (policy). Other states were set to zero.

Cost Weights $\left(B_1-B_5\right)$: The weights in the cost function (7) were set to prioritize the reduction of infected nodes $\left(B_2=\right.$ 900) as they pose the most immediate threat. The weights for susceptible and quarantined nodes $\left(B_1,B_3=300\right)$ were set lower but significant to discourage excessive quarantine that cripples network functionality. The control weights $\left(B_4,B_5=12\right)$ were tuned to reflect the non-negligible resource cost of applying security measures, preventing the algorithm from finding a trivial solution that applies maximum control at all times.

Delay parameters $\left({\tau }_i,{\tau }_p,{\tau }_s,h\right)$: These temporal parameters are crucial for realism. The malware installation delay $\left({\tau }_i=1\right)$ and patch delay $\left({\tau }_p=0.5\right)$ are shorter than the patch failure delay $\left({\tau }_s=2\right)$ and quarantine failure delay $\left(h=1\right)$, reflecting the fact that remediation efforts often take longer to complete than the time malware needs to activate.

Communication parameters $\left(q_c,\chi ,\alpha ,{\sigma }_n^2,{\gamma }_{th},\rho \right)$: These values were adopted from our previous work [5] and standard models for wireless sensor networks to simulate a realistic communication channel based on Rayleigh fading, path loss, and signal-to-noise ratio (SNR) constraints. The node density $\left(\rho =0.6\right)$ models a moderately dense deployment.

Fractional order $\left(q=0.9\right)$: This value was chosen as the primary order after a sensitivity analysis (Section 5.3). It provides a strong memory effect while keeping the model computationally tractable. Values of $q=0.8$ and $q=0.7$ were also tested to study the impact of memory length.

5.1.2. Algorithm Hyperparameters (Table 3)

The hyperparameters for the MBSAC and baseline algorithms were tuned through an extensive grid search to ensure optimal and fair performance. The Adam optimizer was used for all neural network updates.

Learning rates $\left(a_{\phi },a_{\theta },a_p,a_l\right)$: The policy network learning rate $\left(a_{\phi }=2\times {10}^{-3}\right)$ was set higher than the critic learning rate $\left(a_{\theta }=2\times {10}^{-4}\right)$ to ensure stable policy updates relative to the value estimates, a common practice in actor–critic methods. The predictive model learning rate ( $a_p=1\times {10}^{-3}$ ) was chosen for rapid fitting to the recently collected data.

Discount factor and soft update $\left(\gamma =0.85,\tau =4\times {10}^{-3}\right)$: A discount factor of $\gamma =0.85$ was chosen to balance the importance of immediate rewards against future outcomes, which is suitable for a finite-horizon control problem. The target network update factor $\tau$ is very small to ensure slow and stable tracking of the critic networks, which is critical for convergence.

Buffer sizes and batch sampling $\left(K_1=8000,K_2=2000\right.$, Batch size $=128$ ): The real replay buffer $\left(K_1\right)$ is larger to maintain the rich history of interactions. The predictive buffer $\left(K_2\right)$ is smaller as it contains fresher, model-generated data. A batch size of 128 provides a good compromise between stable gradient estimates and computational efficiency.

Predictive model parameters $\left(P_k=5,s_l=0.9\right)$: The agent performs $P_k=5$ prediction steps after each truncation to generate sufficient synthetic data without compounding excessive model error. The real data utilization ratio ${\epsilon }_l=0.9$ indicates that the update is primarily guided by real data (90%), with 10% coming from the predictive model, ensuring learning stability while boosting sample efficiency.

5.2. Validation of Control Measure Validity

In order to prove that the treatment and quarantine measures exert the necessary effects in the VCISQ model, this experiment was conducted by implementing an ablation experiment on the control volume. In Figure 5a–e, three different states of VCISQ models with only treatment measures ${\alpha }_1$, only quarantine measures ${\alpha }_2$, and mixed controls are analyzed. In the VCISQ model, the control objective is to minimize the amount of malware. This means that we expect to have a higher number of $V\left(t\right)$ and $S\left(t\right)$ and a lower number of $I\left(t\right)$ and $C\left(t\right)$.

In Figure 5a–e, it can be seen that when only the treatment measure ${\alpha }_1$ is available, the infected node $I\left(t\right)$ is not quarantined in time, so the decline rate of $V\left(t\right)$ is not suppressed. Meanwhile, a large number of WRSNs carrier nodes $C\left(t\right)$ still exist at the final moment of the system. When only the quarantine measure ${\alpha }_2$ is available, the infected node $I\left(t\right)$ is not treated in time, so the rising rate of $S\left(t\right)$ is not enhanced. At the same time, the secured node $S\left(t\right)$ is transformed into a carrier node $C\left(t\right)$ by the high concentration of infected nodes $I\left(t\right)$. It is noteworthy that $V\left(t\right)$ and $S\left(t\right)$ are able to retain more WRSN nodes under the hybrid control. This is due to the fact that the two control strategies complement each other under the hybrid control measure. Timely quarantine of the infected node $I\left(t\right)$, along with timely treatment, effectively reduces the number of WRSNs carrying the nodes $C\left(t\right)$ and $I\left(t\right)$.

Table 4. Performance comparison of control methods.

	Parameter	Control
		Optimal Control	DQN	SAC	PPO	MAPPO	MBSAC
Validation of control effectiveness	Only treatment measure ${\alpha }_1$	8634.91	–	–	–	–	–
	Only quarantine measure ${\alpha }_2$	10,347.14	–	–	–	–	–
	Hybrid control measure	7143.22	–	–	–	–	–
The influence of different fractional orders q on the VCISQ model	$q=0.9$	7143.22	–	–	–	–	–
	$q=0.8$	7265.18	–	–	–	–	–
	$q=0.7$	7415.57	–	–	–	–	–
Performance of RL under different hyperparameters	$a_{\phi }=1\times {10}^{-3}$	7143.22	8845.16	8547.18	7965.44	7763.58	7352.41
	$a_{\phi }=2\times {10}^{-3}$	7143.22	8753.24	8125.71	7847.51	7632.24	7266.29
	$a_{\phi }=3\times {10}^{-3}$	7143.22	8949.33	8219.52	8088.93	7607.16	7448.52
	$a_{\theta }=1\times {10}^{-4}$	7143.22	8964.21	8487.31	8004.55	7721.63	7511.36
	$a_{\theta }=2\times {10}^{-4}$	7143.22	8753.24	8125.71	7847.51	7574.18	7266.29
	$a_{\theta }=3\times {10}^{-4}$	7143.22	8865.63	8192.64	7935.82	7513.55	7373.21
	Batch size = 64	7143.22	8931.36	8419.69	7948.06	7832.29	7428.87
	Batch size = 128	7143.22	8753.24	8125.71	7847.51	7683.77	7266.29
	Batch size = 256	7143.22	8799.22	8349.90	8056.99	7686.73	7298.74
	$\gamma =0.95$	7143.22	8931.37	8257.25	8189.18	7568.24	7374.52
	$\gamma =0.85$	7143.22	8753.24	8125.71	7847.51	7469.56	7266.29
	$\gamma =0.75$	7143.22	9023.86	8564.18	8031.89	7711.23	7435.96
Investigation of relative optimal $s_l$ under different fractional orders q	$q=0.9$, $s_l=0.9$	7143.22	–	–	–	–	7266.29
	$q=0.9$, $s_l=0.8$	7143.22	–	–	–	–	7285.64
	$q=0.9$, $s_l=0.7$	7143.22	–	–	–	–	7377.86
	$q=0.8$, $s_l=0.9$	7265.18	–	–	–	–	7431.72
	$q=0.8$, $s_l=0.8$	7265.18	–	–	–	–	7338.14
	$q=0.8$, $s_l=0.7$	7265.18	–	–	–	–	7536.87
	$q=0.7$, $s_l=0.9$	7415.57	–	–	–	–	7623.66
	$q=0.7$, $s_l=0.8$	7415.57	–	–	–	–	7682.57
	$q=0.7$, $s_l=0.7$	7415.57	–	–	–	–	7538.14

shows that the final cost of the fractional-order optimal control method is 8634.91 under the treatment-only measure ${\alpha }_1$. The final cost of the fractional-order optimal control method is 10,347.14 under the control-only measure ${\alpha }_2$. Under the hybrid control measure, the final cost of the fractional-order optimal control method is 7143.22. Therefore, the cost of the treatment-only measure is $17.28\%$ higher than that of the hybrid control. The hybrid control is $17.28\%$ higher. The control cost under quarantine only is $44.85\%$ higher than the hybrid control.

From Figure 5f,g, it can be seen that when only the treatment measure ${\alpha }_1$ is available, the treatment measure outputs a higher treatment intensity compared to the mixed control. This is because the VCISQ model loses the assistance of the quarantine measure. The VCISQ model can only rely more on the treatment measure to stop the generation of a large number of infected nodes $I\left(t\right)$. When there are only quarantine measures, the quarantine measures output a higher quarantine effort compared to the quarantine measures under hybrid control. At this point, the VCISQ model does not have access to the treatment to convert $S\left(t\right)$ to $I\left(t\right)$. Therefore, the VCISQ model can only output a large number of quarantine measures to transform $I\left(t\right)$ into the quarantine node $Q\left(t\right)$ to prevent further malware propagation.

5.3. Impact on VCISQ Model with Different Fractional Orders q

The purpose of this experiment is to investigate the effect of different fractional orders of q on the malware propagation dynamics in the VCISQ model. Fractional order q is chosen as the research variable because in real WRSNs, malware propagation is characterized by complexity, time-varying, and memory. And the fractional order differential equation can more accurately portray this propagation phenomenon with genetic characteristics and cumulative effects.

The curves in Figure 6a–e show the state transitions of the VCISQ model under different q values. For the susceptible node $V\left(t\right)$ curve, it can be seen that as the q value decreases from 0.9 to 0.7, the decline of the curve gradually decreases. This indicates that at lower q values, susceptible nodes convert to other states more slowly, and the propagation rate of malware decreases. At the same time, the rise and peak of the infected node $I\left(t\right)$ curve also decrease as the q value decreases. This shows that the infection intensity of the malware is effectively suppressed. In contrast, the carrier node $C\left(t\right)$ curve shows a different trend, with its rise and stable value increasing as the q value decreases. This means that at lower q values, more nodes can return to a safe state. In addition, the peak of the quarantine node $Q\left(t\right)$ curve decreases when the q value is small, reflecting a change in the frequency or intensity of quarantine measures. This may be due to the slowdown in the spread rate, which reduces the need for quarantine. When $q=0.9$, the $C\left(t\right)$ and $S\left(t\right)$ curves fluctuate significantly. Among them, the peak of $C\left(t\right)$ is high, and the decline process is relatively slow. The rise rate of $S\left(t\right)$ is relatively fast. However, as q decreases to 0.8 and 0.7, the amplitude of the curve fluctuations decreases significantly. At this point, the peak of $C\left(t\right)$ decreases and the rate of decline accelerates, while the rate of increase of $S\left(t\right)$ slows down, and the system reaches a stable state more quickly. This indicates that a lower fractional order q value enables the system to exhibit a more stable dynamic response when dealing with malware propagation. This reduces the instability caused by severe fluctuations, thereby reaching a relatively balanced state more quickly. This improvement in stability is likely related to the fractional-order model’s consideration of the cumulative effects of historical states. The fractional-order model enables the system to more effectively regulate the transitions between state nodes during propagation. Under such conditions, excessive fluctuations and imbalances are effectively avoided.

As shown in Table 4, when q is 0.9, 0.8, and 0.7, the fractional-order optimal control costs are 7143.22, 7265.18, and 7415.57, respectively. It is evident that as the fractional order q decreases, the control cost shows a gradually increasing trend. This phenomenon is closely related to the curves in Figure 6f,g. A deeper analysis reveals that there are significant differences in the output intensity of control measures under different q values. At lower q values, in order to effectively maintain system stability and control effectiveness, the VCISQ model has to output stronger control intensity, and the system needs to pay a higher cost to deal with the spread of malware in order to achieve better control effectiveness. In contrast, at higher q values, although the control intensity input is reduced, this also leads to insufficient resistance to a large number of infected nodes $I\left(t\right)$. Nevertheless, relatively low control intensity also means reduced resource consumption. From a practical point of view, when $q=0.7$, the VCISQ model shows a more excellent state. At this time, the model not only has fewer infected nodes $I\left(t\right)$ in WRSNs, but can also balance the control effect and resource consumption to a certain extent. This balance makes the model more advantageous in practical applications. Because it can effectively suppress the spread of malware under limited resource conditions. Thus, it improves the security and stability of the entire WRSN system.

5.4. Performance of Reinforcement Learning Under Different Hyperparameters

Hyperparameters are directly related to the performance and application effect of RL algorithms. They directly affect the learning efficiency, convergence speed, and final control performance of the algorithm. In order to better understand and optimize the RL algorithm in this paper, we conducted a detailed analysis of the RL operation results under different hyperparameters.

Figure 7a–l show the variation of average rewards of different reinforcement learning algorithms under various hyperparameters. The curves in Figure 7a–l show that the MBSAC algorithm significantly outperforms the DQN, SAC, PPO, and MAPPO algorithms in terms of learning efficiency and convergence speed. For the policy network learning rate $a_{\phi }$, the average reward value of MBSAC rises rapidly and stabilizes when $a_{\phi }=1\times {10}^{-3}$, while the reward values of DQN, SAC, PPO, and MAPPO rise slowly and fluctuate greatly. Even at $a_{\phi }=2\times {10}^{-3}$ or $a_{\phi }=3\times {10}^{-3}$, MBSAC still maintains a stable upward trend. The fluctuation of the baseline algorithm is still obvious and cannot converge effectively. Obviously, MBSAC at $a_{\phi }=2\times {10}^{-3}$ has the highest reward. For the criticism network and the target criticism network learning rate $a_{\theta }$, MBSAC exhibits fast convergence and high average rewards at different settings of $1\times {10}^{-4}$, $2\times {10}^{-4}$, and $3\times {10}^{-4}$. In contrast, the reward values of DQN, SAC, PPO, and MAPPO rise slower and fluctuate more, making it difficult to reach the stable state of MBSAC. It is not difficult to find that MBSAC has a superior performance at $a_{\theta }=2\times {10}^{-4}$. In Batch size, MBSAC also performs well. When Batch size increases from 64 to 128 and 256, the average reward of MBSAC fluctuates slightly. However, it is still much higher than the other algorithms overall and stabilizes faster at batch sizes of 128 and 256. In particular, the control cost of MBSAC at Batch size = 128 is lower than the other two Batch size cases. Whereas DQN, SAC, PPO, and MAPPO are less effective in learning under various Batch size, and the reward value fluctuates more. The change in the discount factor $\gamma$ also has an impact on the algorithm’s performance. As $\gamma$ decreases from 0.95 to 0.85 and 0.75, the control cost of MBSAC gradually increases. Nevertheless, the decrease is much smaller than that of the baseline algorithm, and it is able to converge stably. However, DQN, SAC, PPO, and MAPPO fluctuate more and are difficult to converge effectively. It is worth noting that the final control cost of MBSAC is lower at $\gamma =0.85$ than at $\gamma =0.95$ and $\gamma =0.75$. The related experimental results are shown in Table 4.

At $a_{\phi }=1\times {10}^{-3}$, the control cost of MBSAC is 7352.41, and the error from the optimal control is $2.92\%$; at $a_{\phi }=2\times {10}^{-3}$, the control cost of MBSAC is 7266.29, and the error is reduced to $1.72\%$; at $a_{\phi }=3\times {10}^{-3}$, the control cost of MBSAC is 7448.52, and the error is $4.27\%$.

When $a_{\theta }=1\times {10}^{-4}$, the control cost of MBSAC is 7511.36, with an error of $5.15\%$; when $a_{\theta }=2\times {10}^{-4}$, the control cost of MBSAC is 7266.29, with an error of $1.72\%$; at $a_{\theta }=3\times {10}^{-4}$, the control cost of MBSAC is 7373.21, with an error of $3.22\%$.

When Batch size = 64, the control cost of MBSAC is 7428.87, with an error of $3.99\%$; when Batch size = 128, the control cost of MBSAC is 7266.29, with an error of $1.72\%$; when Batch size =256, the control cost of MBSAC is 7298.74 with an error of $2.17\%$.

In the case of $\gamma =0.95$, the control cost of MBSAC is 7374.52, with an error of $3.24\%$; in the case of $\gamma =0.85$, the control cost of MBSAC is 7266.29, with an error of $1.72\%$; and in the case of $\gamma =0.75$, the control cost of MBSAC is 7435.96, with an error of $4.09\%$.

In this experiment, the hyperparameter design of the algorithms was not involved; instead, we followed similar experimental ideas for testing and optimization. By fine-tuning the hyperparameters and performing performance evaluations, a set of superior hyperparameter combinations was determined in Table 3. MBSAC utilizes this set of hyperparameter combinations to effectively achieve accurate suppression of malware propagation when executing the VCISQ model problem. It provides an efficient and reliable solution for malware prevention and control in WRSNs.

5.5. Relative Optimization at Different Fractional Orders $q{s}_l$ Investigations

The parameter $s_l$ determines how much MBSAC relies on real and predicted data in the updating process. This experiment aims to deeply investigate the control effect of the MBSAC algorithm proposed in this paper under different fractional orders q and different $s_l$. The performance of the MBSAC algorithm under different conditions is evaluated by comparing it with the optimal control method. The observation of the experimental results is of great significance to improve the control effect of MBSAC.

Figure 8a,d,g show the average reward curves for various $s_l$ cases at different score orders. With the iteration of training episodes, the reward value of the MBSAC algorithm gradually increases and stabilizes. When the score order q is set to 0.9, the convergence of the MBSAC algorithm gradually slows down as the $s_l$ value increases from 0.7 to 0.9. This is reflected in the fact that the slope of the rise in reward values flattens out as the $s_l$ increases, implying that the algorithm requires a greater number of training rounds to reach a steady state. When $s_l=0.7$, the reward value rises rapidly and stabilizes in an earlier number of rounds. Whereas when $s_l=0.9$, the reward values rise more slowly but eventually reach the highest stabilized average reward values. This indicates that the algorithm is able to obtain a better control strategy based on fully utilizing the real data. This may be due to the fact that higher $s_l$ values make the algorithm more dependent on real data, and the intelligences are more cautious in the exploration process. Although the learning process is prolonged, the final performance is better. For the case of $q=0.8$, the experimental results show the most moderate convergence rate at $s_l=0.8$. It is able to achieve a better performance balance. At this point, the algorithm achieves a better balance between utilizing real data and predicting data. It ensures a faster convergence rate without significantly reducing the final reward value. This indicates that at this score order, $s_l=0.8$ is a more ideal choice to effectively utilize both real and predicted data. Ultimately, it improves the control effect while ensuring the learning efficiency. For the case of $q=0.7$, the experimental results show a similar trend. At $s_l=0.8$, the algorithm not only exhibits a faster convergence speed, but also reaches a relatively high average reward value in the end. In contrast, the higher $s_l=0.9$ value leads to a significantly slower convergence rate. The lower $s_l=0.7$ value, while converging faster, ends up with a lower average reward value. At the same time, the volatility of the curve increases slightly. It is worth noting that—at this point—$s_l=0.8$ provides a better balance between performance and convergence speed. It avoids the problem of performance degradation or slow convergence caused by over-reliance on real or predicted data.

The running results of the optimal control and MBSAC under different conditions in this experiment are given in Table 4. Specifically, when the fractional order $q=0.9$:

(1)

$s_l=0.9$: The control cost of MBSAC is 7266.29, compared to 7143.22 for the optimal control, and has an error of $1.72\%$.

(2)

$s_l=0.8$: The cost of control for MBSAC is 7285.64, compared to 7143.22 for the optimal control, which has an error of $2.00\%$.

(3)

$s_l=0.7$: The control cost of MBSAC is 7377.86, which is an error of $3.28\%$ compared to the optimal control of 7143.22.

When fractional order $q=0.8$:

(1)

$s_l=0.9$: The control cost of MBSAC is 7431.72, which is an error of about $2.30\%$ compared to the optimal control of 7265.18.

(2)

$s_l=0.8$: The control cost of 7333.14 for MBSAC is about $1.01\%$ error compared to the optimal control 7265.18.

(3)

$s_l=0.7$: The control cost of MBSAC is 7536.87, which is about $3.74\%$ error compared to the optimal control 7265.18.

When fractional order $q=0.7$:

(1)

$s_l=0.9$: The control cost of MBSAC is 7623.66, which is about $2.81\%$ error compared to the optimal control 7415.57.

(2)

$s_l=0.8$: The control cost of MBSAC is 7682.57, which is about $3.85\%$ error compared to the optimal control 7415.57.

(3)

$s_l=0.7$: The control cost of MBSAC is 7538.14, which is about $1.46\%$ error compared to the optimal control 7415.57.

Combining Figure 8b,c, Figure 8e,f, and Figure 8h,i control curves, the control curves of the algorithm are closer to the optimal control curves when the values of $s_l$ are adapted with the values of q, e.g., $q=0.9$ and $s_l=0.9$, $q=0.8$ and $s_l=0.8$. This indicates that the control strategy at this point is more optimal and the error is smaller. For example, in the case of $q=0.9$ and $s_l=0.9$, the control curve of MBSAC is highly consistent with the fluctuation trend of the optimal control curve. This indicates that the algorithm is able to adjust the control strength in time according to the changes of the system state, thus effectively suppressing the propagation of malware. Similarly, when $q=0.8$ and $s_l=0.8$, the control curve is in high agreement with the optimal control curve. The algorithm can better balance the control effect and resource consumption under this parameter combination. On the contrary, when $s_l$ does not match with q, such as $q=0.7$ and $s_l=0.8$, the control curve deviates more from the optimal control curve. This is manifested in the form of large fluctuations in the control effort and insufficient or excessive control effort in certain time periods. This will lead to the spread of malware that is not effectively controlled, so that the control cost increases. Unreasonable parameter combinations will lead to a mismatch between the control strategy of the algorithm and the actual needs of the system, which in turn affects the control effect.

Further analysis reveals that as the value of q increases, the system’s dependence on the historical state is gradually weakened, and at this time, appropriately increasing the value of $s_l$ can enhance the algorithm’s efficiency in utilizing the real data, thus improving the control accuracy. For example, when q increases from 0.7 to 0.9, the value of $s_l$ should be increased accordingly in order to maintain a better control effect. Specifically, when $q=0.7$, the error of $s_l=0.7$ is minimized; while when q increases to 0.8 and 0.9, the error of $s_l$ reaches the minimum when the value is increased to 0.8 and 0.9, respectively. This shows that when the value of q is high, increasing the value of $s_l$ can better utilize the performance of the algorithm and make it closer to the optimal control effect.

6. Conclusions

We propose a fractional-order VCISQ model for accurate malware dynamics in WRSNs and derive its theoretical optimal control. To address RL inefficiency, we develop MBSAC, a model-based reinforcement learning algorithm. MBSAC leverages a predictive network to forecast environmental changes from existing data and co-optimizes the agent using both real and predicted data, significantly improving learning efficiency, reducing control costs, and accelerating convergence.

MBSAC’s effectiveness may decline in highly dynamic environments due to its reliance on stable environmental characteristics and prior knowledge. Future research will explore Meta-Reinforcement Learning (Meta-RL) to enable rapid adaptation to new dynamics using minimal data via meta-learning, enhancing the generalization capability and robustness of malware control strategies in WRSNs.