Deep Reinforcement Learning-Based Dual-Loop Adaptive Control Method and Simulation for Loitering Munition Fuze

Abstract

To address the poor adaptability and rigid initiation modes of the loitering munition fuze in complex environments and the inadequacy of single fuzzy control against strong interference, this paper proposes a dual-loop adaptive reconfiguration control method. The architecture integrates the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm with fuzzy logic. The inner loop uses TD3 to dynamically optimize fuzzy scaling factors based on real-time interference and state deviations. Concurrently, the outer loop utilizes a Fuze Readiness Index (FRI) and a finite state machine to manage real-time multi-modal mission switching (e.g., proximity, delay, and airburst) and reverse safety-state conversions. Co-simulations under non-stationary composite interference show that the proposed method reduces the burst height RMSE by 82.4% and 61.6% compared with the fixed-threshold and standard fuzzy baselines under the considered non-stationary composite interference setting, respectively. The false alarm rate (FAR) is reduced to 0.15%, and the reconfiguration response time under sudden interference is shortened to 12 ms. Even under extreme conditions, such as 400 ms sensor signal loss, the relative error remains within 5%. These simulation results demonstrate the potential of the proposed architecture to improve precision, responsiveness, and robustness under dynamic interference conditions and show good robustness to intermittent observation loss within the simulated operating envelope.

1. Introduction

Loitering munitions combine the mobility of unmanned aerial vehicles with the strike capability of precision-guided ammunition. As the core component responsible for timing control, the fuze must respond reliably to target variation, mission switching, and electromagnetic interference in complex battlefield environments. However, conventional fuze control strategies are usually based on pre-mission parameter setting and fixed initiation logic, making it difficult to adapt online once the engagement condition changes during flight. Under strong interference or sudden target variation, such rigid configurations may lead to delayed response, false triggering, or deviation from the optimal burst point, thereby reducing the operational effectiveness of loitering munitions in dynamic combat scenarios. In addition, unlike many traditional ammunition fuzes, the loitering munition fuze may require not only forward arming but also reverse transition to a safer state when the target is lost, the mission is aborted, or the initiation condition is no longer satisfied. This recoverable safety characteristic further increases the complexity of fuze control design [1,2,3].

Existing studies have improved fuze performance mainly from two directions. The first direction is hardware-oriented enhancement, including multi-modal sensing, multi-source information fusion, and the integration of proximity, impact, and delay-related functions into a single fuze architecture [4,5,6,7]. The second direction is control and signal processing enhancement, such as anti-jamming signal processing, filtering, fixed-threshold decision strategies, and fuzzy logic-based control methods, which are used to improve robustness against uncertainty and noise [8,9,10]. Although these approaches have practical value, they still face clear limitations in highly dynamic and non-stationary interference environments. In particular, many existing methods rely on preset thresholds, manually tuned fuzzy parameters, or offline configuration and therefore lack sufficient online adaptability when the target state, interference intensity, or tactical intent changes rapidly during engagement.

To overcome this limitation, it is necessary to move from static configuration toward online adaptive parameter reconfiguration. Reconfigurability has been widely studied in electronic and control systems, but in fuze applications, it cannot be treated as a generic functional switching problem only. Instead, it must be tightly coupled with fuze-specific requirements, including safety constraints, timing sensitivity, mode switching, and limited onboard computing resources [11]. Therefore, a practical reconfiguration strategy for the loitering munition fuze should not only support flexible parameter adjustment but also maintain strict safety logic and low-latency execution under interference.

Fuzzy logic control is attractive for fuze applications because of its interpretability and robustness in nonlinear and uncertain environments [12]. Nevertheless, its performance strongly depends on the choice of scaling factors, which are often fixed or manually tuned. When the battlefield environment becomes non-stationary, such static settings may no longer provide a satisfactory trade-off between sensitivity and robustness. Deep reinforcement learning offers a possible way to address this problem by learning adaptive parameter-tuning policies directly from interaction data. Among commonly used methods, meta-heuristic algorithms such as GA, PSO, and ACO are effective for offline optimization but are difficult to use for millisecond-level online control [13,14,15]. DQN is not well suited to continuous parameter adjustment [16], while DDPG may suffer from value overestimation and unstable learning in noisy environments [17,18]. By contrast, TD3 provides a more suitable framework for low-dimensional continuous control because clipped double-Q learning, delayed policy updates, and target policy smoothing can improve learning stability and suppress action jitter [19]. In comparison, SAC is also a strong candidate for continuous control, but its stochastic policy mechanism is not the focus of the present low-dimensional deterministic online tuning problem [20].

It should be noted that hybrid TD3–fuzzy or reinforcement learning-assisted fuzzy controllers have already been explored in other control domains [21]. Therefore, the contribution of this work is not simply the generic combination of TD3 and fuzzy logic. Instead, the novelty lies in a fuze-oriented integration strategy: an outer loop is introduced to handle initiation-mode switching and safety rollback, while an inner loop performs online tuning of fuzzy scaling factors under interference. In this way, the proposed method connects continuous parameter adaptation with discrete fuze mission logic and recoverable safety requirements, which are central issues in the loitering munition fuze control. In addition, from the perspective of safety-constrained decision making, the present problem is closely related to safe reinforcement learning, where control performance must be balanced against explicit safety requirements [22,23].

Based on the above considerations, this paper proposes a deep reinforcement learning-based dual-loop adaptive control method for the loitering munition fuze. In the proposed framework, the outer loop combines a finite state machine with a Fuze Readiness Index (FRI) to realize mission-mode switching and reverse safety-state transition, whereas the inner loop uses a TD3 agent to tune fuzzy scaling factors online according to the observed engagement state and interference condition. The fuze parameter reconfiguration problem is formulated as a Markov Decision Process, and a Python (3.10)-MATLAB (r2023a)/Simulink co-simulation platform is established for algorithm verification. The results show that within the considered simulation setting, the proposed method can improve burst height tracking accuracy, reduce false alarms, and shorten reconfiguration response time under non-stationary composite interference.

The main contributions of this work are summarized as follows:

(1)

A dual-loop fuze control architecture is developed, in which the outer loop handles mode-level decision and safety rollback, and the inner loop performs continuous online tuning of fuzzy scaling factors.

(2)

A fuze-oriented Markov Decision Process is formulated by incorporating engagement deviation, interference intensity, readiness-related information, and mode-dependent control objectives.

(3)

A co-simulation verification framework is established to evaluate the proposed method under dynamic interference and task-switching conditions.

The remainder of this paper is organized as follows: Section 2 introduces the dynamic reconfiguration requirements and the proposed hierarchical control architecture. Section 3 presents the MDP formulation, fuzzy inference model, and safety-state decision model. Section 4 describes the TD3-based optimization algorithm and the co-simulation training framework. Section 5 gives the simulation results and comparative analysis. Finally, Section 6 concludes the paper and discusses future work.

2. Dynamic Reconfiguration Requirements and Technical Route of Loitering Munition Fuze

2.1. Dynamic Reconfiguration Requirements

Compared with conventional ammunition fuze, the loitering munition fuze operates in a more uncertain engagement environment and therefore impose higher requirements on online adaptability, safety management, and mission flexibility. In particular, the fuze must respond not only to target-state variation and electromagnetic interference but also to tactical changes such as mission abortion, target switching, and re-attack preparation. Based on these characteristics, the dynamic reconfiguration requirements of the loitering munition fuze can be summarized in three aspects.

1. Online mission-mode reconfiguration requirement: The loitering munition fuze should be able to receive mission-related information from onboard sensors, seekers, or the upper-level control system and adjust key initiation parameters online. In practical engagements, different target categories such as personnel, light vehicles, fortifications, and armored targets may require different initiation modes, including proximity, airburst, impact, and delay modes. Therefore, the fuze should support real-time mode switching and parameter updating rather than relying solely on pre-launch fixed configuration.

2. Recoverable safety-state transition requirement: Unlike many traditional one-way arming mechanisms, the loitering munition fuze may need to transition from an armed or semi-armed state back to a safer standby state when the attack is cancelled, the target is lost, or the initiation condition is no longer satisfied. This recoverable safety characteristic is closely related to the life-cycle safety of the weapon, especially in scenarios involving target re-selection, mission interruption, or post-mission recovery. Therefore, the fuze control architecture should include a rollback mechanism with explicit safety constraints instead of considering only forward arming logic.

3. Potential cooperative extension requirement in swarm deployment: In swarm deployment scenarios, loitering munitions may further benefit from coordinated timing and strike sequencing through inter-platform communication. However, such swarm-level collaboration involves additional issues, including communication delay, cooperative decision-making, and distributed resource allocation, which are beyond the scope of the present study. Therefore, this paper focuses on the control layer of a single loitering munition fuze while regarding swarm-level cooperative reconfiguration as a potential extension for future work [24,25,26].

2.2. Hierarchical Dual-Loop Control Architecture

2.2.1. Overall Architecture

The proposed architecture of the loitering munition fuze consists of an outer loop and an inner loop.

The outer loop serves as the logic and safety decision layer. It combines a finite state machine (FSM) with a Fuze Readiness Index (FRI) to determine whether the fuze should remain in a safe state, enter an approach/judgment state, or switch into a terminal initiation state. At the same time, it determines the appropriate initiation mode according to task demand and target characteristics. Therefore, the outer loop provides explicit logical constraints for the entire control system and prevents unsafe parameter updates from being executed outside the permitted state window.

The inner loop serves as the continuous parameter adaptation layer. Under a confirmed initiation mode issued by the outer loop, it uses a TD3-based policy to tune the scaling factors of the fuzzy controller online according to the observed engagement state and interference condition. In this way, the fuzzy controller retains its interpretability and nonlinear mapping capability, while the TD3 agent provides adaptive parameter tuning ability under non-stationary disturbance.

This hierarchical design is particularly suitable for fuze control because it separates safety-critical discrete decisions from continuous online optimization. As a result, the learning problem becomes lower dimensional and more stable, while the rollback and mode-switching logic remains explicitly manageable.

2.2.2. Fuzzy Logic as the Execution Layer

Fuzzy logic control is used as the execution layer of the inner loop because it is well suited to nonlinear systems with uncertainty and incomplete information [12]. In the proposed framework, the fuzzy controller receives deviation-related variables and outputs parameter correction commands for fuze initiation control. Gaussian membership functions are used to ensure smooth inference and reduce abrupt switching near decision boundaries.

However, the performance of fuzzy control depends strongly on the choice of scaling factors. Under fixed settings, it is difficult to maintain a satisfactory balance between sensitivity and anti-interference robustness across different engagement conditions. This limitation is especially evident in non-stationary interference environments, where a scaling factor set that performs well in one stage may become unsuitable in another. Therefore, the fuzzy controller is retained as the interpretable nonlinear execution mechanism, while its scaling factors are tuned online by the reinforcement learning layer.

2.2.3. TD3-Based Online Tuning Layer

To achieve online adjustment of fuzzy scaling factors, this paper introduces the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm into the inner loop. The reason for selecting TD3 is that the reconfiguration problem considered here is a low-dimensional continuous-control problem with strong requirements on stability and smoothness. Compared with DQN, TD3 can handle continuous action output directly [16]. Compared with DDPG, it reduces value overestimation and suppresses action jitter through clipped double-Q learning, delayed policy updates, and target policy smoothing [17,19].

It should also be noted that similar hybrid ideas combining reinforcement learning and fuzzy control have been explored in other control domains [21]. Therefore, the contribution of this work does not lie in a generic algorithmic combination alone but in embedding TD3-based online parameter tuning into a fuze-oriented dual-loop structure with mode switching and recoverable safety rollback. In addition, although SAC is another representative continuous-control algorithm, the present study focuses on deterministic low-latency online tuning under a low-dimensional action space and therefore adopts TD3 as the core solver [20,27].

2.2.4. Multi-Modal Reconfiguration and Safety Modeling Based on MDP

To connect the learning algorithm with the physical fuze control problem, the parameter reconfiguration process is formulated as a Markov Decision Process (MDP). The observation state includes engagement deviation, interference-related information, and readiness-related information, while the action is defined as the tuning gain for the fuzzy scaling factors. In this formulation, the reinforcement learning agent is responsible only for continuous parameter adaptation within the admissible operating window specified by the outer loop.

From a control perspective, this architecture can be interpreted as a constrained decision-making process: the outer loop imposes mode-dependent logical and safety boundaries, while the inner loop optimizes parameter adaptation inside these boundaries. This design is consistent with the general idea of safety-constrained reinforcement learning, where performance improvement must be balanced against explicitly enforced safety requirements [22]. Therefore, the proposed architecture does not replace safety logic with a pure learning policy; instead, it combines data-driven adaptation with explicit mission and safety constraints.

2.2.5. Technical Route

As shown in Figure 1, based on the above architecture, the technical route of this study can be summarized as follows: First, the fuze engagement process is abstracted into a state–action–reward framework under explicit safety constraints. Second, the outer loop determines the admissible initiation mode and safety state according to FRI and task logic. Third, the inner loop performs online tuning of fuzzy scaling factors through TD3 policy inference. Finally, the resulting control parameters are demonstrated in simulation in a Python-MATLAB/Simulink co-simulation environment under dynamic interference and task-switching conditions.

3. Multi-Modal Reconfiguration and Fuzzy Safety Control Modeling Based on Markov Decision Process

The adaptive reconfiguration and control of the loitering munition fuze in complex battlefield environments can fundamentally be formulated as a Sequential Decision-Making Problem under Uncertainty. Given that traditional deterministic models struggle to capture these dynamic and stochastic characteristics, this paper selects the Markov Decision Process (MDP) as the mathematical description framework. It constructs a comprehensive control model including three coupled subsystems: continuous parameter reconfiguration optimization based on MDP, reconfigurable fuzzy inference, and multi-modal task and safety-state reconfiguration decision making. This model aims to formalize the fuze’s perception information, control decisions, and reward/punishment mechanisms into computable mathematical tuples, thereby endowing the fuze system with the capabilities of real-time perception and policy evolution and providing theoretical support for subsequently using deep reinforcement learning (DRL) to solve for the global optimal policy.

3.1. Physical State Vector Normalization and Observation State Space Construction

3.1.1. Physical State Vector and Execution Parameter Definition

The proposed offline DRL-based fuzzy adaptive control and reconfiguration method relies heavily on the precise perception of multidimensional battlefield features. To convert heterogeneous sensor data into standard variables processable by the controller, a unified fuze state space needs to be constructed and quantized. This section defines the input state vector $x\left(k\right)$ that drives the fuzzy algorithm in detail. In the initiation control process of the loitering munition fuze, the commonly used parameters are the proximity distance threshold, impact acceleration threshold, timing trigger time, and trigger delay time. To realize the interaction between the offline DRL agent and the virtual battlefield, a standardized state space is constructed. The physical state vector of the fuze at time $k$ is defined as $x\left(k\right)$:

$$x(k)=\left[\begin{array}{c}r\left(k\right)\\ v\left(k\right)\\ a\left(k\right)\\ J\left(k\right)\\ C\left(k\right)\end{array}\right]$$

(1)

The elements in $x\left(k\right)$ are all environmental state perception quantities, where $r\left(k\right)$ is the relative target distance; $v\left(k\right)$ is the relative target velocity; $a\left(k\right)$ is the self-acceleration measured by the loitering munition fuze; $J\left(k\right)$ is the environmental interference/deception signal index, which is a normalized index mainly characterizing the credibility of the electromagnetic environment, with $J\left(k\right)\in [0,1]$, where 1 represents a strong suppression environment and 0 represents an interference-free environment; and $C\left(k\right)$ is the target confidence, mainly used to characterize the certainty of tactical intent, with $C\left(k\right)\in [0,1]$, where 1 represents full target confirmation and 0 represents target loss.

The parameter regulation output vector $\theta \left(k\right)$ is defined as follows:

$$\theta (k)=\left[\begin{array}{c}{r}_0\left(k\right)\\ a_0\left(k\right)\\ T_{delay}\left(k\right)\\ t_0\left(k\right)\end{array}\right]$$

(2)

$r_0\left(k\right)$ represents the proximity distance threshold, which determines the initiation distance of the fuze. It needs to be increased during high-speed engagements to provide lead time and decreased under strong interference to reduce the false alarm rate. $a_0\left(k\right)$ represents the impact acceleration threshold, determining the sensitivity of the impact sensor. It is used to distinguish the high overload generated by normal missile body maneuvers from the impact overload of hitting the target. $T_{delay}\left(k\right)$ represents the initiation delay time, used for anti-interference filtering of signals or to control the penetration depth of the warhead in penetration mode. $t_0\left(k\right)$ represents the mission/self-destruct time threshold, used for dynamic management of the mission window.

3.1.2. State Normalization Processing and Observation Space Construction

As the variables in the state vector of Equation (1) are heterogeneous in physical meaning and numerical range, quantitative normalization is required before they can be used as inputs to the fuzzy controller and the reinforcement learning agent. The purpose of this preprocessing step is not to claim a universally optimal normalization scheme but to establish a unified engineering representation of the key engagement and interference features within the operating range considered in this study.

• Quantization of interference intensity $J\left(k\right)$

The interference intensity $J\left(k\right)$ reflects the severity of the current electromagnetic environment and influences whether the fuze should remain in a precision-sensitive state or switch to an anti-interference mode. In this work, the raw interference descriptor is estimated from the automatic gain control response and the receiver noise-floor estimate, and the corresponding signal-to-noise-ratio-like quantity is defined as in Equation (3) [28].

$$J_S\left(k\right)=10\mathrm{l}\mathrm{n}\left({\displaystyle \frac{P_t\left(k\right)-P_e\left(k\right)}{P_r}}\right)$$

(3)

In the formula, $P_t\left(k\right)$ is the total power measured by the receiver at the current moment, $P_e\left(k\right)$ is the estimated echo signal power, and $P_r$ is the receiver sensitivity noise floor reference value.

To adapt to the input domain [0,1] of the fuzzy controller, a piecewise linear function is used to normalize $J_S\left(k\right)$:

$$J(k)\left\{\begin{array}{c}0,\\ {\displaystyle \frac{J_S\left(k\right)-J_{min}}{J_{max}-J_{min}}},\\ 1,\end{array}\hspace{1em}\begin{array}{c}{J}_S\left(k\right)<J_{min}\\ J_{min}<J_S\left(k\right)<J_{max}\\ J_S\left(k\right)>J_{max}\end{array}\right.$$

(4)

In the formula, $J_{min}$ is the effective threshold, set to 3 dB. When the interference is less than this value, it is regarded as interference-free, and the fuze maintains high sensitivity. $J_{max}$ is the saturation threshold, set to 15 dB. When the interference is greater than this value, it is regarded as strong suppression interference, normalized intensity $J\left(k\right)=1$, forcibly triggering the “anti-interference mode”. The effective threshold and saturation threshold are set to 3 dB and 15 dB, respectively. These two values are used here as engineering initialization thresholds, selected to distinguish weak interference from strong suppression in a practically interpretable manner rather than as theoretically unique constants. Specifically, interference levels below 3 dB are treated as low-disturbance conditions under which the fuze can retain higher sensitivity, whereas levels above 15 dB are treated as strong suppression conditions and are therefore mapped to the saturation region. The same thresholds are kept unchanged throughout all comparative simulations to ensure consistency.

2.

Quantization of target confidence $C\left(k\right)$

The target confidence $C\left(k\right)$ is derived from seeker or detector output and is used to characterize the certainty of target recognition.

To suppress short-duration recognition fluctuations, a sliding-memory weighted model is employed. Assuming the transient recognition probability output at time $k$ is $P_i\left(k\right)$, the smoothed confidence $C\left(k\right)$ is defined as:

$$C\left(k\right)=\beta ·P_i\left(k\right)+(1-\beta )·C\left(k-1\right)$$

(5)

In the formula, $P_i\left(k\right)$ is the transient recognition probability, with $P_i\left(k\right)\in [0,1]$.

$\beta$ is the memory forgetting factor, taken as 0.3~0.5, as an engineering compromise between noise suppression and response speed. A smaller value provides stronger smoothing but slower response, whereas a larger value improves responsiveness at the cost of fluctuation sensitivity. In this study, this range is adopted to maintain stable confidence evolution under transient disturbance rather than to represent an optimal universal choice.

When $C\left(k\right)>0.85$, the fuzzy controller judges it as “High”, allowing precise strikes; when $C\left(k\right)<0.4$, it is judged as “Low”, and impact or standby logic is executed.

3.

Quantization of relative target velocity $v\left(k\right)$

Derived from the seeker’s Doppler shift or the integrated navigation solution, this state quantity is subjected to a first-order low-pass filter to mitigate sensor measurement noise, yielding the effective closing velocity $V_{eff}\left(k\right)$. Setting the maximum design velocity for terminal attack of the loitering munition as $V_{max}$ to adapt to the fuzzy controller’s input domain, a linear proportional function is used to map it to the $[0,1]$ interval:

$$v^{*}\left(k\right)=\left\{\begin{array}{ccc}\hfill {\displaystyle \frac{V_{eff}(k)}{V_{max}}},& & 0\le V_{eff}(k)<V_{max}\hfill \\ \hfill 1,& & V_{eff}\left(k\right)\ge V_{max}\hfill \end{array}\right.$$

(6)

4.

Quantization of loitering munition acceleration $a\left(k\right)$

Acceleration $a\left(k\right)$ is the core physical quantity characterizing the maneuvering overload state of the loitering munition and determining whether the delay/impact acceleration reaches the initiation threshold. It is directly derived from the real-time synthesized overload value of the onboard accelerometer or IMU. To balance the overload during the loitering munition’s flight with the overload signal at the moment of impact, the raw data need to be smoothed to extract the current maneuvering overload amplitude of the loitering munition. Setting the maximum allowable overload for normal flight maneuvers as $A_m$ to enable the fuzzy controller to perceive whether the missile body is in a high-overload maneuvering state, a saturation function mapping is adopted:

$$a^{*}\left(k\right)=\mathrm{m}\mathrm{i}\mathrm{n}({\displaystyle \frac{a\left(k\right)}{A_m}},1)$$

(7)

5.

Quantization of relative target distance $r\left(k\right)$

This is directly derived from the ranging value of the laser or radio detector. An attention and action truncation function is introduced. Based on the actual damage capability of the ammunition, a maximum focus distance $R_{focus}$ is set. Inverse proportional mapping is used to improve the resolution at close range (the closer the distance, the larger the normalized value, and the higher the sensitivity):

$$r^{*}(k)=\left\{\begin{array}{ccc}\hfill 1-{\displaystyle \frac{r(k)}{R_{focus}}},& & 1\le r\left(k\right)<R_{focus}\hfill \\ \hfill 0,& & r\left(k\right)\ge R_{focus}\hfill \end{array}\right.$$

(8)

6.

Construction of deviation-related observation variables

To endow the controller with the ability to pre-judge the missile-target engagement situation, this paper performs differential processing on the normalized core deviation quantities. The core input deviation variable for the fuzzy controller is defined as:

$$e\left(k\right)=r^{*}\left(k\right)-r_{opt}^{*}$$

(9)

where $r_{opt}^{*}$ is the normalized optimal burst height setpoint, determined by the outer loop baseline parameters.

The error change rate is defined as the difference between the error at the current sampling time and the error at the previous time:

$$\dot{e}\left(k\right)=e\left(k\right)-e\left(k-1\right)$$

(10)

Through the above normalization and feature extraction, the full-state observation vector of the DRL agent at time $t$ is constructed:

$$s_t={[e\left(t\right),\dot{e}\left(t\right),J\left(t\right),C\left(t\right),M\left(t\right)]}^T$$

(11)

where the deviation between the current distance and the ideal burst height is expressed as $e\left(t\right)$, and $\dot{e}\left(t\right)$ reflects the current control deviation trend. $J\left(t\right)$ and $C\left(t\right)$ reflect the complexity of the external environment, and $M\left(t\right)$ is the initiation mode state decided by the current outer loop.

3.1.3. Practical Setting Rationale for Threshold Parameters

The normalization thresholds and weighting parameters used in this study are selected as engineering initialization values according to typical anti-interference discrimination requirements, target-confidence smoothing needs, and safety-state transition logic in the loitering munition fuze scenarios [1,2,3,4,5,28]. Their role is to establish a consistent and physically interpretable simulation framework for validating the proposed control architecture. The present work does not claim that these values are globally optimal. Instead, they are kept fixed across the comparative experiments so that the influence of the proposed dual-loop strategy can be evaluated under a unified setting. A broader sensitivity analysis of these parameters will be included in future work.

3.2. Continuous Reconfiguration Optimization Modeling Based on MDP

To reduce the subjectivity of manual parameter tuning, the online adjustment of fuzzy scaling factors is formulated as a Markov Decision Process (MDP), represented by the tuple $〈S,A,R,\gamma 〉$, where $S$ is the state space observed by the agent; $A$ is the action space output by the agent; $R$ is the reward function guiding policy optimization; and $\gamma$ is the discount factor balancing current and future returns. It should be emphasized that the learning agent is responsible for continuous parameter adaptation only within the admissible operating region defined by the outer-loop safety logic, rather than replacing the fuze safety mechanism itself.

3.2.1. Action Space

Rather than directly outputting control voltages, the agent modulates the sensitivity and stiffness of the control system by adjusting the scaling factors of the fuzzy controller. The action vector $u_t$ is defined as the reconfiguration gain for the scaling factors:

$$u_t={[{\lambda }_e,{\lambda }_{ec},{\lambda }_u]}^T,\hspace{1em}{\displaystyle u_t\in [0.5,2.0]}$$

(12)

Based on this action, the scaling factors $K$ of the fuzzy controller are reconstructed in real time:

$$\left\{\begin{array}{c}{K}_e\left(t\right)=K_{e0}\left(t\right)·{\lambda }_e\\ K_{ec}\left(t\right)=K_{ec0}\left(t\right)·{\lambda }_{ec}\\ K_u\left(t\right)=K_{u0}\left(t\right)·{\lambda }_u\end{array}\right.$$

(13)

In the formula, $K_{e0}\left(t\right)$, $K_{ec0}\left(t\right)$, and $K_{u0}\left(t\right)$ denote the baseline scaling parameters of the fuzzy controller. Through the dynamic adjustment of $u_t$, the system can be continuously reconfigured between high sensitivity and high robustness.

3.2.2. Reward Function Design

To guide the agent toward a balance among control precision, false-alarm suppression, and action smoothness, a composite reward function is defined in Equation (14). The reward is composed of three parts: a precision-related term, a safety-related term, and a smoothness-related term.

$$R_t=r_{prec}+r_{safe}-p_{act}$$

(14)

• Precision reward $r_{prec}$: Real-time encouragement to minimize burst height error, adopting the form of a Gaussian kernel function:

  $$r_{prec}=\mathrm{e}\mathrm{x}\mathrm{p}(-{\displaystyle \frac{{\left|r\left(t\right)-r_{opt}\right|}^2}{{\sigma }^2}})$$

  (15)

This term provides continuous positive feedback when the tracking error remains close to the desired burst height setpoint.

2.

Safety constraint reward $r_{safe}$: If a false alarm (false safety release) occurs in a non-initiation zone (e.g., outside the safe distance), a massive penalty is given:

$$r_{safe}=\left\{\begin{array}{ccc}\hfill \mathrm{-100},& & ifFalsePositive\hfill \\ \hfill 0,& & otherwise\hfill \end{array}\right.$$

(16)

Its purpose is to encode the priority of safety over aggressive sensitivity improvement and to prevent the learned policy from exploiting risk-prone actions to gain short-term tracking benefits.

3.

Action smoothness penalty $p_{act}$: Penalizes abrupt changes in the reconfiguration gain, so as to reduce high-frequency parameter jitter and improve the practical executability of the control signal:

$$p_{act}=\eta ·{‖u_t-u_{t-1}‖}_2$$

(17)

In this study, the overall reward is written as a linear weighted combination of the above terms. This form is adopted as an engineering compromise for three reasons. First, it preserves interpretability by allowing each objective to remain physically identifiable. Second, it simplifies training and stabilizes numerical optimization in the considered low-dimensional control task. Third, it makes it easier to explicitly reflect the relative importance of precision, safety, and smoothness in fuze control. The authors acknowledge that the coupling among these objectives may become nonlinear outside the operating range considered here. Therefore, the linear weighted reward used in this paper should be understood as a practically tractable approximation within the current simulation framework rather than as a universally optimal reward design. The reward is designed to balance tracking precision, safety preservation, and action smoothness under the mode and safety constraints imposed by the outer loop.

3.3. Fuzzy Inference Model for Loitering Munition Fuze Initiation Parameter Reconfiguration

3.3.1. Fuzzy Subsets and Membership Function Design

In the control architecture constructed in this paper, the fuzzy inference layer acts as the inner-loop executor, responsible for performing nonlinear mapping of the real-time deviation situation based on the scaling factors issued by the external DRL agent. To achieve precise parameter correction, this paper defines fuzzy subsets for the input variables, error E and error rate EC, as well as the output variable, parameter correction amount $\Delta \theta$.

The fuzzy subsets for the input variables E and EC are defined as {NB, NS, ZE, PS, PB}. Their physical meaning corresponds to the degree to which the burst height deviates from the baseline value and its changing trend.

The fuzzy subsets for the output variable $\Delta \theta$ are defined as {NB, NS, ZE, PS, PB}.

The membership functions for all fuzzy subsets adopt a Gaussian distribution to ensure the smoothness of the inference process. To ensure the completeness and overlap of adjacent fuzzy subsets within the domain, a membership overlap factor is introduced for mathematical constraint. A Gaussian membership function (Gaussian MF) is used to fuzzify the input error $E=K_e·e\left(t\right)$ and error change rate $EC=K_{ec}·\dot{e}\left(t\right)$:

$$\mu \left(x\right)=\mathrm{e}\mathrm{x}\mathrm{p}\left(-{\displaystyle \frac{{(x-c_{ij})}^2}{{2\sigma }_{ij}^2}}\right)$$

(18)

$c_{ij}$ represents the characteristic prototype point of the fuzzy linguistic state. When the input $x=c_{ij}$, the membership degree reaches its maximum value $\mu =1$, indicating that the current state’s degree of belonging to this fuzzy subset reaches saturation, and at this time, this rule has the maximum contribution weight to the inference result.

${\sigma }_{ij}$ determines the coverage range and sensitivity of the fuzzy set. In this architecture, ${\sigma }_{ij}$ remains unchanged as the baseline shape, while the DRL achieves equivalent stretching or compression of the domain by adjusting the external scaling factors $K_e$ and $K_{ec}$, thereby dynamically changing the control stiffness of the system. To ensure moderate overlap between adjacent fuzzy sets and smooth transition of control quantities, a width coefficient of 1/2~1/3 of the distance between adjacent centers is usually set. This paper sets the overlap rate to 0.5, meaning the membership degree is 0.5 at the midpoint between two centers, calculated as:

$${\sigma }_{ij}={\displaystyle \frac{\left|c_{i,j+1}-c_{ij}\right|}{2\sqrt{2ln2}}}\approx 0.21$$

(19)

3.3.2. Dynamic Fuzzy Inference

The Mamdani inference model is adopted. Since the input scaling factors $K_e$ and $K_{ec}$ change in real time with the DRL action $a_t$, the same physical input $e\left(t\right)$ will activate different fuzzy rule strengths:

$${\omega }_l={\mu }_A(K_e\left(t\right)·e)·{\mu }_B(K_{ec}\left(t\right)·\dot{e})$$

(20)

This formulation implies that the DRL agent dynamically reshapes the system’s inference surface. This mechanism allows the fuze to autonomously explore the inference logic most suitable for the current situation when encountering unknown unstructured interference, improving adaptive reconfiguration capability under non-stationary disturbance.

3.3.3. Defuzzification and Parameter Output

The Center of Gravity method is used to calculate the final parameter correction amount $\Delta \theta \left(k\right)$:

$$\Delta \theta \left(k\right)=K_u\left(t\right)·{\displaystyle \frac{{\sum }_{l=1}^N{\omega }_l·y_l}{{\sum }_{l=1}^N{\omega }_l}}$$

(21)

The final fuze execution parameter is the superposition of the baseline value and the correction value and is subject to the gating constraint of the safety-state function $S(·)$:

$$\theta \left(k\right)=S\left({\theta }_{req}\left(k\right),FRI\left(k\right)\right)=\left\{\begin{array}{ccc}\hfill \mathsf{\Psi }(u_{cmd}\left(k\right),I_s\left(k\right))+\Delta \theta \left(k\right)& ,\hfill & ifFRI\left(k\right)\ge T_{arm}\hfill \\ 0\hfill & ,\hfill & ifFRI\left(k\right)<T_{safe}\hfill \end{array}\right.$$

(22)

$\Psi \left(·\right)$ is the tactical intent and parameter mapping function. This function receives control commands $u_{cmd}\left(k\right)$ from the flight control system, such as forced self-destruct or mission abort commands, as well as information $I_s\left(k\right)$ from the seeker, such as target type, and outputs the physical baseline vector ${\theta }_{base}={[r_{base},a_{base},T_{base},t_{base}]}^T$ at the start of regulation. $FRI\left(k\right)$ is the Fuze Readiness Index mentioned in Section 3.4, which determines whether parameters can be reconfigured.

3.4. Multi-Modal Task and Safety-State Reconfiguration Decision Model

The DRL-based parameter tuning model and the fuzzy inference layer described above constitute the continuous control part of the proposed fuze system. However, continuous parameter adaptation alone is insufficient for the loitering munition fuze control because abrupt mission changes and recoverable safety-state transitions must also be handled at the logic level. Therefore, a higher-level discrete decision mechanism is introduced in this study. This mechanism is implemented through a finite state machine (FSM) and is responsible for initiation-mode switching, state rollback, and safety gating of parameter updates.

3.4.1. Multi-Modal Task Reconfiguration Logic

According to the four fuze initiation modes described above, fuze dynamic regulation needs to establish a mapping function $\mathsf{\Phi }$ from target characteristics to initiation modes based on the tactical information $I_s\left(k\right)$ from the seeker.

According to the strike requirements of loitering munitions against different target categories, the fuze should be able to switch among several initiation modes rather than operate under a single fixed logic. In this study, the set of representative initiation modes is defined as:

$$M=\left\{M_{prox},M_{air},M_{contact},M_{delay}\right\}$$

(23)

where $M_{prox}$ represents proximity mode, $M_{air}$ represents airburst mode, $M_{contact}$ represents impact mode, and $M_{delay}$ represents delay mode.

The firing mode selection model is established as follows:

$$M\left(k\right)=\mathsf{\Phi }(I_s\left(k\right),x(k\left)\right)$$

(24)

where $M\left(k\right)$ is the confirmed working mode at time $k$, $I_s\left(k\right)$ is the target type, and $x\left(k\right)$ is the current state vector.

The mode-selection function in Equation (24) maps target characteristics and the current engagement state to the most appropriate initiation mode. Once a mode change is triggered, the outer loop updates the admissible control objective, and the inner loop correspondingly performs parameter tuning within the new mode-dependent operating region. In this way, discrete task switching and continuous parameter adaptation are explicitly decoupled rather than being mixed in the same decision space. The representative mapping between target categories and initiation modes used in this study is summarized in

Table 1. Mapping logic between target types and firing modes.

Target Type	Firing Mode (M)	Core Control Parameter
Personnel/Radar	Proximity Mode M = 00	Burst Height Threshold $r_0$
Trenches	Airburst Mode M = 01	Delay Time $t_0$
Light Vehicles/General Fortifications	Impact Mode M = 10	Impact Acceleration $a_0$
Tanks/Bunkers	Delay Mode M = 11	Penetration Delay $T_{delay}$

.

3.4.2. Safety-State Reconfiguration Logic

To describe whether the current system state is sufficiently mature for arming and terminal execution, a Fuze Readiness Index (FRI) is introduced as a dimensionless scalar indicator: $FRI\left(k\right)\in [0,1]$. This indicator combines target-related confidence, environmental suitability, and self-check status into a unified readiness measure, as defined in Equation (25). In the proposed architecture, the FRI serves as a supervisory safety variable rather than as a direct actuation command.

$$FRI\left(k\right)={\epsilon }_C·C\left(k\right)+{\epsilon }_J·\left(1-J\left(k\right)\right)+{\epsilon }_S{\sigma }_{sys}\left(k\right)$$

(25)

In the formula, ${\epsilon }_C,{\epsilon }_J,\mathrm{a}\mathrm{n}\mathrm{d}{\epsilon }_S$ are weighting coefficients, and ${\epsilon }_C+{\epsilon }_J+{\epsilon }_S=1$. To ensure tactical orientation, the tactical weight ${\epsilon }_C$ is set to the highest value, and ${\sigma }_{sys}\left(k\right)$ is the system self-check flag bit.

For the loitering munition fuze system, it is necessary to define the corresponding operational meanings for the fuze at different combat phases, as detailed in

Table 2. Correspondence between combat phases and operational meanings.

Combat Phase Name	Phase Parameter	Operational Meaning
Safe/Loitering Mode	$S$1	Firing prohibited, parameters cannot be regulated
Approach/Judgment Mode	$S$2	Parameter fine-tuning, trigger judgment begins
Attack/Firing Mode	$S$3	Enters final firing window

.

To reduce frequent state oscillation caused by noisy observations, a dual-threshold hysteresis mechanism is introduced. For the loitering munition fuze safety system, this paper designs two safety-state transition logics. By defining an arming threshold $T_{arm}=0.85$ and a mission cancel threshold $T_{safe}=0.40$, a stable control mechanism is established, where the fuze only enters the attack state when the FRI is consistently higher than the arming threshold and is forcibly rolled back to the safe loitering state only when the FRI drops below the mission cancel threshold. In the hysteresis interval 0.4 < FRI < 0.85, the state machine remains locked, performing only parameter fine-tuning.

Logic A: Forward Precession (Approach → Attack), if currently in S2 (Approach Phase) and satisfies:

$$\left(FRI\right(k)\ge T_{arm})\cap \left(r\right(k)\le r_{cri})$$

(26)

The system transitions to S3, where final initiation execution is permitted. At this stage, the parameter outputs generated by the inner-loop controller are allowed to enter the hardware execution path.

Logic B: Reverse Safety-State Conversion (Approach/Attack → Loitering), if currently in S2 or S3 and satisfies:

$$\left(FRI\right(k)\le T_{safe})\cup (u_{cmd}==ABORT)$$

(27)

The system returns to the safe loitering state S1. In this case, the fuze disables initiation execution and resets the corresponding safety-related actuation path. This mechanism enables mission interruption, target re-selection, and safe rollback under degraded engagement conditions.

Through the above design, the proposed fuze control architecture combines continuous parameter adaptation with discrete safety-constrained mission logic. The normalization model provides a unified state representation, the MDP formulation supports online parameter tuning, the fuzzy inference layer performs interpretable nonlinear execution, and the FRI-FSM mechanism enforces mode switching and safety rollback. Together, these components establish the theoretical basis for the co-simulation verification presented in the following sections.

4. Design of Initiation Parameter Reconfiguration Optimization Algorithm Based on Deep Reinforcement Learning

4.1. Principle and Optimization Mechanism of TD3 Algorithm

The architecture proposed in this paper adopts a decision-optimization hierarchical collaborative mechanism. The outer-loop decision layer determines the macro-initiation mode M based on the target type. Subsequently, the inner-loop TD3 agent incorporates the current initiation mode M as part of the observation state to specifically output the optimal parameter reconfiguration gain $u_t$. This design avoids mixing discrete initiation mode switching and continuous parameter adjustment within the same action space, effectively reducing the training dimensionality of reinforcement learning and improving the stability of policy learning. Addressing the continuous action space control problem in fuze parameter reconfiguration, traditional discrete-action algorithms like DQN prove fundamentally inadequate. Although the Deep Deterministic Policy Gradient (DDPG) algorithm can handle continuous actions, it is prone to overestimation bias in complex stochastic environments, frequently trapping the fuze parameters in local optima or inducing severe oscillations in the early stages of training. To this end, this paper selects the TD3 algorithm as the core solver, enhancing the stability and convergence speed of policy learning by introducing three key mechanisms.

4.1.1. Clipped Double Q-Learning Mechanism

To eliminate the overestimation bias of the value function, this paper constructs two independent Critic networks $Q_{{\theta }_1}$ and $Q_{{\theta }_2}$. When calculating the target Q-value, the minimum of their outputs is taken as the evaluation baseline. This mechanism leverages underestimation bias to counteract overestimation, ensuring the agent refrains from outputting overly aggressive reconfiguration parameters during sudden interference (e.g., abrupt spikes in $J\left(t\right)$). The formula for calculating the target value is as follows:

$$y=r+\gamma \underset{i=\mathrm{1,2}}{\mathit{min}}{Q}_{{\theta }_i^{\prime }}(s^{\prime },u^{\prime })$$

(28)

where $Q_{{\theta }_i^{\prime }}$ is the target Critic network, and $u^{\prime }$ is the action output by the target Actor network.

4.1.2. Target Policy Smoothing

The fuze control system has extremely high requirements for parameter smoothness. To prevent the Critic network from overfitting to specific action peaks, this paper introduces clipped normal noise $ϵ$ when calculating the target action. This is equivalent to performing a fuzzification process around the action space, forcing the network to learn a robust policy that holds high value within the action neighborhood.

$$u^{\prime }=clip\left(u_{{\varphi }^{\prime }}\left(s^{\prime }\right)+ϵ,u_{min},u_{max}\right),\hspace{1em}ϵ~clip(\mathcal{N}\left(0,\tilde{\sigma }\right),-c,c)$$

(29)

After applying this regularization treatment, the high-frequency jitter of the reconfiguration gain $\lambda$ in the steady state is effectively suppressed.

4.1.3. Delayed Policy Updates

Because the Critic network’s value evaluation underpins the Actor network’s optimization, prematurely updating the Actor prior to Critic convergence invariably precipitates policy divergence. This paper sets the delay frequency d = 2, meaning that for every 2 updates of the Critic network, the Actor network and all target networks are updated once. This mechanism ensures that the policy gradient is executed under the premise of relatively accurate value evaluation, significantly improving the convergence stability during the online learning process.

4.2. Policy Network Architecture Design

4.2.1. Actor–Critic Network Topology

• Actor network ${\mu }_{\varphi }$: Responsible for outputting the reconfiguration gain action $u_t$.

Input layer: Receives the normalized state vector $s_t={[e\left(t\right),\dot{e}\left(t\right),J\left(t\right),M\left(t\right)]}^T$.

Hidden layer: Contains two Fully Connected Layers with 400 and 300 nodes, respectively; the ReLU activation function is adopted to enhance nonlinear expression capabilities.

Output layer: Outputs a 3-dimensional action vector $u_t={[{\lambda }_e,{\lambda }_{ec},{\lambda }_u]}^T$; a Tanh activation function squashes the output into the $[\mathrm{-1},1]$ range, subsequently undergoing an affine transformation to scale it to the valid action space $[0.5,2.0]$.

2.

Critic network ($Q_{{\theta }_1},Q_{{\theta }_2}$): Adopts a dual-network structure, responsible for evaluating action values.

Input layer: Simultaneously receives the state $s_t$ and action $u_t$.

Structure: A dual hidden layer structure similar to the Actor network.

Output layer: Outputs a single scalar Q-value used to guide the Actor update.

4.2.2. Optimizer Configuration and Parameter Settings

The network training utilizes the Adam optimizer, which possesses adaptive learning-rate adjustment capabilities, making it suitable for handling sparse and non-stationary reward signals in fuze control. To strike an optimal balance between exploration and exploitation while accommodating the real-time constraints of online learning, the core training hyperparameters are set as shown in

Table 3. Core training hyperparameter settings.

Parameter Name	Sign	Set Value	Meaning
Actor Learning Rate	${\alpha }_{\mu }$	${10}^{-4}$	A small learning rate ensures stable policy evolution
Critic Learning Rate	${\alpha }_Q$	${10}^{-3}$	A larger learning rate accelerates the convergence of value evaluation
Discount Factor	$\gamma$	$0.99$	Focuses on long-term cumulative returns
Experience Replay Buffer Capacity	$\mathcal{R}$	${10}^6$	Stores a large number of historical samples to eliminate correlation
Batch Size	B	$256$	The number of samples sampled from the buffer each time
Soft Update Coefficient	$\tau$	$0.005$	The moving average update rate for target network parameters
Exploration Noise	${\sigma }_{exp}$	$0.1$	Gaussian noise applied to actions in the early stage of training

.

The hyperparameters in Table 3 were selected through preliminary tuning to balance training stability, response smoothness, and implementation simplicity in the considered co-simulation environment. In particular, the actor learning rate was set lower than the critic learning rate to reduce policy oscillation, while the delayed update and target smoothing settings followed the standard TD3 design philosophy for suppressing overestimation and action jitter. These hyperparameters were then fixed across the reported experiments to ensure a consistent basis for comparison.

4.2.3. Hyperparameter Sensitivity and Baseline Algorithm Comparison

To further justify the selected training configuration and the choice of TD3, additional comparative experiments were conducted under the same co-simulation environment, state–action definition, and training horizon. Two aspects were evaluated: the sensitivity of the training process to the learning rate and the convergence behavior of different continuous-control DRL baselines.

As shown in Figure 2a, the learning rate has a clear influence on both training speed and reward stability. A relatively high learning rate (${10}^{-3}$) accelerates reward growth in the early stage but introduces larger oscillations in the later stage. In contrast, a relatively low learning rate (${10}^{-5}$) results in slow and incomplete training progress within the same episode budget. The selected learning rate (${10}^{-4}$) provides a more balanced trade-off between convergence speed and training stability and was therefore adopted in the subsequent experiments.

To further support the selection of TD3, a convergence comparison among TD3, SAC, and DDPG was performed, as shown in Figure 2b. The results show that the TD3-based method reaches a higher and more stable reward plateau than SAC under the present low-dimensional continuous-control setting, while DDPG exhibits a pronounced instability region during training. This observation is consistent with the role of clipped double-Q learning, delayed policy updates, and target policy smoothing in improving learning stability. Therefore, the additional comparison supports the use of TD3 as the core online tuning algorithm in the proposed dual-loop architecture.

4.3. Co-Simulation Training Platform Setup

To verify the effectiveness of the algorithm in dynamic environments, this paper builds a co-simulation platform based on Python and MATLAB/Simulink.

4.3.1. Training Environment Interface Design

The simulation platform employs a decoupled “Decision–Execution” architecture. The Python environment hosts the agent, executing the PyTorch deep learning framework. The MATLAB/Simulink side serves as the environment host, responsible for solving fuze ballistic equations and fuzzy inference logic and generating environmental interference signals. A high-speed data link is established between the two via the MATLAB Engine API for Python. At each simulation time step ($∆t=1\mathrm{m}\mathrm{s}$), the interaction workflow proceeds as follows:

State Reporting: MATLAB calculates the current deviation $e\left(t\right)$ and environmental features, packages them into the state vector $s_t$, and sends it to Python.

Action Decision: The Actor network on the Python side infers the reconfiguration action $u_t$ based on $s_t$.

Parameter Reconfiguration: MATLAB receives $u_t$ and updates the scaling factors $K_e,K_{ec},K_u$ of the fuzzy controller in real time.

Physical Evolution: Simulink executes one simulation step to obtain the next state $s_{t+1}$ and calculates the reward $R_t$.

4.3.2. Algorithm Online Execution Pseudocode

Based on the interaction mechanism described in Section 4.3.1, the online execution process of the proposed TD3-based reconfiguration algorithm can be summarized in Algorithm 1. The pseudocode presents the complete training workflow, including state acquisition, action generation, environment feedback, experience storage, critic update, delayed actor update, and target-network soft update. This description is intended to clarify the implementation logic of the proposed co-simulation framework and improve the reproducibility of the algorithm.

Algorithm 1: TD3 Pseudocode

1. Initialize the Actor policy network ${\mu }_{\varphi }$ and two Critic value networks $Q_{{\theta }_1}, Q_{{\theta }_2}$ with random parameters. Initialize the corresponding target networks and copy the main network parameters to the target networks. Establish an experience replay buffer $D$ with capacity $\mathcal{R}$. 2. For each training episode $k=1~M$, reset the MATLAB/Simulink simulation environment and obtain the initial fuze observation state $s_1$. 3. Within each simulation time step $t=1~T$, execute the following operations in sequence: (1) Action Generation: Output the reconfiguration action based on the current policy network, and add Gaussian exploration noise to enhance exploration capability: ${\mu }_t=clip\left(u_{\varphi }\left(s_t\right)+\mathcal{N}\left(0,{\sigma }_{exp}\right),u_{min},u_{max}\right)$. (2) Environment Response: Pass ${\mu }_t$ into the MATLAB environment to correct the fuzzy scaling factors in real time and execute one simulation step. If the safety truncation mechanism is triggered (FRI < 0.4), forcibly abort the episode and impose a penalty; otherwise, observe the immediate reward $r_t$ and the next state $s_{t+1}$. (3) Experience Storage: Store the state transition tuple ($s_t,{\mu }_t,r_t,s_{t+1}$) into the experience replay buffer $D$. If the number of samples in $D$ exceeds the capacity $\mathcal{R}$, overwrite the oldest data. 4. When the data volume in the replay buffer $D$ meets the batch size B requirement, randomly sample a batch of data. Add clipped noise when calculating the target action to achieve policy smoothing and calculate the target value $y$ through the clipped double Q-learning mechanism, taking the minimum of the two target Critic networks. Update the two Critic network parameters using the mean squared error minimization criterion. 5. If the current update step meets the delay frequency (i.e., $tmodd=0$), perform one Actor network update by maximizing the Q-value via the deterministic policy gradient. Simultaneously, use the soft update method to synchronously update all target network parameters. 6. Update the current state $s_t⟵s_{t+1}$. If burst height convergence or a timeout is not triggered, the current episode is not over, so loop back to step 3 to continue execution. Otherwise, end this episode; if the convergence conditions are met, save the current network parameters as the optimal parameters ${\varphi }^{*}⟵\varphi$ and end the training.

5. Implementation and Simulation of Optimization Algorithm Based on Deep Reinforcement Learning

5.1. Simulation Environment Setup and Signal Characteristics

5.1.1. Virtual Battlefield Environment Model

The simulation platform employs a 6-DOF rigid-body dynamics model to characterize the terminal engagement motion of the loitering munition and to generate the corresponding fuze-related state variables. Aerodynamic disturbance, target echo fluctuation, and interference-related uncertainty are incorporated into the simulation environment so as to evaluate the online adaptability of the proposed control method under representative non-stationary conditions.

To verify the anti-interference capability of the algorithm, a non-stationary composite interference model $J\left(t\right)$ is constructed. This model is formed by the superposition of background Gaussian white noise and random burst pulses:

$$J\left(t\right)=\mathcal{N}\left(0,{\sigma }_n\right)+{\displaystyle \sum _k}{A}_k·rect\left({\displaystyle \frac{t-t_k}{{\tau }_k}}\right)$$

(30)

$A_k$ is the pulse amplitude, and ${\tau }_k$ is the duration.

In the present work, the environment model focuses mainly on electromagnetic interference, background noise, burst disturbance, and intermittent observation loss, which are the dominant uncertainty sources considered in this study. Atmospheric variations such as wind, temperature, air density change, and pressure fluctuation are not explicitly modeled here. Therefore, the simulation results should be interpreted as validation within the current interference-oriented environment model rather than as a full high-fidelity battlefield reproduction.

5.1.2. Evaluation Metrics Definition

To quantitatively evaluate the performance of the control algorithm, the following core metrics are defined:

• Root Mean Square Error (RMSE): Quantifies the robustness of the model under extreme working conditions.

$$RMSE=\sqrt{{\displaystyle \frac{1}{N}}{\displaystyle \sum _{t=1}^N}{(y_i-\widehat{y_i})}^2}$$

(31)

Here, $y_i$ and $\widehat{y_i}$ are the actual value and the estimated output value, respectively.

2.

Mean Absolute Error (MAE): Measures how much the algorithm deviates from the true distance on average throughout the entire combat process.

$$MAE={\displaystyle \frac{1}{N}}{\displaystyle \sum _{t=1}^N}\left|y_i-\widehat{y_i}\right|$$

(32)

3.

False Alarm Rate (FAR): Measures the probability of the fuze triggering falsely in a non-initiation zone, characterizing safety.

$$FAR={\displaystyle \frac{N_{fp}}{N_{total}}}\times 100\%$$

(33)

Here, $N_{total}$ represents the total number of Monte Carlo target shooting simulation experiments set as no-target or pure interference conditions; $N_{fp}$ represents the number of times the fuze state machine erroneously enters the armed state or outputs an initiation pulse under the aforementioned conditions.

4.

Reconfiguration Response Time ($T_{resp}$): Measures the adaptation speed of the algorithm to sudden interference.

5.2. Empirical Training Stability Analysis

In the co-simulation platform, the maximum number of training episodes is set to M = 5000, and the maximum number of steps per episode is T = 2500. After initiating training, the average reward convergence curve of the TD3 agent is shown in Figure 3.

As shown in Figure 3, the learning process of the agent exhibits significant stage-like characteristics:

1. Exploration Phase (Episodes 0–2000): Because the Actor network is in an initialized state, the agent conducts large-scale action space exploration in the complex electromagnetic confrontation environment. Due to the inability to accurately predict the impact of interference intensity $J\left(t\right)$ on the perception field, the reconfiguration gain output fluctuates erratically, causing the burst height error to frequently breach acceptable thresholds, and the reward value oscillates drastically at a low level around −400.

2. Transition and Improvement Phase (Episodes 2000–3000): With the accumulation of high-quality samples in the experience replay buffer, the Critic network’s evaluation of state–action values becomes more accurate. The agent begins to realize the importance of lowering perception sensitivity when interference spikes, and the average reward curve shows a significant upward trend.

3. Convergence and Stabilization Phase (Episodes 3000–5000): The average reward value converges and stabilizes at the highest level of around 100. At this point, the agent has mastered the optimal reconfiguration policy and can output precise scaling factor gains for transiently changing battlefield situations. This indicates that the agent has successfully explored the optimal parameter reconfiguration policy for different interference modes and target characteristics.

Overall, the training result suggests that the learned policy achieves empirical convergence or, more precisely, a stable performance plateau in the present co-simulation environment. However, no formal mathematical proof of convergence is claimed in this work. Therefore, Figure 3 should be interpreted as evidence of stable training behavior under the current experimental setup rather than as a rigorous convergence proof.

5.3. Full-Process Task Decision and Safety Logic Simulation

To verify the completeness of the decision logic of the fuze system under variable tactical backgrounds, this section constructs a full-process task simulation scenario, focusing on evaluating the logical response correctness of the system to multi-modal switching commands and safety recovery signals.

The full simulation cycle is set to include three key logic triggering events, and subsequent simulation processes are based on this combat flow:

• Initial State (T = 0~1.5 s): The system is in the target approach phase. Input conditions include: high target confidence $C_k$, target identified as personnel, and fuze readiness $FRI>0.85$.
• Tactical Change (T = 1.5 s): The seeker secondarily confirms that the target nature has changed to a reinforced bunker. The system needs to respond to this tactical input by performing task modal reconfiguration, changing the initiation mode to penetration mode.
• Safety Fuse (T = 2.5 s): This simulates a sudden mission abort command, causing a Fuze Readiness Index $FRI<0.4$, falling below the safety threshold. The system must immediately execute reverse safety-state reconfiguration, returning the fuze to a safe state.

The full-process task decision and safety logic simulation of the loitering munition fuze is shown in Figure 4.

As shown in Figure 4, at T = 1.5 s, the tactical input command changes. The initiation mode rapidly jumps from proximity mode (M = 00) to penetration mode (M = 11) within the next simulation step. The entire switching process exhibits no logic race or deadlock, verifying that the recognition and execution of multi-modal task commands by the finite state machine (FSM) are accurate and reliable. At T = 2.5 s, the FRI curve drops steeply due to the mission abort command, instantly falling below the safety threshold $FRI<0.4$. The system logic responds immediately, and the state machine forcibly resets directly from the penetration mode state to the safe state. This result indicates that within the present simulation setup, the system can execute logic rollback once the safety threshold is violated.

5.4. Reconfiguration Performance Verification Under Typical Strike Scenarios

To simulate real terminal strike conditions, a combat scenario based on the simulation in Section 5.3 is set up: conventional suppression jamming is encountered during t = 0.5~1 s, with J = 0.6; at this time, the fuze initiation mode is proximity mode. From t = 1.5~2.5 s, the fuze initiation mode switches to penetration mode, and continuous background noise exists in the environment. When t > 2.45 s, strong interference with intensity J > 0.95 is injected, and the fuze performs a safety-state reset. The simulation results are shown in Figure 5.

From t = 0.5~1 s, as can be seen from Figure 5b, when conventional suppression jamming strikes, the TD3 agent quickly adjusts the reconfiguration gain ${\lambda }_e$ downwards from the steady state 1.5 to 0.8. This action significantly reduces the sensitivity of the control loop to high-frequency noise. As seen in Figure 5a, thanks to the timely downward adjustment of the gain, the red tracking error $e\left(t\right)$ is effectively suppressed within a safe range and does not diverge synchronously with the interference intensity. At t > 1.5 s, transitioning to penetration mode disengages the proximity feedback loop. As shown in Figure 5a, the tracking error signal is cut off. Simultaneously, the agent locks the internal gain ${\lambda }_e$ to a unit value or zeroes it out, indicating that the algorithm has correctly identified the current mission attribute, stopped invalid adjustments to burst height parameters, and achieved task decoupling of the control logic.

Furthermore, to verify the actual driving effect of the algorithm on the fuze execution mechanism, Figure 6 plots the coupled evolutionary trajectory of the external interference stimulus $J\left(t\right)$ and the physical execution parameters $r_0,T_{delay}$.

As shown in Figure 6, during the conventional interference from t = 0.5~1 s, the upper graph shows the interference intensity J increasing, which drives the proximity threshold $r_0$ in the lower graph to compress the action threshold from 10 m to about 5 m. In the penetration phase from t = 1.5~2.45 s, although environmental background noise still fluctuates, the delay time $T_{delay}$ steps from 0 ms and is locked at 0 ms. At t = 2.45 s, the interference intensity instantly breaches the interference safety limit of 0.9, the safety logic immediately takes over control, forcing $r_0$ and $T_{delay}$ to zero instantly, and the fuze returns to a safe state.

The above simulation results indicate that the proposed algorithm can maintain adaptive parameter adjustment across several representative interference and mission-switching conditions considered in this study. These control characteristics cannot be achieved by traditional fuzzy control or single-threshold methods.

5.5. Comparative Analysis of Performance of Different Control Strategies

For fairness, the fixed-threshold strategy and the standard fuzzy controller were tuned to stable representative operating settings under the same simulation environment before comparison. To objectively evaluate the comprehensive advantages of the proposed algorithm, the typical fixed-threshold strategy and standard fuzzy control in traditional the loitering munition fuze are selected as control groups. Under the working conditions set in the Section 5.3 simulation, the response characteristics of different strategies in various tactical phases are compared and examined. Furthermore, random seeds are set to generate 500 groups of random scenarios with different signal-to-noise ratios and interference timings, and the burst height Root Mean Square Error (RMSE) and false alarm rate (FAR) of each strategy are calculated to evaluate the statistical consistency of the algorithms.

5.5.1. Strategy Response Comparative Analysis

This simulation focuses on the physical parameter reconfiguration efficiency of the three methods at the moment of encountering interference. The fuze is in proximity detection mode, and the interference pulse is set to burst at t = 0.5 s.

As shown in Figure 7, the traditional fixed-threshold method lacks the ability for dynamic parameter reconfiguration; even under strong interference conditions, its threshold remains unchanged. After detecting a sudden change in the battlefield environment, the TD3 algorithm proposed in this paper only requires a reconfiguration response time $T_{resp}$ of 12 ms to complete the reconfiguration of the fuze’s key initiation parameters to a safe state. Compared with the traditional fuzzy control ($T_{resp}$ = 88 ms), the response speed is increased by 86.4%. Moreover, the fuzzy control is accompanied by noticeable high-frequency jitter caused by rule switching during the adjustment process, whereas the curve output by the TD3 algorithm is extremely smooth, avoiding detection logic instability caused by severe threshold oscillations. The fixed-threshold strategy completely fails during the interference period. This time-domain responsiveness suggests that the proposed method is more suitable for interference-driven online parameter reconfiguration than the compared baseline strategies.

5.5.2. Statistical Performance Metrics Evaluation

To objectively evaluate the performance differences of each control strategy, this section selects the fixed-threshold model, the standard fuzzy control model, and the TD3-based parameter reconfiguration model proposed in this paper for comparative analysis. Each type of model underwent 500 Monte Carlo target shooting experiments under the same complex battlefield conditions, and the obtained evaluation metrics were processed by arithmetic averaging. The statistical results are shown in Figure 8.

As evidenced by Figure 8, the proposed TD3-based parameter reconfiguration model exhibits clearly improved performance. In terms of Root Mean Square Error, this model can accurately capture state changes under non-stationary interference and reconfigure parameters in real time, with an average RMSE of only 0.43 m. Compared to 2.45 m for the fixed-threshold model and 1.12 m for the traditional fuzzy control model, the prediction accuracy is improved by 82.4% and 61.6%, respectively. In the false alarm rate statistics, this model effectively evades the interference penetration of strong noise pulses through an extremely fast response mechanism (12 ms). The experimental results show that the false alarm rate of this model is only 0.15%, far lower than the 14.2% of the fixed-threshold model and 4.8% of the traditional fuzzy control model.

5.6. Robustness Analysis

5.6.1. Robustness Analysis Under Different Interference Noise Ratios

To evaluate the resistance capability of the proposed TD3 fuze parameter reconfiguration model against sudden strong electromagnetic interference during the attack process, the simulation adopts the combat flow in Section 5.3. Both the variance and mean of the noise are set to 0.1. The selected noise ratios are 30%, 60%, 80%, and 100% of the detection sample set, respectively. Notably, the 100% noise injection simulates an extreme, relentless interference suppression environment. Subsequently, the samples with added noise are input into the trained reconfiguration model, and the comparison between the reconfiguration parameters output by the algorithm and the target’s true distance trajectory is shown in Figure 9.

As shown in the figure, for the core penetration delay parameter reconfiguration phase, Root Mean Square Error (RMSE) and Mean Absolute Error (MAE) are used to quantitatively evaluate the fitting quality. Under 30~100% noise interference, the RMSE of the algorithm in the tracking section always remains at a low level. Considering the large dynamic range of the vertical axis, its relative tracking error is controlled within 2%. Furthermore, under the 80% interference noise condition, the model exhibits the best signal-to-noise ratio balance, with the RMSE reaching the lowest value of 0.512. Even under 100% extreme noise, although the curve fluctuates significantly, the RMSE is only 0.685, without target loss or divergence, indicating that the algorithm maintains acceptable anti-noise reconfiguration capability over the tested noise ratio range. The RMSE and MAE evaluation metrics for proximity distance reconfiguration under different noise ratios are shown in

Table 4. RMSE and MAE evaluation metrics for proximity distance reconfiguration under different noise ratios.

Noise Ratio	RMSE	MAE
30%	0.542	0.431
60%	0.589	0.475
80%	0.512	0.410
100%	0.685	0.544

. Nevertheless, the conclusions remain limited to the simulated interference conditions considered in this study.

5.6.2. Robustness Analysis Under Intermittent Sensor Signal Loss

Under the combat flow defined in Section 5.3, intermittent loss of the airborne sensor signal is introduced to evaluate the robustness of the trained TD3-based parameter reconfiguration policy under temporary observation interruption. The simulation focuses on the penetration-delay mode and considers random signal interruptions of 100 ms, 200 ms, 300 ms, and 400 ms. The simulation results are shown in Figure 10.

To objectively assess the model’s reliability during intermittent observation, the blind-zone RMSE is introduced as a quantitative metric. This metric only counts the reconfiguration deviation during the signal loss period. Under 100 ms and 200 ms signal loss conditions, the reconfiguration errors are 0.542 and 0.615, respectively, which are extremely low and basically on par with the accuracy under normal signal states; under 300 ms and 400 ms signal loss conditions, the reconfiguration errors are 0.892 and 1.265, respectively, showing a weak nonlinear accumulation of prediction error. Even under the prolonged 400 ms sensor signal loss condition, the relative error remains within 5%, indicating that the learned policy retains acceptable tolerance to temporary observation interruption within the tested time window.

5.6.3. Robustness Analysis Under Different Sensor Noise Levels

To further evaluate the influence of measurement uncertainty, additional simulations were conducted under different sensor noise levels. In this test, additive Gaussian noise with standard deviations of 0.1 m, 0.5 m, and 1.2 m was injected into the distance-related sensing channel, corresponding to mild, moderate, and severe sensor noise conditions, respectively. An engineering tracking tolerance band of ±0.8 m was introduced for visual comparison.

The resulting tracking error responses are shown in Figure 11. As can be seen, the proposed method maintains stable tracking behavior under all three tested noise levels, and the error trajectories remain within the tolerance band for most of the engagement interval. Although the tracking fluctuation increases with the sensor noise level, no obvious divergence or unstable oscillation is observed.

The inset RMSE comparison further shows that the RMSE increases from 0.043 m under mild noise to 0.067 m under moderate noise and 0.078 m under severe noise. These results indicate that the proposed controller retains acceptable tolerance to sensor noise within the tested range. It should be noted that this analysis focuses on measurement-noise effects only. Atmospheric disturbances such as wind, pressure, and density variation are not explicitly modeled in the present study and remain part of future work.

6. Conclusions

To overcome the rigid initiation modes, static parameters, and sub-optimal adaptability, this study proposes a multi-modal adaptive reconfiguration control framework for the loitering munition fuze based on deep reinforcement learning. By constructing a hierarchical architecture of outer-loop safety decision making and inner-loop intelligent execution, an online parameter evolution method fusing the TD3 algorithm and fuzzy logic is proposed. The main conclusions are as follows:

1. An online adaptive reconfiguration control method based on TD3–fuzzy is proposed. Circumventing the limitations of expert-reliant static tuning, the TD3 algorithm is used to establish a dynamic mapping between battlefield conditions and underlying control parameters. Simulation results indicate that this method significantly alleviates the parameter solidification problem in strong noise environments, reducing the burst height Root Mean Square Error (RMSE) to 0.43 m, which is an improvement of 82.4% and 61.6% compared to the fixed threshold and standard fuzzy control, respectively, thereby improving initiation control precision of the fuze under complex engagement conditions.

2. The robustness and rapid response capabilities of the system under extreme working conditions are demonstrated in simulation. Simulations confirm that the proposed method possesses millisecond-level online reconfiguration capabilities, with a response time of only 12 ms when facing sudden strong interference, effectively evading false alarm risks and keeping the false alarm rate as low as 0.15%. Simultaneously, the algorithm demonstrates good anti-interference resilience; even under extreme conditions where the sensor signal experiences an intermittent loss of 400 ms, it can still keep the relative tracking error within 5% under the tested interruption window, indicating acceptable tolerance to temporary observation loss.

3. Multi-modal mission switching and full-process reverse safety recovery are achieved. The dual-threshold hysteresis decision model constructed based on the Fuze Readiness Index (FRI) successfully bridges the logical links of multi-modal missions such as proximity, delay penetration, and airburst. Full-process simulations show that this mechanism can not only accurately switch initiation logic according to tactical commands but also achieve reliable reverse safety-state reconfiguration and locking under mission aborts or safety baseline breaches, thereby helping ensure the weapon system’s life-cycle safety and operational versatility.

This study is limited to co-simulation validation and does not yet include hardware-in-the-loop or embedded deployment experiments. In addition, representative learning-rate sensitivity analysis, baseline algorithm comparison, and sensor noise robustness evaluation have been incorporated into the revised manuscript to further support the selection of the proposed control framework. These factors, together with broader sensitivity analysis of threshold parameters and comparison with additional DRL baselines such as SAC, will be addressed in future work.