Reinforcement Learning-Enhanced Botnet Defense System in Grid Topology Networks Using the SIRO Framework

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

The manuscript studies reinforcement-learning-based botnet containment in grid-topology networks using a white-hat worm launcher and proposes the SIRO framework, which combines state representation, action masking, reward shaping, and reward optimization. The topic is timely and relevant, particularly for autonomous cyber defense in resource-constrained networked systems. The paper is clearly structured, the implementation details are relatively extensive, and the cross-algorithm evaluation on VPG, DQN, PPO, and MuZero is a positive aspect. The reported gains in winning rate and win efficiency suggest that the proposed workflow is practically useful in the authors’ simulation setting. However, the current version still needs stronger technical validation and a clearer positioning of the novelty. In particular, the present results show that SIRO improves performance in a simplified environment, but they do not yet fully establish that the framework is robust, security-realistic, or generalizable enough for publication without further strengthening.

1. At present, SIRO is presented as a four-part framework, but its main ingredients, state stacking, action masking, reward shaping, and hyperparameter optimization, are already standard tools in reinforcement learning practice. The manuscript would be stronger if it explicitly stated what is genuinely new here: for example, whether the contribution lies in the particular cyber-defense formulation, the Chebyshev-based masking strategy, or the combination of these elements for white-hat worm deployment. As written, the paper risks reading as a careful engineering integration rather than a clearly novel method.
2. The comparison is mainly against a deliberately weak “naive RL” setup that uses only illegal-move masking and hand-tuned rewards. This makes it difficult to separate the contribution of each SIRO component from the effect of comparing against an under-optimized baseline. A stronger study would include intermediate baselines such as action masking only, Optuna reward optimization only, state redesign only, and full SIRO. That would be more convincing than comparing only against a weak reference and then performing a reward-only ablation.
3. The current environment assumes a deterministic propagation order, a single botnet insertion at step 0, a binary infection threshold (r < 0.5), fixed local spreading rules, and a non-adaptive attacker. These assumptions are understandable for a first study, but they substantially limit the security realism of the conclusions. The paper should be more explicit that this is a stylized benchmark rather than a faithful operational threat model. This would also be a good place to cite Enhancing Cyber-Resiliency of DER-Based Smart Grid: A Survey, especially to position the work against broader ideas such as layered defense, threat modeling, and recovery.
4. As written, Equation (7) defines the terminal loss case as lose = lose_penalty, while Table 3 assigns lose_penalty = 1. Combined with Equation (10), this appears to add a positive terminal reward even when the agent loses. If this is only a notation issue, it should be corrected immediately; if it reflects the actual implementation, it materially affects the validity of the results. More generally, the sign convention for all reward terms should be clarified with one short worked example.
5. The manuscript is detailed, but several points still need tightening. In Appendix A, Algorithm A1 appears to use bounds such as 0 <= x+1 < m-1 and 0 <= y+1 < n-1, which would exclude valid boundary neighbors if interpreted literally. That should be checked carefully. In addition, the scalability claim is based only on DQN, while the paper motivates SIRO as a framework for multiple RL families. The authors should either soften the general scalability claim or provide evidence for at least one more algorithm.
6. The proposed defense relies heavily on learned policies, yet the paper treats the defender as trustworthy once trained. That is a weak assumption in adversarial settings. A useful improvement would be to discuss whether the observation tensor, reward process, or trained policy could itself be manipulated, for example through poisoned training episodes or adversarial state perturbations. This is a natural place to cite Vulnerability of Machine Learning Approaches Applied in IoT-Based Smart Grid: A Review, which would strengthen the cybersecurity framing and make the discussion more balanced.

Author Response

We appreciate the reviewer comments and the time they have taken to read the manuscript. The following is the comment from the reviewer and how we address it in the current revision.

Question 1:

At present, SIRO is presented as a four-part framework, but its main ingredients, state stacking, action masking, reward shaping, and hyperparameter optimization, are already standard tools in reinforcement learning practice. The manuscript would be stronger if it explicitly stated what is genuinely new here: for example, whether the contribution lies in the particular cyber-defense formulation, the Chebyshev-based masking strategy, or the combination of these elements for white-hat worm deployment. As written, the paper risks reading as a careful engineering integration rather than a clearly novel method.

Answer 1:

We agree with the comment about further clarifying the manuscript contribution to avoid it looking like careful engineering integration. To address the concern, we highlight the contribution of our work to the Chebyshev-based masking strategy used in the SIRO framework.

Page 2, lines 67-71:

The main contribution is the implementation of a Chebyshev-based masking strategy within the SIRO framework to reduce the complexity in discrete action space for RL to solve. The framework is tested on vanilla policy gradient (VPG), deep Q-network (DQN), proximal policy optimization (PPO), and MuZero to highlight its effectiveness on different algorithms [18]. Furthermore, scalability and ablation studies were conducted.

Question 2:

The comparison is mainly against a deliberately weak “naive RL” setup that uses only illegal-move masking and hand-tuned rewards. This makes it difficult to separate the contribution of each SIRO component from the effect of comparing against an under-optimized baseline. A stronger study would include intermediate baselines such as action masking only, Optuna reward optimization only, state redesign only, and full SIRO. That would be more convincing than comparing only against a weak reference and then performing a reward-only ablation.

Answer 2:

We agree with the comment to expand the ablation study to cover Survey, Identify, Reinforce, and Optimize. We reworked the ablation study in Section 4.4, page 21 to include:

1. DQN with Survey component only
2. DQN with Identify component only
3. DQN with Reinforce component only
4. DQN with Optimize component only
5. DQN with survey and Identify
6. DQN with Survey, Identify, and Reinforce
7. DQN with Survey, Identify, Reinforce, and Optimize.

Question 3:

The current environment assumes a deterministic propagation order, a single botnet insertion at step 0, a binary infection threshold (r < 0.5), fixed local spreading rules, and a non-adaptive attacker. These assumptions are understandable for a first study, but they substantially limit the security realism of the conclusions. The paper should be more explicit that this is a stylized benchmark rather than a faithful operational threat model. This would also be a good place to cite Enhancing Cyber-Resiliency of DER-Based Smart Grid: A Survey, especially to position the work against broader ideas such as layered defense, threat modeling, and recovery.

Answer 3:

We agree with the comment to clarify that the current infection model is simplified and deterministic to avoid confusion with more advanced infection models and real-world scenarios. To address this issue, we have added the statement that the network simulation and infection model is simplified and deterministic.

Page 5, lines 118-121:

The simulated network and infection model are simplified and deterministic, where the infection state is s∈{0, 1, 2}, the device resistance is defined as 0≤r≤1, and comparison operators decide if the botnet’s infectability exceeds the device resistance, as shown in Figure 4.

Additionally, we added the citation into the following sentences.

Page 6, lines 129-132

While the propagation follows the same concept as other infection models [27–30], the infection model implemented in this article is simplified to infect neighboring devices where r<0.5. The stylized implementation is to speed up the simulation and shorten the time taken to train RL.

Question 4:

As written, Equation (7) defines the terminal loss case as lose = lose_penalty, while Table 3 assigns lose_penalty = 1. Combined with Equation (10), this appears to add a positive terminal reward even when the agent loses. If this is only a notation issue, it should be corrected immediately; if it reflects the actual implementation, it materially affects the validity of the results. More generally, the sign convention for all reward terms should be clarified with one short worked example.

Answer 4:

We appreciate and thank the reviewer for noticing the error with the negative sign for the lose_penalty and presence_scale. We established that the error in the notations has no effect on our experiments because the penalty values are correctly configured as negative. We added the Sign column in Table 3 (page 13) and Table 4 (page 14) to better clarify the corrected sign for incentives and penalties.

While substituting Equation 10 on page 3 with a single short example may make the reader's understanding better, we chose not to do so because Equation 10 is shared by the rewards and penalties coefficients in Table 4.

Question 5:

The manuscript is detailed, but several points still need tightening. In Appendix A, Algorithm A1 appears to use bounds such as 0 <= x+1 < m-1 and 0 <= y+1 < n-1, which would exclude valid boundary neighbors if interpreted literally. That should be checked carefully. In addition, the scalability claim is based only on DQN, while the paper motivates SIRO as a framework for multiple RL families. The authors should either soften the general scalability claim or provide evidence for at least one more algorithm.

Answer 5:

1. We appreciate and thank the reviewer for noticing the error with our pseudocodes. To address the issues, we have corrected the bound checking to exclude -1 in Algorithm A2 (previously known as Algorithm A1) and Algorithm A4 (previously known as Algorithm A2).

Page 26-27, Appendix A

1. We agree that the scalability of the SIRO framework should be softened, as the current setup only tested on DQN. We address this by specifically mentioning that the scaling is only tested on DQN.

Page 21, lines 377-383

The winning rate for DQN with the SIRO framework increases as the network size grows because larger grids provide more opportunities for DQN to place the white-hat worms, which help eliminate the botnet and prevent it from reaching the boundary devices.  The oscillation of the area under the learning curves on L(21, 21) is due to a limitation of the ML model in DQN itself, not the SIRO framework. Although the input and output of the ML model are designed to dynamically adjust to the network size, the number of layers remains constant, which could hinder DQN with the SIRO framework from scaling up.

Question 6:

The proposed defense relies heavily on learned policies, yet the paper treats the defender as trustworthy once trained. That is a weak assumption in adversarial settings. A useful improvement would be to discuss whether the observation tensor, reward process, or trained policy could itself be manipulated, for example through poisoned training episodes or adversarial state perturbations. This is a natural place to cite Vulnerability of Machine Learning Approaches Applied in IoT-Based Smart Grid: A Review, which would strengthen the cybersecurity framing and make the discussion more balanced.

Answer 6:

We appreciate the suggestions to consider how botnet could utilize adversarial attacks to trick the agents trained with falsified data. We have added this as future work in Section 6: Conclusion.

Page 25, lines 492-495

In future work, we aim to expand the RL-based BDS launcher strategies to handle multiple botnet insertions and evaluate their performance on scale-free networks, including the use of partitioning techniques and addressing adversarial attacks [43].

Quality of English Language:

We appreciate the comment on improving the quality of English language. We have reworked some of the sentences (highlighted in yellow colors) and checked the grammar using Quillbot. 

Reviewer 2 Report

Comments and Suggestions for Authors

I strongly have few suggestions to improve manuscript before acceptance.

• Rewrite the contribution statement so SIRO is presented as a framework for RL design in this simulation setting, not a broadly validated defense system.

• Add stronger baselines beyond the in-house naive RL setup.

• Expand the ablation study to cover Survey, Identify, and Optimize, not only reward coefficients.

• Increase the number of runs and add significance testing or confidence intervals.

• Clarify that the current infection model is simplified and deterministic, and reduce real-world claims accordingly.

• Separate demonstrated results from future-work ideas on partitioning, GNNs, and scale-free networks.

• Make code, seeds, and environment configuration publicly available if possible.

• Perform a full language edit and fix figure/table typos.

• Add a short paragraph on the ethics and governance of white-hat worm deployment.

• Recheck the conclusion so it does not claim more than the experiments support

Author Response

We appreciate the reviewer comments and the time they have taken to read the manuscript. The following is the comment from the reviewer and how we address it in the current revision.

Question 1:

Rewrite the contribution statement so SIRO is presented as a framework for RL design in this simulation setting, not a broadly validated defense system.

Answer 1:

We agree with the comment to clarify SIRO as a framework for RL design and highlight the research contribution. However, as noted by another reviewer, the SIRO framework is composed of standard practices used in reinforcement learning, which weakens the originality. To address these concerns, we highlight the contribution of our work to Chebyshev-based masking strategy used in the SIRO framework.

Page 2, lines 67-71:

The main contribution is the implementation of a Chebyshev-based masking strategy within the SIRO framework to reduce the complexity in discrete action space for RL to solve. The framework is tested on vanilla policy gradient (VPG), deep Q-network (DQN), proximal policy optimization (PPO), and MuZero to highlight its effectiveness on different algorithms [18]. Furthermore, scalability and ablation studies were conducted.

Question 2:

Add stronger baselines beyond the in-house naive RL setup.

Answer 2:

We appreciate the reviewer’s suggestion to make the comparison more robust and fairer. However, adding some of the components from the SIRO framework, which was done in the previous manuscript, could cause confusion about how much the component in SIRO influences the result, which is discussed more in the ablation study. Additionally, a Manhattan-based masking strategy does not guarantee simulation solvability in L(7, 7). To address the issues, we reworked the naïve RL experiment to not use any components from the SIRO framework by masking only illegal moves, representing the current infection state as a single-channel input tensor, and using only the terminal reward of 1 for winning and -1 for losing.

Page 16, lines 300-305:

To compare the performance against the SIRO framework, a naive RL implementation for the agents using VPG, DQN, PPO, and MuZero was constructed, where only illegal moves are masked, the current infection state is represented as a single channel tensor, and a terminal reward of 1 for winning and −1 for losing is used. These restrictions are used to emulate the common pitfall in designing RL agents [36] and highlight what is lacking in the absence of the SIRO framework.

For a fairer comparison, we reworked the ablation study in Section 4.4, page 21 to include:

1. DQN with Survey component only
2. DQN with Identify component only
3. DQN with Reinforce component only
4. DQN with Optimize component only
5. DQN with survey and Identify
6. DQN with Survey, Identify, and Reinforce
7. DQN with Survey, Identify, Reinforce, and Optimize.

Question 3:

Expand the ablation study to cover Survey, Identify, and Optimize, not only reward coefficients.

Answer 3:

We agree with the comment to expand the ablation study to cover Survey, Identify, Reinforce, and Optimize. In page 21, Section 4.4, we also added the following comparison:

1. DQN with survey and Identify
2. DQN with Survey, Identify, and Reinforce
3. DQN with Survey, Identify, Reinforce, and Optimize.

Question 4:

Increase the number of runs and add significance testing or confidence intervals.

Answer 4:

We appreciate the reviewer’s suggestions to increase the number of runs and add significance testing or confidence intervals to increase the reliability of our experimental results. However, conducting significance testing or confidence intervals requires at least 20 runs or more for each experiment, which is not doable with our current computational setup. As a middle ground, we use mean with range band to show the minimum and maximum values achievable in Section 4.2. For Section 4.3 and Section 4.4, mean with standard deviation is used instead.

Question 5:

Clarify that the current infection model is simplified and deterministic, and reduce real-world claims accordingly.

Answer 5:

We agree with the comment to clarify that the current infection model is simplified and deterministic to avoid confusion with more advanced infection models and real-world scenarios. To address this issue, we have added the statement that the network simulation and infection model is simplified and deterministic.

Page 5, lines 118-121:

The simulated network and infection model are simplified and deterministic, where the infection state is s∈{0, 1, 2}, the device resistance is defined as 0≤r≤1, and comparison operators decide if the botnet’s infectability exceeds the device resistance, as shown in Figure 4.

Page 6, lines 129-132

While the propagation follows the same concept as other infection models [27–30], the infection model implemented in this article is simplified to infect neighboring devices where r<0.5. The stylized implementation is to speed up the simulation and shorten the time taken to train RL.

Question 6:

Separate demonstrated results from future-work ideas on partitioning, GNNs, and scale-free networks.

Answer 6:

We appreciate the feedback to separate the results from the future work. The future work on partitioning, GNNs, and scale-free networks is in Section 5: Discussion.

Question 7:

Make code, seeds, and environment configuration publicly available if possible.

Answer 7:

We appreciate the interest in making the code, seeds, and environment configuration publicly available. We aim to improve the environment's source code and release it to the public with the goal of making this research more accessible in the future. While the current code, seeds, and environment configuration work, they need to be refactored with supporting documentation to be prepared. In this stage of our research, we decided to share the raw data supporting the conclusions of this article will be made available on request.

Question 8:

Perform a full language edit and fix figure/table typos.

Answer 8:

We appreciate the comment on improving the quality of English language. We have reworked some of the sentences (highlighted in yellow colors) and checked the grammar using Quillbot.

Question 9:

Add a short paragraph on the ethics and governance of white-hat worm deployment.

Answer 9:

We appreciate the suggestions to add on ethics. In the updated manuscript, we clarified that the Algorithms A1 and A2 only simulate the propagation of botnets and white-hat worms on a 2D array, with no malicious code included to adhere to the EC-Council Code of Ethics.

Page 6, lines 137-139

These algorithms only simulate the propagation of botnets and white-hat worms on a 2D array, with no malicious code included to adhere to the EC-Council Code of Ethics [31].

Question 10:

Recheck the conclusion so it does not claim more than the experiments support

Answer 10:

We fully agree that the conclusion should not claim more than what is supported in the experiment. To reframe our conclusion on scaling, we specify that the scaling experiment is done using DQN-based agent only, while removing the claim that grid partitioning is proven to extend beyond the limit.

Page 25, lines 476-477

Furthermore, a DQN-based agent that utilizes the SIRO framework can be scaled up to L(20, 20).

Reviewer 3 Report

Comments and Suggestions for Authors

The paper addresses the exposure of grid topology networks to the spread of malicious botnet infections across grid nodes by leveraging the botnet's ability to move from node to node. Although the issue is known and there are already solutions to overcome it, such as reinforcement learning. However, its performance is suboptimal due to the time required to eliminate this threat. The author proposes an efficient framework that achieves a 36.5% improvement in the winning rate, with win efficiency improving by 59.1863%.  

Summary:

The paper is well presented. The figures provide an intuitive presentation of the proposed solution.

The authors comprehensively support their claim by presenting several comparison tables that demonstrate the superiority of the LH3SC scheme over other existing schemes. 

 Comments:

1. Algorithms 1, 2, and 3 may be better moved to the appendix. Instead, a short description or a figure suffices

Author Response

We appreciate the reviewer comments and the time they have taken to read the manuscript. The following is the comment from the reviewer and how we address it in the current revision.

Comment 1:

Algorithms 1, 2, and 3 may be better moved to the appendix. Instead, a short description or a figure suffices.

Answer 1:

We agree with the comment to improve the readability of our manuscript by moving the algorithms to the appendix. We have moved Algorithms 1 and 2 into the appendix and replaced the blank spaces in the main text with flowcharts (Figures 7 and 18) instead. For Algorithm 3, we find it difficult to represent the operations as a flowchart because some programming expressions are too complex. Additionally, simplifying the flowchart will lose some of the context needed to understand the operation. We decided to retain the original placement for Algorithm 3 (in the new manuscript, it is named Algorithm 1) and rework the explanations.

Page 6, lines 133-137

Figure 7 shows the process for simulating the botnets’ and white-hat worms’ infection, with resilience_map and in infection_map denoting the 2D arrays of device resiliency and infection state maps. The detailed implementation is defined in Appendix A, which includes the infection simulation in Algorithm A2 and the Manhattan distance of 1 in Algorithm A2.

Quality of English Language:

We appreciate the comment on improving the quality of English language. We have reworked some of the sentences (highlighted in yellow colors) and checked the grammar using Quillbot. 

Round 2

Reviewer 2 Report

Comments and Suggestions for Authors

Although authors tried to address several points but still paper is far from publication stage.  So there are some more suggestions to improve before publications. 

1.  Your own ablation shows that Identify only is the strongest single component, while Survey only can under-perform. Rewrite the contribution narrative so it matches that evidence. 
2. The positioning against prior work is still not sufficiently clear. The introduction mentions earlier supervised-learning-based and reinforcement-learning-based botnet defense systems, including the limitation that prior RL required more training episodes, but the paper should explain more explicitly what this manuscript adds beyond those earlier studies and why the advance is not merely incremental.
3. The experimental baseline is too weak for a strong comparative claim. The main comparison uses a “naive RL implementation” with only illegal-move masking, single-channel input, and terminal reward only. While such a comparison is acceptable as an initial sanity check, it is not enough to establish superiority. The paper would be stronger if it included comparisons against stronger baselines such as Manhattan-based masking, reward-shaped RL without the full framework, or the authors’ earlier RL launcher.
4. The simulator design is highly simplified and deterministic, which limits external validity. The paper explicitly states that the infection model is simplified, deterministic, uses a threshold rule for infection, and follows a fixed sequence of actions at each timestep. These choices may be reasonable for controlled experiments, but the manuscript should more carefully state that the reported gains are shown in a stylized environment rather than a realistic cyber-defense deployment.
5. Reproducibility should be improved. For a paper of this type, it would be highly beneficial to release the simulator, training code, seeds, model configurations, and Optuna settings. Without that, the work is harder to verify and extend. The current manuscript should include a stronger data/code availability statement.

Author Response

We appreciate the reviewers' comments and the time they have taken to verify the manuscript. The following is the comment from the reviewer and how we address it in the current revision.

Question 1:

Your own ablation shows that Identify only is the strongest single component, while Survey only can under-perform. Rewrite the contribution narrative so it matches that evidence. 

Answer 1:

We agree that the contribution narrative should reflect the ablation evidence. We have revised the Abstract and Introduction to explicitly position Chebyshev-based masking strategy (Identify) as the primary technical contribution, and we clarify that Survey is most effective when combined with other components rather than as a standalone modification

Page 1, lines 14-16:

An ablation study indicates that the masking used in Identify accounts for the majority of the improvement, whereas multi-channel in Survey alone can reduce performance without complementary masking, rewards, and optimization.

Page 3, lines 68-71:

The main contribution is the implementation of a Chebyshev-based masking strategy used in the Identify component to reduce the complexity. Consistent with our ablation results, Identify contributes the largest performance gain, while Survey provides benefits primarily in combination with other components due to added complexity.

Question 2:

The positioning against prior work is still not sufficiently clear. The introduction mentions earlier supervised-learning-based and reinforcement-learning-based botnet defense systems, including the limitation that prior RL required more training episodes, but the paper should explain more explicitly what this manuscript adds beyond those earlier studies and why the advance is not merely incremental.

Answer 2:

We agree that the positioning against prior work should be improved. To address this issue, we explicitly mentioned that compared to our previous work that relies on single-channel input, illegal-move masking, and suboptimal rewards, the proposed work redesigns the observation, action, reward, and tuning to improve the learning curve.

Page 3, lines 74-76:

Compared to our previous works [16], which relies on single-channel input, illegal-move masking, and suboptimal rewards, SIRO systematically redesigns observation, action, reward, and tuning to improve the learning curve.

Question 3:

The experimental baseline is too weak for a strong comparative claim. The main comparison uses a “naive RL implementation” with only illegal-move masking, single-channel input, and terminal reward only. While such a comparison is acceptable as an initial sanity check, it is not enough to establish superiority. The paper would be stronger if it included comparisons against stronger baselines such as Manhattan-based masking, reward-shaped RL without the full framework, or the authors’ earlier RL launcher.

Answer 3:

We agree that expanding the comparison with naïve RL implementation remains weak. To address this issue, we replaced the naive RL implementation with single-channel input, illegal-move masking, and suboptimal rewards used in our previous RL launcher and compared it to the SIRO framework in Section 4.2 (pages 16-19) and Appendix C (page 29). Meanwhile, the full rewards and penalties designs are discussed in Appendix B (page 28).

Question 4:

The simulator design is highly simplified and deterministic, which limits external validity. The paper explicitly states that the infection model is simplified, deterministic, uses a threshold rule for infection, and follows a fixed sequence of actions at each timestep. These choices may be reasonable for controlled experiments, but the manuscript should more carefully state that the reported gains are shown in a stylized environment rather than a realistic cyber-defense deployment.

Answer 4:

We agree that the simulator is stylized and is intended for controlled evaluation rather than direct deployment realism. While Section 3.1 already describes the environment as simplified and deterministic with a threshold infection rule and fixed per-timestep action sequence, we have strengthened the manuscript by adding statements in the Abstract and Discussion clarifying that the reported gains are demonstrated in a stylized grid-topology simulation.

Page 1, lines 11-14:

Four reinforcement learning algorithms are implemented to evaluate the framework, which are vanilla policy gradient, deep Q–network, proximal policy optimization, and MuZero in a stylized grid topology network simulation.

Page 1, lines 16-17:

With the mean winning rate improved by 49.129% and mean win efficiency improved by 118.8031% against our previous work, the framework effectiveness is confirmed in stylized simulations.

Page 25, lines 472-473:

In this article, the SIRO framework is proposed to improve the RL-based BDS launcher to eliminate botnet infections in the stylized grid-topology network simulation.

Question 5:

Reproducibility should be improved. For a paper of this type, it would be highly beneficial to release the simulator, training code, seeds, model configurations, and Optuna settings. Without that, the work is harder to verify and extend. The current manuscript should include a stronger data/code availability statement.

Answer 5:

We have uploaded the simulator, training codes, the seed (self-contained in the code), model configurations, and Optuna to the Github, with complete experiment data in the Releases page. Additionally, the data availability statement was updated to reflect this change.

Page 25, lines 507-508:

Data Availability Statement: The original data presented in this study are openly available in GitHub at: https://github.com/yamaguchishingo/siro_mdpi2026