Dual-Priority Delayed Deep Double Q-Network (DPD3QN): A Dueling Double Deep Q-Network with Dual-Priority Experience Replay for Autonomous Driving Behavior Decision-Making

Comments 1: The lack of line number makes it difficult to pinpoint the location where I would like to leave a comment.

Response 1: Thank you for pointing this out. I agree with this comment. Therefore, I have added line numbers.

Comments 2: Abstract: grammar issue "Compared with the currently popular DQN and DDQN algorithms, the success rates of this algorithm in the challenging."

Response 2: Agree. I have corrected the grammar in the mentioned section. The revision is in the 1st page of the manuscript, abstract section, lines 23 and 24. “Compared with the currently popular DQN and DDQN algorithms, this algorithm achieves higher success rates in challenging scenarios.”

Comments 1: Title should be better connected with the paper, i.e. area of application should be mentioned, in this particular case.

Response 1: Thank you to the reviewers for your valuable feedback. I have revised the title to "DPD3QN: A Dueling Double Deep Q Network with Dual-Priority Experience Replay for Autonomous Driving Behavior Decision-Making".

Comments 2: At the end of the Intro section, a structure of the paper should be provided, with a detailed explanation regarding chapters in the paper.

Response 2: Thank you to the reviewers for your valuable feedback. I have provided a detailed explanation of the chapter contents in the concluding section of the Introduction. The added content is located in the fourth paragraph of page 3, lines 113 to 118. “The remainder of this paper is organized as follows. Section 2 introduces related work in the field of deep reinforcement learning and autonomous driving. Section 3 presents the proposed DPD3QN method, including the network structure and du-al-priority experience replay mechanism. Section 4 describes the simulation environment and experimental setup. Section 5 provides the simulation results and comparative performance analysis. Finally, Section 6 concludes the study and outlines future research directions.”

Comments 3: Related works should be improved by adding more relevant and up-to-date research.

Response 3: Thank you to the reviewers for your valuable feedback. The Related Work section primarily focuses on methods relevant to this study. To strengthen the discussion, several recent research findings have been incorporated into the Introduction and the methodological comparison section of Chapter 3. The additions are located in the second paragraph on page 2 (lines 61–62) and the third paragraph on page 10 (lines 386–387), respectively. “Noisy Deep Q-Network (Noisy DQN)” “Deep Deterministic Policy Gradient (DDPG)”

Comments 4: All abbreviations should be explained before appearing in the text.

Response 4: Thank you to the reviewers for your valuable feedback. All abbreviations in the manuscript have been thoroughly checked and are properly defined upon their first appearance.

Comments 5: Simulation Results Analysis should be extended, and further discussion is needed.

Response 5: Thank you to the reviewers for your valuable feedback. In the experimental simulation section of Chapter 4, the experimental setup has been enriched, with additions located on page 14, first paragraph (lines 538–541). “The simulation environment is shown in Fig. 5. The experimental setup adopts a dual-agent system, consisting of a self-driving vehicle and several environment vehicles that mimic human driving behavior. All vehicles operate based on the previously mentioned motion model and environmental configuration, aiming to study the interactive dynamics under standardized highway conditions.“ Additionally, in the adaptability testing part of the experimental results in Chapter 5, a three-lane highway simulation figure has been incorporated, added on page 19, first paragraph (lines 708–709).

Comments 6: Conclusions should be better connected with obtained results, also, limitations should be discussed in a more detailed manner. Besides, directions for future research should be provided.

Response 6: Thank you to the reviewers for your valuable feedback. The additional content in the conclusion section is located on page 21, paragraph 8, lines 791 to 799. “The realism of the simulation environment is limited, and the generalization capability to real-world scenarios still needs to be validated. There remain gaps com-pared to real road environments in terms of perception errors, behavioral uncertainty, and the complexity of traffic rules. Future research could incorporate higher-fidelity simulation platforms (such as CARLA or LGSVL) or real-world driving data for further validation. The modeling capability for multi-agent interaction needs to be improved. The current method primarily focuses on single-vehicle decision-making tasks and does not fully account for the dynamic impact of complex games and collaborative behaviors among multiple vehicles.”

Comments 1: I find this article to be in a more or less publishable condition, such that the introduction, methodology, results, and conclusion are very well presented. Hence, the article as a whole makes a nice contribution to science. Nevertheless, I notice that there is no discussion part included in the article highlighting such issues as theoretical implications, practical implications, limitations, and recommendations for future research. 

Minor point: On p. 7 there is a paragraph starting with Chinese signs. Since this article is addressing an international audience, I suggest that a translation to English is made, or else that another solution to the problem is found

Response 1: We sincerely thank the reviewer for the positive feedback and constructive suggestions. We have added content to the conclusion section addressing the practical implications, research limitations, and recommendations for future work. “The realism of the simulation environment is limited, and the generalization capability to real-world scenarios still needs to be validated. There remain gaps compared to real road environments in terms of perception errors, behavioral uncertainty, and the complexity of traffic rules. Future research could incorporate higher-fidelity simulation platforms (such as CARLA or LGSVL) or real-world driving data for further validation. The modeling capability for multi-agent interaction needs to be improved. The current method primarily focuses on single-vehicle decision-making tasks and does not fully account for the dynamic impact of complex games and collaborative behaviors among multiple vehicles.” The revision is in the 21st page of the manuscript, eighth paragraph, lines 791 to 799. In addition, we have thoroughly reviewed the characters on page 7 and corrected the Chinese characters accordingly.

Comments 1: Novelty and Ablation. The combination of dueling, double-Q, and prioritized replay is sensible, but it is not clear which component contributes most. An ablation study with exist variants of prioritized replay could be very helpful

Response 1: Thank you for this valuable suggestion. We fully agree that ablation studies can provide deeper insight into the contribution of each component in composite algorithms like DPD3QN. However, in this study, our primary objective was to evaluate the overall effectiveness of the proposed integrated approach in complex highway scenarios, rather than to isolate individual components. As such, we did not conduct separate ablation experiments at this stage. That being said, we recognize the importance of this direction and plan to include a comprehensive ablation study and comparison with alternative prioritized replay mechanisms (e.g., rank-based or stochastic prioritization) as part of our future work. This extension will help further validate the role and impact of each module within the DPD3QN.

Comments 2: Hyperparameter Sensitivity. The paper introduces several weighting factors (α, β, ε) for the dual-priority calculation but does not justify their chosen values or report sensitivity analyses. A brief sweep showing performance versus α and β would help readers understand robustness.

Response 2: Thank you for this valuable suggestion. I have provided explanations for the selection of the hyperparameters used in the dual-priority calculation and conducted a sensitivity analysis. The corresponding revisions can be found on pages 11 and 12 of the manuscript, specifically in paragraphs 7 and 8, and paragraph 1, covering lines 455 to 467. “Moreover, since both the TD error and the segment weights are not inherently bounded, the combined priority value ​ may theoretically exceed 1. To address this, we apply a normalization process during the sampling stage by dividing each priority by the total priority sum ​, resulting in the normalized sampling probability ​​. This ensures that each sample is selected according to its relative importance, regardless of the absolute value. At the same time, it prevents high-priority samples from dominating the training process, thereby reducing the risk of overfitting or biased learning.

This approach not only maintains training efficiency but also enhances the model’s ability to learn from diverse types of scenarios with increased robustness. By adjusting α and β, researchers and developers can fine-tune the training process based on specific task demands and environmental complexities. In future versions, we also plan to include a schematic illustration and a numerical example to further improve the transparency and interpretability of the dual-priority formula.”

Comments 3: Clarity of Dual-Priority Formula. Equation 23 combines TD error priority and segment weights with coefficients α, β.  Explain whether α + β = 1 or if priorities can exceed unity. A schematic or small example calculation would aid understanding.

Response 3: Thank you for this valuable suggestion. The formula does not necessarily require that α+β=1, and the combined priority may also exceed 1. It is recommended to add a schematic diagram or a simple calculation example to enhance the clarity and interpretability of the formula. The added explanatory content is located on page 11, paragraph six, lines 445 to 454 of the manuscript. The calculation example has been added after the first paragraph on page 12, covering lines 467 to 472. “It is important to note that α+β is not strictly constrained to equal 1. This design provides greater flexibility, allowing the relative importance of TD error and segment-based priority to be dynamically adjusted during different training stages. For example, in the early stages of training, when the model has not yet learned an effective policy and the TD error tends to fluctuate significantly, a higher αcan be used to emphasize learning from large-value errors and accelerate policy correction. In contrast, during the later stages—when the policy becomes more stable—β can be increased to direct the model's attention to rare but critical driving scenarios, such as high-speed lane changes or emergency braking. This helps improve the generalization and real-world adaptability of the learned policy.”

Comments 4: Equation (19) redefines the sampling probability with a bias term b; please specify the value of b used in experiments.

Response 4: Thank you for this valuable suggestion. The corresponding revisions are located on page 9, paragraph 4, lines 355 to 356 of the manuscript. “In our experiments, the value of the bias term ? is set to 0.01, which helps avoid zero probability and ensures sufficient sample diversity.”

Comments 1: The authors did not respond to my general comments, which questions the novelty of the study and lack of advanced baseline for case study results comparison.

Response 1: Thank you very much for your thoughtful feedback. We fully understand the concern regarding the level of novelty. While it is true that D3QN and TD-error PER are well-established methods, our contribution focuses on the integration of segment-based priority into the experience replay framework, tailored for autonomous highway scenarios.

This segment priority component, although heuristic in design, introduces task-relevant semantic weighting (e.g., spatial position and velocity categories) that enhances learning in sparse but critical situations. As our current work aims to demonstrate the practical effectiveness of this dual-priority structure in a combined framework, we believe that a full ablation study or formal separation of component-wise contributions is beyond the intended scope of this paper. Nonetheless, we appreciate the suggestion and consider it a valuable direction for future research.

In designing our experimental comparisons, we focused on demonstrating incremental improvements over standard baselines (DQN and DDQN), as these are the most widely used and well-understood reference points in the field. While more advanced configurations such as D3QN or DPER are relevant, our primary objective was to validate the effectiveness of the proposed DPD3QN method in contrast to foundational models.

We believe that the current comparisons are sufficient to support our claims, and including additional variants may dilute the focus of the study. However, we fully acknowledge the value of broader baseline comparisons and will consider them in follow-up studies.

Comments 2: There is still reference error in the revised manuscript (e.g. page 8).

Response 2: Thank you very much for your thoughtful feedback. I have corrected the citation error on page 8. Please refer to the first line of the first paragraph on page 8 for the specific revision.

Comments 1: Thank you for improving your paper

Response 1: Thank you for your review and suggestions.

Comments 1: Thanks for the modification. Please provide the source code to the paper for reproductivity. Please do what you promised.

Response 1: Thank you for your review and suggestions. All the data used in the study are already included in the manuscript, and the corresponding option has been selected during the submission process in the system.