Adaptive Traffic Signal Control Using Multi-Agent Reinforcement Learning: A Comparison of Control Strategies

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

This paper is entitled “Adaptive Traffic Signal Control Using Multi-Agent Reinforcement Learning: A Comparison of Control Strategies.” In this case, the paper's ideas and results are interesting, but the following comments can be used to improve it in the future.

 

Abstract

• Some methodological details are excessive for an abstract, particularly the detailed descriptions of SUMO, Python, TraCI, and the specific corridor. The abstract would benefit from a more concise presentation.
• The abstract mentions comparison among control strategies, but the specific novelty and research gap addressed by the study should be stated more clearly. For example:
  1. Why is comparing Tabular Q-Learning and MADQN important?
  2. What limitation in previous MARL traffic-control studies does this work address?
• The inclusion of both a synthetic corridor and a real-world digital twin is a strong point of the study. However, the abstract should explain more clearly why this dual evaluation framework is important.
• The abstract contains several detailed comparative statements between Tabular Q-Learning and MADQN. These findings may be condensed into a more focused summary emphasizing the key conclusion.
• The concluding statement: “Overall, the study confirms the potential of MARL as a scalable and adaptive framework...” is broad and expected. A stronger concluding sentence could better emphasize the comparative insights and operational recommendations derived from the study.

 

Algorithmic Implementation

• The section should provide more information regarding:
  1. cycle length selection,
  2. phase timing configuration,
  3. coordination logic between intersections,
  4. and whether optimization methods (e.g., Webster) were used for baseline calibration.
• Since the fixed-time controller serves as the benchmark, the manuscript should clarify whether the baseline was carefully optimized to ensure fair comparison with RL methods.
• The manuscript should acknowledge known MARL issues such as:
  1. non-stationarity,
  2. partial observability,
  3. policy instability,
  4. and local-vs-global optimization conflicts.
• The Tabular Q-Learning implementation is clearly described However, the manuscript should justify:
  1. why tabular methods were retained despite scalability limitations,
  2. and under what traffic/network conditions tabular methods remain practical.
• The paper mentions state compression and discretization, but the exact discretization logic is insufficiently explained. The authors should clarify:
  1. number of bins,
  2. discretization thresholds,
  3. and sensitivity to discretization granularity.
• The neural-network architecture is adequately described However, the rationale behind selecting:
  1. three hidden layers,
  2. neuron sizes (256–256–128),
  3. and ReLU activation

should be explained more clearly.

 

• The manuscript lacks training stability discussion. Since DRL training can be unstable, the authors should discuss:
  1. convergence behavior,
  2. reward oscillation,
  3. and training reproducibility.

 

• The section should provide more implementation details regarding:
  1. replay-memory size,
  2. target-network update frequency,
  3. mini-batch size,
  4. optimizer selection,
  5. and training episodes.

Evaluation & Results

• The evaluation methodology is generally appropriate, However, additional clarification is needed regarding:
  1. number of simulation runs,
  2. random seed control,
  3. variability between runs,
  4. and whether reported values represent averages or single experiments.
• Although performance improvements are reported, no statistical significance analysis is provided. The manuscript should include:
  1. confidence intervals,
  2. standard deviation/error bars,
  3. or statistical hypothesis testing.
• The manuscript should explain whether:
  1. all controllers received equal training time,
  2. identical demand conditions,
  3. and comparable optimization effort.
• The manuscript should explain why:
  1. MADQN performs better under low-demand and queue-dissipation scenarios,
  2. while Tabular Q-Learning remains competitive under higher demand conditions.
• Since RL agents are trained within the same simulation environment used for testing, the manuscript should discuss:
  1. generalization capability,
  2. robustness to unseen traffic patterns,
  3. and possible overfitting to corridor-specific dynamics.

 

Author Response

 A detailed response to the reviewers' comments:

 

The following sections provide a detailed response to each reviewer's comment. Our answers are presented in italic, bold font.

 

 

Reviewer #1:

 

Abstract

• Some methodological details are excessive for an abstract, particularly the detailed descriptions of SUMO, Python, TraCI, and the specific corridor. The abstract would benefit from a more concise presentation.
• The abstract mentions comparison among control strategies, but the specific novelty and research gap addressed by the study should be stated more clearly. For example:
1. Why is comparing Tabular Q-Learning and MADQN important?
2. What limitation in previous MARL traffic-control studies does this work address?
• The inclusion of both a synthetic corridor and a real-world digital twin is a strong point of the study. However, the abstract should explain more clearly why this dual evaluation framework is important.
• The abstract contains several detailed comparative statements between Tabular Q-Learning and MADQN. These findings may be condensed into a more focused summary emphasizing the key conclusion.
• The concluding statement: “Overall, the study confirms the potential of MARL as a scalable and adaptive framework...” is broad and expected. A stronger concluding sentence could better emphasize the comparative insights and operational recommendations derived from the study.

 

We thank the reviewer for this helpful comment. We agree that the original abstract contained excessive methodological detail and did not sufficiently foreground the study's specific research gap and comparative contribution. Accordingly, we revised the abstract to be more concise, reduced implementation-level details such as the explicit mention of SUMO, Python, and TraCI, and clarified why the comparison between Multi-Agent Tabular Q-Learning and MADQN is important. The revised abstract now emphasizes that previous MARL-based traffic-signal-control studies often focus on either isolated/synthetic networks or deep-learning architectures without directly comparing tabular and deep value-based multi-agent controllers under the same demand and network conditions. We also clarified the importance of evaluating the controllers on both a synthetic corridor and a real-world digital twin, since this allows controlled algorithmic comparison while also testing robustness under realistic geometry and asymmetric traffic conditions. The concluding sentence was also strengthened to highlight the operational implication that controller selection should depend on demand level, network complexity, and deployment objectives rather than assuming that a deeper MARL architecture is always superior.

 

Algorithmic Implementation

• The section should provide more information regarding:
1. cycle length selection,
2. phase timing configuration,
3. coordination logic between intersections,
4. and whether optimization methods (e.g., Webster) were used for baseline calibration.

We thank the reviewer for this valuable comment. We have revised the fixed-time baseline description to provide explicit details on cycle length selection, phase timing configuration, coordination logic, and the role of Webster's method. In the synthetic two-junction network, both Node2 and Node3 used identical fixed-time programs composed of four 42 s green phases, each followed by a 3 s yellow transition, resulting in a total cycle length of 180 s. Since both intersections used an offset of 0 s, the system was synchronized by identical cycle structure but did not include explicit progression-based offset coordination. For the 25 January real-world corridor, the southern intersection used a 111 s cycle, whereas the northern intersection used a 109 s cycle. Because these cycles were mismatched, the real-world fixed-time baseline was treated as isolated/independent fixed-time control rather than a coordinated arterial progression system. We also clarified that Webster's method was not used to optimize or calibrate the implemented fixed-time signal timings. Instead, Webster's delay formulation was used only as a theoretical reference for interpreting fixed-time delay behavior, while the operational baseline remained the predefined SUMO fixed-time signal program.

• Since the fixed-time controller serves as the benchmark, the manuscript should clarify whether the baseline was carefully optimized to ensure fair comparison with RL methods.

We thank the reviewer for this important comment. We have revised Sections 3.2.3 and 4.1 to clarify the status of the fixed-time benchmark and to avoid implying that it was optimized using Webster's method. The fixed-time controller was implemented as a predefined static SUMO signal program rather than as an adaptive or Webster-optimized baseline. The synthetic two-junction network used identical 180 s cycles at both intersections with zero offset, whereas the 25 January corridor used mismatched cycle lengths of 111 s and 109 s, so it was treated as an isolated/independent fixed-time configuration rather than a fully coordinated arterial progression system. We also clarified that Webster's delay model was used only as a theoretical reference for interpreting fixed-time delay behavior, while the operational benchmark in all simulations remained the predefined SUMO fixed-time controller. To ensure a fair comparison, all controllers were evaluated under the same network geometry, detector configuration, demand files, turning-movement assumptions, simulation horizon, and performance metrics.

• The manuscript should acknowledge known MARL issues such as:
1. non-stationarity,
2. partial observability,
3. policy instability,
4. and local-vs-global optimization conflicts.

We thank the reviewer for this important comment. We have revised the manuscript to explicitly acknowledge key MARL challenges, including non-stationarity, partial observability, policy instability, and local-versus-global optimization conflicts. A dedicated clarification was added in Section 4.5, where the CTDE coordination strategy is described, and an additional limitation paragraph was added in Section 6 before the future-research paragraph. We also clarified that replay memory, target-network stabilization, ε-greedy exploration decay, and the SMDP minimum-green lock were used to improve training stability, while acknowledging that these mechanisms reduce but do not fully eliminate the inherent challenges of decentralized MARL.

• The Tabular Q-Learning implementation is clearly described However, the manuscript should justify:
1. why tabular methods were retained despite scalability limitations,
2. and under what traffic/network conditions tabular methods remain practical.

We thank the reviewer for this helpful comment. We have revised the manuscript to justify the retention of Tabular Q-Learning despite its known scalability limitations. We clarified that Tabular Q-Learning was included as an interpretable, low-complexity value-based MARL benchmark against MADQN, allowing us to assess whether a deeper architecture is always necessary under equivalent traffic conditions. We also added that tabular methods remain practical when the number of intersections is limited, the action space is small, traffic states can be meaningfully discretized, and demand patterns are relatively structured. This clarification was added in Section 4.3, and the practical interpretation was reinforced in Section 5.3, where the results already show that Tabular Q-Learning remained competitive in simple and moderately complex networks with manageable state-action spaces

• The paper mentions state compression and discretization, but the exact discretization logic is insufficiently explained. The authors should clarify:
1. number of bins,
2. discretization thresholds,
3. and sensitivity to discretization granularity.

We thank the reviewer for this valuable comment. We have revised Section 3.3 to explicitly describe the state-compression and discretization procedure used for Tabular Q-Learning. The revised manuscript now states that detector-level halting vehicle counts were first aggregated by incoming edge and then discretized into four queue-density bins: 0 vehicles, 1–9 vehicles, 10–18 vehicles, and more than 18 vehicles. The current signal phase indices were then concatenated with the discretized queue bins to form the tabular state representation. We also clarified that this discretization was selected to balance traffic-state resolution and Q-table tractability. Finally, we acknowledged that a full sensitivity analysis of alternative bin thresholds was not performed and added this as a future research direction.

• The neural-network architecture is adequately described However, the rationale behind selecting:
1. three hidden layers,
2. neuron sizes (256–256–128),
3. and ReLU activation

should be explained more clearly.

We thank the reviewer for this helpful comment. We have revised Section 4.4 to clarify the rationale for the fully connected DQN/MADQN architecture and to avoid implying that a single neural-network size was used universally across all experiments. The revised manuscript explains that compact multilayer perceptrons were selected to balance nonlinear function approximation with computational efficiency during SUMO–TraCI online training. The hidden layers were used to extract nonlinear relationships from detector-based queue states, while progressively smaller layers compressed the learned representation before outputting Q-values for the binary maintain/switch action space. ReLU activation was used because it is computationally efficient, supports stable gradient propagation, and is widely used in DQN-based value-function approximation. We also clarified that these architectures were practical baseline designs rather than globally optimized neural-network configurations.

 

• The manuscript lacks training stability discussion. Since DRL training can be unstable, the authors should discuss:
1. convergence behavior,
2. reward oscillation,
3. and training reproducibility.

We thank the reviewer for this important comment. We have revised Section 4.6 to discuss DRL training stability more explicitly. The revised manuscript explains that training was performed in a fully online setting over a 10,000-step SUMO horizon and that learning behavior was monitored descriptively using the evolution of cumulative reward, queue length, delay, and TTC traces recorded every 100 simulation steps. We also clarified that reward oscillations are expected during early online training because the agents use ε-greedy exploration, stochastic Poisson vehicle arrivals, and simultaneous multi-agent learning. Replay memory, randomized mini-batch sampling, target-network updates, ε-decay, and the SMDP minimum-green lock were used to reduce unstable value updates and unrealistic switching. However, we also acknowledge that these mechanisms reduce but do not eliminate training variability, and that future work should report multiple-seed convergence curves and confidence intervals.

• The section should provide more implementation details regarding:
1. replay-memory size,
2. target-network update frequency,
3. mini-batch size,
4. optimizer selection,
5. and training episodes.

We thank the reviewer for this helpful comment. We have revised Section 4.6 to provide explicit implementation details for the DQN/MADQN training process. The revised manuscript now reports the replay-memory size, mini-batch size, target-network update interval, optimizer, loss function, learning rate, exploration schedule, SMDP minimum-green constraints, and training horizon. Specifically, the DQN/MADQN agents were trained online for one 10,000-step SUMO execution horizon per demand scenario, using Adam optimization with Huber loss, randomized replay sampling, soft target-network updates every 200 steps, and ε-greedy exploration decayed from 1.0 to a minimum of 0.1. The replay buffer size was 200,000 transitions for the multi-agent DQN/MADQN implementations, while some single-agent DQN delay experiments used a smaller 50,000-transition buffer. These details were added to improve reproducibility and methodological transparency.

 

Evaluation & Results

• The evaluation methodology is generally appropriate, However, additional clarification is needed regarding:
1. number of simulation runs,
2. random seed control,
3. variability between runs,
4. and whether reported values represent averages or single experiments.

We thank the reviewer for this important comment. We have revised the evaluation methodology to clarify how the simulations were conducted and how the reported values should be interpreted. The implemented experiments used a controlled fully online simulation protocol in which each controller was evaluated for one 10,000-step SUMO execution horizon under each demand scenario and network configuration. The reported values therefore represent the outputs of the implemented controlled simulation runs rather than averages over multiple independent random-seed replications. We have corrected the manuscript to avoid claiming repeated-run statistics where they were not computed. We also added a limitation stating that future work should repeat each scenario under multiple random seeds and report mean values, standard deviations, and confidence intervals.

• Although performance improvements are reported, no statistical significance analysis is provided. The manuscript should include:
1. confidence intervals,
2. standard deviation/error bars,
3. or statistical hypothesis testing.

We thank the reviewer for this important comment. We agree that statistical significance analysis would strengthen the reported comparison. In the revised manuscript, we clarified that the current results were obtained from controlled single-run online simulation experiments for each scenario rather than from repeated independent random-seed replications. Therefore, we do not claim statistical significance, confidence intervals, or hypothesis-test results in the revised version. Instead, we have corrected the evaluation section and added this issue as a limitation. Future work will conduct multiple independent replications for each controller, demand level, and network configuration, enabling the reporting of standard deviations, confidence intervals, error bars, effect sizes, and statistical hypothesis tests.

• The manuscript should explain whether:
1. all controllers received equal training time,
2. identical demand conditions,
3. and comparable optimization effort.

We thank the reviewer for this helpful comment. We have revised the evaluation methodology to clarify the fairness of the comparison. The fixed-time, Tabular Q-Learning, and MADQN controllers were evaluated under the same SUMO network geometry, detector configuration, vehicle-behavior parameters, demand levels, route files, turning-movement assumptions, simulation horizon, and performance metrics. The learning-based controllers were trained online over the same 10,000-step execution horizon under the corresponding demand scenario. The fixed-time controller does not require training; therefore, it was evaluated as a deterministic static benchmark under the same demand exposure rather than being assigned learning episodes. We also clarified that the fixed-time controller was not Webster-optimized, so the comparison is between predefined static signal operation and adaptive RL-based control under identical traffic scenarios.

• The manuscript should explain why:
1. MADQN performs better under low-demand and queue-dissipation scenarios,
2. while Tabular Q-Learning remains competitive under higher demand conditions.

We thank the reviewer for this insightful comment. We have revised the Results and Discussion section to provide a clearer interpretation of the observed demand-dependent performance pattern. Under low-demand and queue-dissipation conditions, traffic states are sparse and variable, and DQN/MADQN can benefit from function approximation because it generalizes across similar queue states rather than treating each discretized state independently. Under medium and high demand, however, congestion states become more recurrent and dominated by persistent queues. In these conditions, Tabular Q-Learning can remain competitive because the selected four-bin discretization captures the dominant congestion regimes and repeatedly exposes the agent to similar state-action patterns. We also clarified that this does not imply universal superiority of either method; rather, controller effectiveness depends on demand regime, state dimensionality, network complexity, and the selected performance metric. This interpretation is now reflected in the revised Results and Discussion section, where the demand-dependent strengths of Tabular Q-Learning and MADQN are discussed in relation to state-space complexity, demand regime, and optimization objective.

Since RL agents are trained within the same simulation environment used for testing, the manuscript should discuss:

1. generalization capability,

 

2. robustness to unseen traffic patterns,

 

3. and possible overfitting to corridor-specific dynamics.

We thank the reviewer for this important comment. We have revised the Discussion and Conclusions sections to clarify that the RL agents were trained and evaluated within the same SUMO-based simulation environments, and therefore the findings should be interpreted as scenario-specific evidence rather than proof of universal transferability. The use of two testbeds—the synthetic two-junction corridor and the 25 January real-world digital twin—improves the breadth of evaluation because the controllers are exposed to both controlled symmetric conditions and more complex asymmetric real-world geometry. However, the learned policies may still reflect the specific detector layout, demand profiles, turning ratios, signal-phase structure, and vehicle-behavior parameters used in the simulations. We therefore added a limitation stating that future work should test the trained controllers under unseen demand patterns, different random seeds, incident scenarios, alternative turning ratios, and additional real-world corridors.

 

"At last, the authors would like to thank the Associate Editor and the reviewers for their constructive comments that have led to significant improvements in the quality of the paper."

 

Author Response File: Author Response.pdf

Reviewer 2 Report

Comments and Suggestions for Authors

The manuscript investigates adaptive traffic signal control based on multi-agent reinforcement learning. It compares Fixed-Time control, Multi-Agent Tabular Q-Learning, and MADQN in a SUMO–Python–TraCI simulation framework, considering both a synthetic two-intersection corridor and a real-world digital-twin corridor. This topic is related to urban traffic management and has certain practical relevance. Nevertheless, several aspects of the manuscript could be improved to make the experimental results clearer, more consistent, and easier to interpret.

1. The network figures in Section 3.1 show the synthetic network and the 25 January Corridor, but they lack traffic flow directions, intersection IDs, detector locations, and signal-control areas. Adding these labels would make the experimental scenarios easier to understand.
2. In Section 3.2, the sentence “Calibration was treated as a prerequisite for credible RL evaluation rather than a post-processing step” appears twice in close succession. This section could be revised to avoid repetition and improve readability.
3. In Sections 3 and 4, many simulation and learning settings are described in the text, but they are scattered across different subsections. It is recommended to add a concise table summarizing the main experimental settings, such as traffic demand levels, vehicle parameters, simulation step, detector settings, and major learning parameters.
4. The conclusion states that the proposed framework has potential for adaptive traffic signal control, but the limitations of the simulation-based study are not fully discussed. The authors should briefly address issues such as real-world deployment, data availability, detector reliability, and validation beyond simulation.

Author Response

A detailed response to the reviewers' comments:

 

The following sections provide a detailed response to each reviewer's comment. Our answers are presented in italic, bold font.

 

 

Reviewer #2:

 

The manuscript investigates adaptive traffic signal control based on multi-agent reinforcement learning. It compares Fixed-Time control, Multi-Agent Tabular Q-Learning, and MADQN in a SUMO–Python–TraCI simulation framework, considering both a synthetic two-intersection corridor and a real-world digital-twin corridor. This topic is related to urban traffic management and has certain practical relevance. Nevertheless, several aspects of the manuscript could be improved to make the experimental results clearer, more consistent, and easier to interpret.

First of all, we want to thank the reviewer for his careful revision and valuable suggestions, which really helped improve our manuscript.

1. The network figures in Section 3.1 show the synthetic network and the 25 January Corridor, but they lack traffic flow directions, intersection IDs, detector locations, and signal-control areas. Adding these labels would make the experimental scenarios easier to understand.

We thank the reviewer for this helpful observation. We agree that the original figures in Section 3.1 did not provide sufficient visual annotation to fully clarify the experimental setup. In response, we have revised Figures 1 and 2 to improve interpretability by explicitly adding:

1. traffic flow directions,
2. intersection IDs,
3. detector locations/labels, and
4. signal-control areas for each controlled junction.

These additions make the synthetic and real-world network configurations easier to understand and help the reader better follow the experimental design and the role of each control agent in the simulation framework.

1. In Section 3.2, the sentence “Calibration was treated as a prerequisite for credible RL evaluation rather than a post-processing step” appears twice in close succession. This section could be revised to avoid repetition and improve readability.

We thank the reviewer for pointing out this repetition. We agree that the sentence appeared twice in close succession in Section 3.2, which affected the readability of the calibration and validation subsection. The repeated sentence has been removed, and the opening paragraph of Section 3.2 has been revised to provide a smoother introduction to the calibration procedure and its role in ensuring credible RL evaluation.

1. In Sections 3 and 4, many simulation and learning settings are described in the text, but they are scattered across different subsections. It is recommended to add a concise table summarizing the main experimental settings, such as traffic demand levels, vehicle parameters, simulation step, detector settings, and major learning parameters.

We thank the reviewer for this constructive suggestion. We agree that several simulation and learning settings were described across different subsections, which may make it difficult for readers to quickly identify the main experimental configuration. Accordingly, we added a concise summary table at the end of Section 3, immediately before the algorithmic implementation section. The new table consolidates the main experimental settings, including network configurations, traffic demand levels, turning-movement assumptions, vehicle-behavior parameters, simulation step length, simulation horizon, detector configuration, action space, SMDP minimum-green lock, reward objectives, and the principal Tabular Q-Learning and MADQN learning parameters. This addition improves the readability of the methodology and supports the reproducibility and interpretation of the experimental design.

1. The conclusion states that the proposed framework has potential for adaptive traffic signal control, but the limitations of the simulation-based study are not fully discussed. The authors should briefly address issues such as real-world deployment, data availability, detector reliability, and validation beyond simulation.

We thank the reviewer for this valuable comment. We agree that the limitations of the simulation-based evaluation should be stated more explicitly, particularly with respect to real-world deployment. Accordingly, the Conclusion section has been revised to include a dedicated limitations paragraph. The added text clarifies that, although the proposed framework was evaluated using both a controlled synthetic corridor and a real-world digital twin, the experiments remain simulation-based and do not fully capture all practical deployment constraints. The revised conclusion now discusses the need for reliable field data, detector accuracy and continuity, robust communication between sensors and controllers, calibration with observed traffic conditions, and validation beyond SUMO through field trials, hardware-in-the-loop testing, or deployment-oriented pilot studies.

Author Response File: Author Response.pdf

Reviewer 3 Report

Comments and Suggestions for Authors

The manuscript addresses an important and timely topic in intelligent transportation systems by comparing Fixed-Time, Multi-Agent Tabular Q-Learning, and MADQN controllers under both synthetic and real-world corridor scenarios. The study is technically detailed, and the integration of a real-world digital twin improves the practical value of the work. However, several points still need to be addressed before publication:

1- In line 126, I suggest presenting the main contributions in bullet-point format to improve clarity and make them easier for readers to follow.

2- In the related work section, the authors are encouraged to include a comparison table summarizing recent studies, their limitations, and the novelty of the proposed work. This would better highlight the contribution of the manuscript compared with existing studies.

3- The sentence “Calibration was treated as a prerequisite for credible RL evaluation rather than a post-processing step” is repeated consecutively with very similar wording. Revising this part would improve readability and reduce redundancy.

4- The manuscript does not provide enough discussion regarding computational overhead, training time, convergence speed, or scalability when the number of intersections increases. These factors are important for evaluating the practicality of deploying the proposed approach in real-world environments.

5- The authors should compare their findings with state-of-the-art studies to further validate the effectiveness and significance of the proposed method.

Author Response

A detailed response to the reviewers' comments:

 

The following sections provide a detailed response to each reviewer's comment. Our answers are presented in italic, bold font.

 

 

Reviewer #3:

 

The manuscript addresses an important and timely topic in intelligent transportation systems by comparing Fixed-Time, Multi-Agent Tabular Q-Learning, and MADQN controllers under both synthetic and real-world corridor scenarios. The study is technically detailed, and the integration of a real-world digital twin improves the practical value of the work. However, several points still need to be addressed before publication:

First of all, we want to thank the reviewer for his careful revision and valuable suggestions, which really helped improve our manuscript

1- In line 126, I suggest presenting the main contributions in bullet-point format to improve clarity and make them easier for readers to follow.

We thank the reviewer for this helpful suggestion. We agree that presenting the main contributions in bullet-point format improves clarity and makes the novelty of the study easier for readers to follow. Accordingly, the contribution paragraph in the Introduction, around line 126, has been revised and reformatted into a concise bullet-point list. The revised text now clearly highlights the main contributions of the study, including the multi-intersection corridor evaluation, the direct comparison of the three control strategies, the use of synthetic and real-world digital-twin testbeds, the assessment under different traffic-demand levels, and the interpretation of the relative strengths of tabular and deep value-based MARL architectures.

2- In the related work section, the authors are encouraged to include a comparison table summarizing recent studies, their limitations, and the novelty of the proposed work. This would better highlight the contribution of the manuscript compared with existing studies.

We thank the reviewer for this constructive suggestion. We agree that a comparison table would improve the clarity of the related work section and make the novelty of the proposed study easier to identify. Accordingly, a new table has been added in Section 2 to summarize representative recent studies, including their control approach, simulation or evaluation environment, main limitations, and relevance to the present work. The added table highlights that many existing studies focus on isolated intersections, simplified synthetic networks, single deep RL architectures, policy-gradient methods, or objectives other than direct comparison between tabular and deep value-based MARL controllers. This comparison clarifies the novelty of the present manuscript, which directly compares Fixed-Time control, Multi-Agent Tabular Q-Learning, and MADQN under equivalent simulation conditions using both a controlled synthetic corridor and a real-world digital-twin corridor.

3- The sentence “Calibration was treated as a prerequisite for credible RL evaluation rather than a post-processing step” is repeated consecutively with very similar wording. Revising this part would improve readability and reduce redundancy.

We thank the reviewer for pointing out this repetition. We agree that the sentence was repeated consecutively with very similar wording in Section 3.2, which reduced readability. Accordingly, the repeated sentence has been removed, and the opening paragraph of Section 3.2 has been revised to improve flow and avoid redundancy while preserving the intended methodological clarification that calibration was treated as a prerequisite for credible RL evaluation.

4- The manuscript does not provide enough discussion regarding computational overhead, training time, convergence speed, or scalability when the number of intersections increases. These factors are important for evaluating the practicality of deploying the proposed approach in real-world environments.

We thank the reviewer for this valuable comment. We agree that computational overhead, training time, convergence speed, and scalability are important factors for assessing the practical deployability of MARL-based adaptive traffic signal control. Accordingly, the manuscript has been revised to include an additional discussion on computational and scalability considerations. The revised text clarifies that Tabular Q-Learning has lower per-step computational cost but suffers from state-space growth as the number of intersections, detectors, and phases increases, whereas MADQN introduces higher training overhead due to neural-network updates, replay memory, and target-network operations, but is more suitable for larger or continuous state representations. We also clarified that the present study used online learning over a fixed 10,000-step simulation horizon and that the observed learning curves should be interpreted as descriptive evidence of stabilization rather than formal convergence proof. Finally, the scalability limitations of the current two-intersection testbeds are now acknowledged, and future work is proposed to evaluate larger corridors and networks using repeated runs, distributed training, transfer learning, and wall-clock computational profiling.

5- The authors should compare their findings with state-of-the-art studies to further validate the effectiveness and significance of the proposed method.

We thank the reviewer for this valuable comment. We agree that comparing the findings with recent state-of-the-art studies strengthens the interpretation of the proposed framework and better highlights its significance. Accordingly, the Results and Discussion section has been revised to include an additional comparative discussion with recent RL-, DRL-, and MARL-based traffic signal control studies. The added discussion explains that the observed improvements over Fixed-Time control are consistent with previous studies reporting the advantages of RL and DRL controllers over static timing strategies. It also clarifies that, unlike many studies focusing on isolated intersections, simplified synthetic networks, or a single deep-learning controller, the present study provides a direct comparison between Multi-Agent Tabular Q-Learning and MADQN under equivalent demand scenarios and in both synthetic and real-world digital-twin corridor testbeds. We further note that direct numerical comparison with prior studies is limited by differences in network geometry, demand assumptions, reward functions, evaluation indicators, and simulation settings; therefore, the comparison is presented at the methodological and performance-trend levels. This added discussion complements the related-work comparison table in Section 2 and links the reported results more directly to the current state of the art.

Author Response File: Author Response.pdf

Reviewer 4 Report

Comments and Suggestions for Authors

Dear authors,
The research in this paper deals with the very current and scientifically relevant topic of applying multi-agent reinforcement learning approaches in adaptive traffic light control in urban traffic networks. Traffic management and regulation in urban areas is of great importance in order to increase safety as well as increase the quality and sustainability of mobility.
The research in the paper belongs to the field of ITS, traffic with a segment that is very significant and refers to the management of traffic lights that are increasingly present in urban traffic networks. A special contribution is the application of advanced tools where Fixed-Time, Multi-Agent Tabular Q-Learning and Multi-Agent Deep Q-Network (MADQN) approaches are analyzed and compared within the SUMO–Python–TraCI simulation environment.
Compared to previous research, the value of the presented research is reflected in the use of a real digital twin, the comparison of the tabular and deep RL approaches, the inclusion of the TTC safety indicator, the analysis of different load levels and the application of the MARL architecture in a corridor environment.
In methodological terms, the research in the paper is well structured with carefully selected references that are in line with the topic of the paper. The figures and tables are correct, clear and clear to the wider academic community. Table 4
In order to improve the quality, it is necessary:
1. In addition to the basic equations, add a formal MDP/MARL definition, a formal MARL tuple, CTDE mathematical formalization, agent interaction model.
2. The results are presented descriptively, but I believe that statistical significance testing, confidence intervals, variance analysis, hypothesis testing should be added.
3. Only one training episode is presented, and it is known that RL algorithms usually require a large number of episodes of long-term policy refinement and repetitive learning. Because of the above, justification or extension to multi-episode training should be added.
4. The security analysis is limited, so it should be expanded through the integration of TTC in reward or use the constrained RL framework,
5. Due to the generalization of the results, it is necessary to add scalability analysis, communication overhead analysis, computational complexity analysis.
6. Due to its volume, I suggest placing Table 4 as an appendix at the end of the paper and adapting it to the guidelines of the journal.
In terms of methodology, the research in the work is well structured with carefully selected references that are in accordance with the topic of the work. After the changes made, rearrange the conclusion.

Author Response

A detailed response to the reviewers' comments:

 

The following sections provide a detailed response to each reviewer's comment. Our answers are presented in italic, bold font.

 

 

Reviewer #4:

 

The research in this paper deals with the very current and scientifically relevant topic of applying multi-agent reinforcement learning approaches in adaptive traffic light control in urban traffic networks. Traffic management and regulation in urban areas is of great importance in order to increase safety as well as increase the quality and sustainability of mobility.

The research in the paper belongs to the field of ITS, traffic with a segment that is very significant and refers to the management of traffic lights that are increasingly present in urban traffic networks. A special contribution is the application of advanced tools where Fixed-Time, Multi-Agent Tabular Q-Learning and Multi-Agent Deep Q-Network (MADQN) approaches are analyzed and compared within the SUMO–Python–TraCI simulation environment.

Compared to previous research, the value of the presented research is reflected in the use of a real digital twin, the comparison of the tabular and deep RL approaches, the inclusion of the TTC safety indicator, the analysis of different load levels and the application of the MARL architecture in a corridor environment.
In methodological terms, the research in the paper is well structured with carefully selected references that are in line with the topic of the paper. The figures and tables are correct, clear and clear to the wider academic community. Table 4
In order to improve the quality, it is necessary:

 

First of all, we want to thank the reviewer for his careful revision and valuable suggestions, which really helped improve our manuscript

1. 
   In addition to the basic equations, add a formal MDP/MARL definition, a formal MARL tuple, CTDE mathematical formalization, agent interaction model.

We thank the reviewer for this important comment. We agree that the manuscript should provide a clearer theoretical formulation of the reinforcement-learning and multi-agent learning setting. Accordingly, we added a new subsection in Section 3 entitled “Formal MDP/MARL Formulation and Agent Interaction Model.” The added subsection defines the single-agent MDP formulation, extends it to a formal MARL tuple, and explains how traffic-signal agents interact through the shared traffic environment. We also clarified the relationship between the implemented framework and CTDE. Specifically, the implemented controllers follow an independent value-based MARL structure, where each traffic signal is represented by an independent agent with its own Q-table or DQN model and its own action selection process. Therefore, the framework should not be interpreted as a strict CTDE architecture with a centralized critic, value-decomposition network, or centralized parameter-update module. Instead, the manuscript now explains that some agents may use a compressed network-level traffic state during learning, providing partial global traffic awareness, while the value updates and action execution remain independent and decentralized. This clarification avoids overstating the architecture while still positioning the proposed method relative to the CTDE paradigm requested by the reviewer.

2. 
   The results are presented descriptively, but I believe that statistical significance testing, confidence intervals, variance analysis, hypothesis testing should be added.

We thank the reviewer for this important comment. We agree that statistical significance testing, confidence intervals, variance analysis, and hypothesis testing would strengthen the quantitative comparison among the Fixed-Time, Tabular Q-Learning, and MADQN controllers. In the revised manuscript, we clarified that the current experiments were conducted as controlled online SUMO simulation runs for each network topology, demand level, controller type, and optimization objective, rather than as repeated independent random-seed replications. Therefore, the present experimental design does not statistically support the reporting of p-values, confidence intervals, between-run variance estimates, or formal hypothesis-test results. Adding such tests to the current single-run scenario outputs would risk overstating the statistical strength of the findings.

To address this point, we revised the evaluation methodology to explicitly state that the reported values should be interpreted as controlled scenario-based simulation outcomes from the implemented runs, rather than population-level statistical estimates. We also revised the wording of the Results section to avoid implying statistical significance where formal statistical testing was not performed. In addition, we added this issue as a limitation and future research direction. Future work will repeat each controller–scenario combination under multiple independent random seeds and report mean values, standard deviations, confidence intervals, effect sizes, and appropriate statistical tests, such as paired t-tests, ANOVA/repeated-measures ANOVA, or non-parametric alternatives, depending on the distributional properties of the replicated outputs.

3. 
   Only one training episode is presented, and it is known that RL algorithms usually require a large number of episodes of long-term policy refinement and repetitive learning. Because of the above, justification or extension to multi-episode training should be added.

We thank the reviewer for this important comment. We agree that many RL and DRL algorithms are commonly trained over multiple episodes to support long-term policy refinement, convergence assessment, and reproducibility analysis. In the revised manuscript, we clarified that the present implementation used a fully online continuous-learning protocol, where each controller was trained and evaluated over one complete 10,000-step SUMO simulation horizon for each network, demand level, controller type, and optimization objective. Therefore, the single episode reported in the current study represents one complete online simulation scenario rather than a short offline training iteration.

We also clarified that the objective of the present work was to compare the behavior of Fixed-Time, Tabular Q-Learning, and MADQN controllers under identical controlled online simulation conditions, rather than to claim fully converged multi-episode policy optimality. To avoid overstatement, we revised the training methodology and limitations sections to state that the current results should be interpreted as controlled online learning outcomes. We added multi-episode training and repeated random-seed replications as an explicit future-work direction. Future work will extend the training protocol by running multiple independent episodes for each controller–scenario combination, tracking convergence across episodes, reporting mean and variance of performance metrics, and assessing policy stability and generalization under unseen demand patterns.

4. 
   The security analysis is limited, so it should be expanded through the integration of TTC in reward or use the constrained RL framework,

We thank the reviewer for this valuable comment. We agree that the safety analysis of reinforcement-learning-based traffic signal control can be strengthened by explicitly incorporating Time-To-Collision (TTC) either into the reward function or into a constrained reinforcement learning framework. In the original implementation, TTC was monitored as a complementary surrogate safety indicator rather than being directly optimized in the reward. The main reason for this design choice was to keep the comparison between Fixed-Time, Tabular Q-Learning, and MADQN controllers focused on equivalent operational objectives, namely queue minimization and delay minimization, while using TTC as an independent post-simulation safety-related measure.

In response to the reviewer's comment, we revised the manuscript to clarify this methodological choice and expanded the TTC-based safety discussion. Specifically, we strengthened the description of TTC in Section 3.4, clarified in Section 3.5 why TTC was not included in the primary reward function, and expanded Section 5.5 to interpret TTC jointly with queue and delay rather than as an isolated indicator. We also added a future extension describing how TTC can be integrated into a safety-aware reward or a constrained RL formulation. This includes adding a TTC penalty term when TTC falls below a predefined safety threshold, or alternatively treating TTC as a formal safety constraint that must remain above a minimum acceptable threshold during future learning and execution.

Accordingly, the revised manuscript now makes clear that the current study performs surrogate safety assessment using TTC, while future work will extend the learning framework toward safety-aware or constrained MARL in which TTC directly influences the policy update. This revision addresses the reviewer’s concern while avoiding overstating the present implementation as a fully safety-constrained RL framework.

5. 
   Due to the generalization of the results, it is necessary to add scalability analysis, communication overhead analysis, computational complexity analysis.

We thank the reviewer for this important comment. We agree that the generalization of the proposed framework should be supported by a clearer discussion of scalability, communication overhead, and computational complexity. In the revised manuscript, we added a new subsection entitled “Scalability, Communication Overhead, and Computational Complexity Analysis” in the Results and Discussion section.

The added subsection clarifies how the computational burden of the proposed controllers grows with the number of intersections, detectors, phases, state variables, and learning parameters. We also added an explicit comparison between Fixed-Time control, Tabular Q-Learning, and MADQN in terms of online decision complexity, memory requirements, and scalability limitations. The revised discussion explains that Fixed-Time control has negligible computational cost, Tabular Q-Learning is computationally simple but suffers from state-space growth, while MADQN has higher per-decision computational cost due to neural-network inference but is more scalable than tabular methods for larger or continuous state representations.

We also expanded the communication-overhead analysis. Since the implemented framework follows independent decentralized value-based MARL, no explicit inter-agent message passing, centralized critic, or centralized parameter-update module is used during execution. Therefore, the online communication overhead between agents is minimal and mainly limited to local detector-to-controller data exchange and TraCI-based communication between SUMO and the Python controller. This clarification strengthens the generalization discussion while avoiding overclaiming full large-scale network validation.

Finally, we added a limitation, noting that the present scalability analysis is analytical rather than based on experiments on large-scale city networks. Future work will extend the evaluation to larger corridors and grid networks with more intersections, where runtime, memory use, communication latency, and performance degradation can be measured empirically.

6. 
   Due to its volume, I suggest placing Table 4 as an appendix at the end of the paper and adapting it to the guidelines of the journal.

We thank the reviewer for this helpful suggestion. We agree that Table 4 is relatively large and may interrupt the flow of the main results section. Accordingly, we have moved the full detailed TTC descriptive-statistics table from the main text to a new Appendix at the end of the manuscript. The table has been reformatted according to the journal style and is now presented as Appendix A, Table A1. In the main Results and Discussion section, we retained only a concise interpretive summary of the TTC findings and added a cross-reference to Appendix A. This revision improves the readability of the main manuscript while preserving the complete safety-related TTC evidence for transparency and reproducibility.

In terms of methodology, the research in the work is well structured with carefully selected references that are in accordance with the topic of the work. After the changes made, rearrange the conclusion.

We thank the reviewer for the positive evaluation of the methodological structure and the relevance of the selected references. Following the reviewer's recommendation, the Conclusion section has been reorganized to better reflect the revisions made throughout the manuscript. In particular, the revised conclusion now follows a clearer logical sequence: first, it restates the study objective and simulation framework; second, it summarizes the main operational findings for Fixed-Time control, Multi-Agent Tabular Q-Learning, and MADQN; third, it discusses the safety-related TTC observations in relation to queue and delay performance; fourth, it highlights the computational-overhead, communication, and scalability implications; and finally, it presents the main limitations and future research directions. This restructuring improves the readability of the conclusion and ensures consistency with the newly added methodological clarifications, scalability discussion, TTC analysis, and appendix placement of the detailed TTC table.

 

"At last, the authors would like to thank the Associate Editor and the reviewer for their constructive comments that have led to significant improvements in the quality of the paper."

 

Author Response File: Author Response.pdf

Round 2

Reviewer 1 Report

Comments and Suggestions for Authors

The authors have addressed comments.