Multi-Agent Deep Reinforcement Learning Framework for Efficient Aerial Wildfire Fighting ^†

Abstract

The increasing severity of global wildfires requires advanced suppression strategies to mitigate impacts on the environment and human life. This work investigates the applicability of Multi-Agent Reinforcement Learning (MARL) to aerial wildfire suppression using the SoSID Toolkit, an agent-based grid simulation grounded in cellular-automata-based fire propagation. To enhance interpretability and support the reconstruction of learned tactics, this work introduces the Dual Decomposition Framework, providing a modular structure for both the reward function and the observation space. This design enables the systematic evaluation of individual components, allowing the identification of the elements most relevant to effective wildfire suppression. The learned MARL policy is compared against a heuristic strategy inspired by real-world firefighting practice. The reward analysis confirms that the Dual Decomposition Framework enhances transparency in agent behavior by analyzing the contribution of individual components. The experiments further show that the learned policy can outperform the heuristic approach in terms of burned-area reduction when fire spread sensitivity is low, demonstrating the potential of MARL for effective suppression strategies. However, performance declines as spread sensitivity increases, indicating limited generalization and signs of overfitting to training conditions. The findings suggest that approaches such as curriculum learning may improve robustness under faster-spreading fire dynamics.

1. Introduction

Motivation

In recent years, the frequency and severity of wildfires in Europe have increased significantly. According to the European Forest Fire Information System (EFFIS) [1], more than 1,025,224 hectares burned in the EU in 2025 alone, almost three times as much as in the same period in 2024. This is also significantly higher than the 19-year average of 323,135 hectares burned during the same period (2006–2024), underscoring the exceptional severity of the current wildfire situation. Forest fires also contribute significantly to greenhouse gas emissions, with 41.83 Mt CO₂ in 2025 alone.

Reinforcement learning (RL) has shown remarkable success in solving complex decision-making problems across a wide range of domains, from robotics to strategic planning. In the context of wildfire management, Multi-Agent Reinforcement Learning (MARL) has emerged as a promising approach for coordinating autonomous systems such as unmanned aerial vehicles (UAVs) and enabling swarms of autonomous agents to collaboratively suppress fires [2,3]. However, one major limitation of RL, as well as machine learning in general, is its black-box nature, which makes it difficult to interpret the underlying decision-making process. Moreover, when the objective is to explore the space of possible tactics and develop new, more efficient suppression strategies, the lack of interpretability is a critical limitation. The field of Explainable Reinforcement Learning (XRL) has emerged to address these challenges. One approach is reward decomposition, which decomposes the reward function into separate components aligned with the reward structure, revealing how different reward components influence decision-making [4].

This work presents the Dual Decomposition Framework to facilitate the interpretability of agent behavior and aims to address the following two research questions:

• How can Reinforcement Learning be used to enable the learning of suppression strategies evaluated by integration into an agent-based wildfire simulation?
• To what extent do decomposed reward functions and observations enhance the interpretability of Reinforcement Learning-based suppression strategies?

2. Materials and Methods

2.1. SoSID Toolkit

The SoSID Toolkit [5,6] serves as the foundational framework for this work and provides an agent-based System-of-Systems (SoS) wildfire simulation. The environment is represented as a grid-based fire model governed by cellular automata rules where each cell can assume distinct states, including suppressed, nonflammable, and multiple burning stages. Terrain recreation is used to replicate real-world landscapes, enabling the reproduction of past wildfire events and the evaluation of new suppression tactics. To simulate suppression efforts, the toolkit utilizes Agent-Based Modeling (ABM) to coordinate multiple aircraft and execute suppression tactics. The simulation implements two primary heuristic strategies that mirror real-world firefighting strategies:

• Indirect Attack: The objective is to contain the fire by establishing a continuous suppression barrier around the fire perimeter.
• Direct Attack: Agents apply the suppressant to extinguish burning cells directly. This tactic relies on predefined points of interest determined by a weighted function considering water sources, residential areas, agent position, and spread rates.

2.2. Deep Reinforcement Learning

Deep Reinforcement Learning (DRL) combines reinforcement learning with deep neural networks to enable agents to learn effective decision-making policies in high-dimensional and partially observable environments. In DRL, an agent interacts with an environment by observing its current state, selecting an action according to a policy, and receiving feedback in the form of rewards. Over time, the agent optimizes its policy to maximize the expected cumulative reward. The use of deep neural networks allows DRL methods to approximate complex value functions or policies that would be infeasible to represent explicitly. This makes DRL particularly suitable for domains such as wildfire simulation, where the state space is large, continuous, and spatially structured.

For this work, the Proximal Policy Optimization (PPO) [7] algorithm is used to train reinforcement learning agents. PPO is a widely adopted policy-gradient method known for its stability and sample efficiency [8,9]. It achieves robust learning by constraining policy updates through a clipped surrogate objective, preventing sudden and destabilizing changes to the policy. This makes PPO well-suited to dynamic and partially observable multi-agent environments, such as the wildfire suppression scenario considered here.

2.3. Dual Decomposition Framework

This work presents the Dual Decomposition Framework, which provides a systematic methodology for developing and analyzing interpretable reinforcement learning policies. Building upon reward decomposition [4], this framework extends this approach by incorporating observation-space decomposition, enabling comprehensive analysis of how both environmental perception and objective structure influence learned behavior.

As shown in Figure 1, the framework architecture consists of three interconnected stages. First, a formal “Problem Setting” defines the operational context through scenario specifications and performance requirements. This Problem Setting establishes the operational context that is utilized for both RL training and heuristic tactics, ensuring that all evaluated approaches address identical challenges. Second, the “Decomposition Module” is leveraged to structure both the reward function and the observation space into individual components during training. This modular design enables the dynamic reconstruction and customization of both the reward signal and the agent’s inputs for each individual experiment. Third, the “Tactic Evaluation” facilitates a quantitative and qualitative comparison between learned policies and heuristic baselines. To assess the effectiveness of the suppression strategy, the burned area is utilized as the objective metric.

3. Implementation

For the integration of MARL into the wildfire simulation, this work employs RLlib [10,11], a scalable reinforcement learning library that is part of the Ray framework [12]. Its modular design allows seamless integration with custom environments, enabling the wildfire simulation to define domain-specific observations, actions, and reward functions. Within RLlib’s multi-agent training framework, PPO is combined with a shared policy architecture, allowing all agents to learn from collective experience while maintaining a consistent decision-making strategy. This setup enhances sample efficiency, reduces training complexity, and supports scalable coordination among multiple suppression agents.

3.1. Action Space

This work uses a discrete action space that includes both movement actions and a suppression action. The suppression action extinguishes the fire in the agent’s current cell, while the movement actions correspond to the cardinal directions. Initially, only the four main directions (North, South, East, West) were implemented to reduce the dimensionality of the action space and facilitate early training stability. However, to improve maneuverability, the final action space was expanded to 16 discrete directions, allowing for more precise movement of the agents.

3.2. Observation Space

The Dual Decomposition Framework is applied to the observation space, implementing the perceptual inputs as modular components. This allows controlled variation of the information available to the agents and enables systematic analysis of how different observation modalities influence policy learning. At the start of each experiment, the observation space is assembled from the selected components. For the experiments conducted in this work, the following observation components are used:

• Local Grid: Enables the agents to perceive their immediate surroundings by providing information about the states of neighboring cells. Each cell within the observation grid corresponds to discrete environmental states, including suppressed, nonflammable, and various burning stages. The size of the local observation grid can be configured at the start of each experiment to adjust the agent’s field of view and thus control how much of the environment is perceived when making decisions.
• Closest N Fire Direction: Provides direction vectors to the N nearest burning cells, with N specified during experiment initialization.
• Cardinal Directions: Provides four direction vectors from the agent’s position to the cardinal extremities of the wildfire perimeter (northernmost, southernmost, easternmost, and westernmost points of the fire boundary), offering a compact representation of the directional distribution of fires.
• Ignition Center Direction: Provides a direction vector from the agent’s position to the ignition center of the fire.

3.3. Reward Function

Similarly, the reward function is structured modularly using the decomposition principle, dividing it into several individual components. At the initialization of each experiment, the reward function is reconstructed by combining the predefined components, each assigned a corresponding weight that determines its contribution to the overall reward. For the experiments, the following components are utilized:

• Proximity Reward: The Proximity Reward provides dense feedback based on an agent’s distance to the wildfire. Increasing the distance results in a penalty, while decreasing it yields a reward. This encourages agents to systematically locate and approach the fire. Unlike sparse rewards, dense feedback offers continuous guidance, accelerating convergence by reducing the exploration burden. In wildfire suppression, such proximity-based signals are essential for learning effective navigation toward active fire fronts. Without such intermediate rewards, agents would have difficulty recognizing the connection between their movement decisions and the ultimate goal.
• Suppression Reward: The purpose of the Suppression Reward is to encourage agents to actively suppress the wildfire. This reward is calculated directly based on the number of successfully suppressed cells.
• Connected Suppression Reward: Provides the reward based on whether the current suppression is adjacent to already suppressed cells. The goal is to encourage the agent to establish a continuous line of suppression rather than suppressing individual cells in isolation, thereby promoting more effective containment strategies.
• Inactivity Penalty: Applies a negative reward when an agent is adjacent to a burning cell but fails to execute a suppression action. This discourages inactive behavior near the fire front.
• Timestep Penalty: Applies a small penalty at every simulation step to encourage efficient suppression. As the cumulative reward decreases with longer episode lengths, strategies that extinguish the fire more efficiently obtain higher overall returns, aligning policy optimization with operational efficiency.
• Impact Reward: Provides a positive reward based on the reduction in the wildfire’s overall spread rate. This component serves as a long-term incentive, thereby rewarding suppression actions that do not yield an immediate extinction reward if they successfully slow the global propagation of the fire.

3.4. Policy

A shared policy is used, where all agents are controlled by the same policy network. During training, each agent interacts with the environment independently and generates its own trajectories, which are then combined to update the shared policy. This approach is used to ensure consistent behavior across all agents and to reduce the overall complexity of the learning setup. Relying on a single shared policy substantially reduces the number of trainable parameters, which generally leads to more stable learning dynamics and improves sample efficiency. It also enhances scalability, as learned behavior can be transferred to an arbitrary number of agents without the need for additional training.

4. Results

4.1. Experiments

The experiments in this work are conducted using the Dual Decomposition Framework, introduced in Section 2.3. The simulation environment is based on a wildfire scenario representing the Sintra region in Portugal. In the wildfire model, the correction coefficient (cc) regulates the sensitivity of the fire to its surroundings. A lower value creates a fire that is less likely to spread, whereas a higher value results in very volatile spreading behavior. The training was conducted over 5000 iterations with four agents and a fixed cc of $0.1$. For the evaluation, six agents were deployed with an initial cc value of 0.1, which gradually increases after each successfully suppressed wildfire. This procedure assesses the generalizability of the RL tactic across increasing levels of wildfire complexity. Using fewer agents during training reduces computational effort and accelerates convergence, while increasing the number of agents during evaluation allows the learned strategy to be assessed under more demanding operating conditions. The burned area is used as an evaluation metric for all experiments.

To assess the effect of the Dual Decomposition Framework, an experiment was conducted using two configurations whose reward functions differ only in a single component. Figure 2 shows the performance comparison between Policy-1 (red), which includes the Suppression-Reward component, and Policy-2 (green), where this component is replaced by the Connected-Suppression-Reward component. All other reward components remained identical throughout both configurations, as listed in Section 3.3. In addition, the Observation Space configuration, as described in Section 3.2, was kept constant to ensure that performance differences can be attributed solely to the variation of the reward component. For the Local Grid, a size 15 × 15 cells was used.

Although both policies succeed in containing the wildfire, Policy-2 clearly outperforms Policy-1. Policy-2 is able to suppress wildfires up to a correction-coefficient value of $cc=0.19$, whereas Policy-1 fails at $cc=0.15$. At $cc$ = 0.14, Policy-2 demonstrated superior performance, resulting in a reduction of 27,000 m² in the final burned area compared to Policy-1. One contributing factor is that Policy-1 does not prioritize the complete extinguishment of individual fire fronts and frequently leaves isolated burning cells unextinguished. These isolated cells can later develop into new fire fronts, increasing the total burned area and reducing overall suppression effectiveness. In contrast, Policy-2 learns to establish a continuous suppression line, completely extinguishing a fire front before advancing to extinguish the remaining fire area. This results in a more coherent and efficient suppression strategy, highlighting the advantage of incorporating the Connected Suppression Reward.

To evaluate the effectiveness of the RL policy itself, the heuristic tactic is used as a baseline. In particular, the direct tactic as defined in Section 2.1, is selected, as it represents the best-performing heuristic strategy within the scenario used for the RL training. Figure 3 illustrates the performance comparison between the two strategies. The results show that the RL tactic outperforms the heuristic strategy for cc-values between $cc=0.10$ and $cc=0.16$, achieving approximately 5000 m² less burned area in this range. However, at a higher sensitivity of $cc=0.19$, the heuristic tactic demonstrates a clear advantage, resulting in roughly 20,000 m² less burned area compared to the RL Policy. One reason for the performance of the RL policy at lower sensitivities is that, by extinguishing a small number of strategically important cells, the agents delay the fire’s advance toward a slope that would otherwise accelerate its spread. This brief intervention reduces overall fire growth and contributes to the RL tactic outperforming the heuristic strategy under these conditions.

4.2. Conclusions

Although the results are preliminary, the experiments demonstrate the potential of MARL to develop effective strategies for wildfire fighting. With the current configuration, the RL strategy can even outperform heuristic tactics in fires with a low spread sensitivity. However, as the spread sensitivity increases, the RL policy is no longer able to fully suppress the wildfire. This indicates that the learned policy overfits to the specific spread dynamics observed during training and does not generalize well to more challenging conditions. Furthermore, the experiments highlight the applicability of the Dual Decomposition Framework, demonstrating how individual reward components can be isolated and evaluated to better understand their influence on the learned behavior. This systematic reward analysis increases the interpretability of the agent’s decision-making and supports the development of effective, explainable suppression strategies.

To improve generalization, curriculum learning could be applied by gradually increasing the complexity of wildfires during training, resulting in policies that remain robust even with a higher spread sensitivity. Additionally, the observation space and reward function can be extended and further analyzed using the Dual Decomposition Framework. Future evaluations may also incorporate broader metrics to enable a more comprehensive assessment of suppression tactics. In addition, training with diverse aircraft types would allow exploration of the fleet-composition space, supporting the identification of optimal fleet sizes and combinations for different wildfire scenarios. Replacing the shared policy with individual agent policies may reveal emergent heterogeneous behaviors and would also enable investigation of how direct inter-agent communication affects coordination effectiveness.