Directional Pheromone Gradient Observations for Decentralized Multi-Agent Reinforcement Learning in Swarm Drone Search and Rescue

Highlights

What are the main findings?

• A novel directional pheromone-gradient observation is introduced to enhance decentralized MARL coordination in swarm-drone search-and-rescue missions.
• The proposed Hybrid stigmergy–MARL framework achieved 98.9% area coverage and 93.3% victim detection, significantly outperforming RL-only and non-learning baselines.

What are the implications of the main findings?

• Directional pheromone gradients provide a low-bandwidth environmental memory that enables effective swarm coordination without centralized control or continuous communication.
• The swarm maintained approximately 99% coverage even under total communication loss, demonstrating strong decentralized coordination without reliance on active communication.

Abstract

Search-and-rescue (SAR) operations in disaster environments require drone swarms to coordinate efficiently despite incomplete information and potential communication failures. Existing stigmergy-based approaches provide low-bandwidth coordination but rely on fixed rules, whereas multi-agent reinforcement learning (MARL) can learn adaptive behaviors but often struggles with coordination under partial observability. To address these limitations, this paper proposes a Hybrid stigmergy–MARL framework that introduces directional pheromone-gradient observations, enabling each drone to infer the direction of likely victims and unexplored regions using locally available information. The proposed framework combines reinforcement learning with four virtual pheromone layers representing coverage history, victim likelihood, environmental risk, and communication quality. Victim detection is modeled through an abstract short-range thermal/visual sensing mechanism, while environmental information is shared through pheromone-based environmental memory to reduce dependence on direct communication. The simulated environment consists of a 40 × 40 grid, where each grid cell represents a discrete two-dimensional location. Victims occupy a single grid cell, and obstacles are modeled as static two-dimensional impassable cells. Experimental results show that the proposed approach achieved 98.9% area coverage and 93.3% victim detection, compared with 81.8% coverage and 71.7% victim detection for the RL-only baseline. Ablation experiments confirmed that directional gradient observations are the primary contributor to these improvements, while communication-loss experiments demonstrated robust performance even under complete communication outage. These findings indicate that directional pheromone-gradient observations provide an effective and communication-efficient mechanism for decentralized swarm coordination, improving search effectiveness and operational robustness in post-disaster SAR scenarios.

1. Introduction

Natural and man-made catastrophes result in hazardous or impenetrable terrain for humans to conduct rescue missions. Quick localization of survivors is key, as research shows less than 50% of rescue probability within 24 h after a structural collapse [1]. Unmanned aerial vehicles (UAVs) can swiftly deploy, capture thermal images, and enable communication despite broken ground-level facilities [2].

One drone alone would be incapable of handling a vast area due to battery limits, sensing range constraints, and the possibility of broken communication. Multiple drones can share tasks [3,4]; however, a fundamental question emerges in such an application scenario: how can many drones coordinate themselves effectively without having a delicate central controller when communication is broken?

This problem has been previously solved in one of two ways, namely through stigmergic principles [5], which enable indirect communication through leaving environmental traces similar to those of social insects, and multi-agent reinforcement learning (MARL) [6]. Stigmergy is efficient in communication costs but not adaptive since it requires fixed rules. MARL is capable of learning policies that can adapt to changing circumstances yet is burdened by non-stationarity and sample inefficiency [7,8].

The critical idea presented by this paper is that by giving directional pheromone gradients as inputs to the RL observation, we can connect both worlds: stigmergy allows for the encoding of structured low-bandwidth coordination memory, whereas RL learns how to interpret and make use of this memory flexibly. Instead of considering the pheromone values as environmental variables, our design of the observation allows us to encode the direction of the victim information and unexplored areas. Unlike conventional pheromone-based systems that provide only local scalar intensity values [9], directional pheromone gradients encode how victim-likelihood or coverage information varies across different spatial directions around a drone [10,11,12]. In practical terms, a drone can determine not only whether useful information exists nearby, but also the direction in which that information is strongest. This reduces ambiguity during decision making and allows agents to navigate more purposefully toward victim-related evidence and unexplored regions [13].

Recent disasters such as earthquakes, building collapses, floods, and wildfires have demonstrated the importance of rapidly locating survivors in environments where infrastructure may be damaged and communication networks partially unavailable [14,15,16]. In such situations, drone swarms must coordinate using incomplete local information while maintaining operational effectiveness despite communication disruptions [17,18]. This motivates the need for decentralized coordination mechanisms that can guide exploration efficiently without continuous reliance on centralized control [16,19,20,21].

The main research question is as follows: How do directional pheromone-gradient observations and neighbor-density awareness improve decentralized multi-agent reinforcement learning (MARL) coordination for multi-drone SAR missions in uncertain disaster environments? To answer this question, three supporting questions are investigated: (1) Can directional pheromone-gradient observations improve area coverage and victim discovery compared with conventional scalar observations? (2) How much does each proposed observation component contribute to overall performance? (3) Can the learned coordination strategy remain effective under partial or complete communication loss? These questions are examined through comparative evaluation, ablation analysis, and communication-loss experiments.

The manuscript is organized as follows: Section 2 reviews related work. Section 3 describes the system and methodology. Section 4 reports results including ablation and communication loss experiments. Section 5 concludes.

The principal novelty of this work is the incorporation of directional pheromone-gradient observations into the agent observation space. Rather than relying solely on local pheromone values, each drone receives directional information about victim likelihood and coverage distribution, allowing learned policies to exploit structured environmental cues for decentralized coordination under partial observability.

The main contributions are as follows:

• A novel directional pheromone-gradient observation representation for decentralized MARL, where drones observe directional victim-likelihood and coverage cues rather than only local scalar pheromone values.
• A decentralized coordination mechanism that operates without requiring a global map, centralized control, or continuous communication, making it suitable for disrupted disaster environments.
• A multi-layer virtual pheromone model comprising coverage, victim-likelihood, risk, and communication layers that act as a shared environmental memory for swarm coordination.
• A Hybrid stigmergy–MARL framework that combines environmental memory with adaptive learning, achieving 98.9% coverage and 93.3% victim detection in the evaluated SAR scenarios.
• A comprehensive evaluation including benchmark comparisons, ablation analysis, and communication-loss experiments that identify directional gradients as the primary contributor to performance gains.

2. Literature Review

2.1. Swarm Robotics

In swarm robotics, the behavior of large numbers of simple robots is studied as a useful collective behavior, arising from local rules and decentralized interaction [22]. The field is inspired by swarm intelligence proposed by Beni and Wang [23].These systems are attractive because they are easy to use: they scale, they are resilient to failure, and they can handle several actions simultaneously. That last point is more important than usual during disaster response. If one drone is down, the mission should not be stopped [24].

2.2. Real-Time Execution Requirements in Decentralized Autonomous Systems

Effective swarm coordination requires not only cooperative behavior but also computationally efficient onboard decision-making. In decentralized systems, individual agents must operate with limited processing resources, incomplete information, and strict real-time constraints [25]. These requirements are particularly important in disaster-response scenarios, where delayed decisions may reduce mission effectiveness and communication with external infrastructure may be unreliable. Recent studies have demonstrated that lightweight machine learning models can provide accurate and low-latency decision support in resource-constrained environments [26,27]. Although these works address application domains different from swarm robotics, they collectively emphasize a common principle: practical autonomous systems benefit from models that maintain strong performance while remaining computationally efficient. This is especially relevant for multi-UAV operations, where each drone must continuously process sensor information, make decisions, and coordinate with neighboring agents during mission execution [28,29,30,31]. These observations highlight the need for coordination mechanisms that provide useful guidance without introducing substantial computational or communication overhead. Motivated by this requirement, the proposed framework employs a compact observation representation based on directional pheromone-gradient information, allowing decentralized agents to exploit structured environmental cues while maintaining efficient onboard execution.

2.3. Stigmergy and Virtual Pheromones

Stigmergy and Virtual Pheromones: Stigmergy is a type of coordination without direct communication between agents. Rather, they change a common environment, and other agents respond to the changes made [5,32]. The idea is based on social insects, bees and ants, in which no central authority is in charge of the group and it arises from simple local actions [33]. The agents leave a trace, and the swarm reads this trace to make  adjustments.

The most popular computational implementation of this idea is Ant Colony Optimization (ACO) [33]. Useful paths and distributed route information develop over time as ants leave pheromones on them that will affect subsequent ants [34]. In swarm robotics, the idea is taken to a different level and the physical pheromone fields are replaced by virtual pheromone fields, which are stored, modified, and sensed by the robots digitally.

The virtual pheromone can be used in the search-and-rescue (SAR) application, where it can facilitate the decentralized control without requiring the ongoing communication among agents [35]. Agents can tag explored areas, probable victim locations, or unsafe locations, and other agents within that area can update their behaviors accordingly. However, the problem in the present SAR systems is that the majority of the existing ones follow a fixed rule and some of the primitive pheromone design. It reduces their adaptability, particularly in a mission that demands exploring, locating victims and avoiding the dangers in fluctuating environments.

While stigmergy provides an efficient mechanism for decentralized coordination with minimal communication overhead, its behavior is largely determined by predefined rules and fixed pheromone-update strategies [36]. This limitation motivates the use of learning-based approaches that can adapt coordination behavior to changing mission conditions, as discussed in the next subsection.

2.4. Reinforcement Learning and MARL for Coverage

Reinforcement learning has been providing autonomous agents with a means to learn decision-making by interacting with the environment [37]. Multi-agent reinforcement learning (MARL) extends this to multi-agent systems, with multiple agents needing to learn to cooperate with each other in a partially observed environment with decentralized execution. The combination of that makes MARL suitable for UAV search, exploration and coverage as agents the operate in dynamic environments that are uncertain and partially visible.

A few studies have focused on directly addressing multi-UAV coordination using MARL. Pham et al. [7] presented a cooperative distributed RL approach for area-searching behavior in UAV field coverage, demonstrating that a set of multiple agents could learn coordinated area-searching behavior from local interaction without the need for centralized control. Later, MARVEL combined graph attention mechanisms into a MARL framework to enhance coordination for multi-robot environments with a large number of robots [8]. These works highlight the ability of MARL to learn complex cooperative behaviors without relying on handcrafted control policies.

The growing relevance of MARL for UAV coordination is reflected in recent surveys. Ekechi et al. identified MARL as a promising approach for UAV control in dynamic and uncertain environments [31], while Blais and Akhloufi reviewed advances in RL-based swarm robotics and emphasized the increasing role of learning-based coordination in multi-robot systems [38]. In disaster-response scenarios, Lee et al. investigated reinforcement learning for UAV-assisted emergency communication [39], and Tan and Zhao applied deep reinforcement learning to coordinate multiple UAVs for post-disaster search-and-rescue operations [1].

Despite these advances, most MARL-based approaches learn coordination solely from agent observations or explicit communication mechanisms. Relatively little attention has been given to incorporating structured environmental memory into the learning process [40]. This is what leads to the need for agents to relearn useful coordination patterns through long training or rely on communication methods that may fail in disaster settings. This creates a clear need to combine stigmergic environmental cues with learned policies. Agents can then use shared information left in the environment while still keeping the flexibility of reinforcement learning.

Although MARL enables adaptive coordination, most existing approaches rely solely on local observations or explicit communication [41]. This raises an important representation challenge: what information should agents observe to support effective coordination under partial observability? Recent advances in representation learning provide useful insights into this question.

2.5. Representation Learning in MARL

Representation learning in MARL Methods for encoding observations directly influence coordination, their robustness and speed of convergence. Raw observations are sometimes not enough. Recent research in MARL has progressed towards the development of structured representations with spatial, temporal, or relational attributes, which enable agents to process partial observability, non-stationarity, and sparse exploration signals.

Hamdi et al. [42] introduce a multimodal attentional representation learning framework which integrates visual and temporal data. CNNs are used to extract features from the visual data, LSTMs to identify temporal patterns, and an attention mechanism to allow the model to concentrate on the most informative inputs. The multimodality joint representation-based learning outperforms single-modality baselines, achieving up to 97% accuracy in classification and a 19–57% boost in prediction performance. This finding is consistent with a larger hypothesis: when there are unclear or inconsistent observations, multimodal learning with attention is beneficial.

Li et al. [43] propose RACE that decouples the representation learning from policy learning in a MARL system. The agents are equipped with a shared observation encoder, while they have different policy representations; this means that transferring knowledge does not result in the same behavior being enforced. RACE deals with the challenges of partial observability by maximizing value-aware mutual information, thus learning features that contain globally relevant information. In several benchmark tasks, it achieves better coordination quality and convergence rate than baseline MARL approaches. The main point is clear: structured representations plus information-aware objectives can make multi-agent learning more effective.

Xu et al. [44] focus on multi-view representation learning, where data from several sources are fused into a more complete representation. Their Progressive Deep Multi-view Fusion (PDMF) method uses two stages. It consists of pre-training to capture relationships between views, then fine-tuning to map partial representations into complete ones. The model captures both consistency and complementarity across views, which fixes some limits of standard fusion methods. Experiments on multiple datasets show steady gains over state-of-the-art approaches, with classification accuracy improving by up to several percentage points. This supports the value of structured feature integration and progressive learning for better robustness and generalization.

Together, these studies show that stronger representation learning, through multimodal fusion, shared encoders, or multi-view integration, can improve learning efficiency, stability, and coordination in complex environments. For MARL, this supports the use of structured observation representations, where richer inputs can directly shape better policy performance.

These findings suggest that carefully designed observation representations can significantly influence coordination quality and learning efficiency. This observation motivates the use of structured environmental representations, such as directional pheromone gradients, within decentralized MARL systems.

2.6. Communication and Coordination in MARL

Communication and coordination are still hard problems in multi-agent reinforcement learning (MARL) especially when the system becomes larger, agents see only part of the environment and bandwidth is limited. Recent work has started to use learnable communication methods like dynamic coordination set selection, noise-aware communication and shared memory structures [45]. These methods do not treat communication as a fixed channel. Instead, they learn when agents should communicate, what information they should send, and how it should be shared.

Zhang and Lesser [46] propose a utility-based method helping agents find useful coordination sets during learning and reducing communication cost while keeping performance. Tung et al. [47] study learning to communicate over noisy physical channels where policies and channel coding are learned together rather than separately. Pesce and Montana [48] introduce a memory-based communication architecture that lets agents form emergent protocols for tasks that need close synchronization. Together, these works show a clear shift in MARL where agents coordinate better when communication and policy learning are designed jointly.

Zhang and Lesser [46] deal with scalability by allowing agents to find useful coordination sets while they learn. Their method uses an interaction measure to estimate how much performance may be lost if agents do not coordinate. Agents communicate only when coordination is important. This reduces overhead while keeping performance close to the full-communication setting. In sensor network target tracking the method reduces communication by more than 80% with only a small performance drop. This shows the trade-off between performance and bandwidth use.

Tung et al. [47] study communication under noisy physical channels. They formulate the problem as a MARL setting, where noisy channels are part of the environment dynamics rather than an external issue. Their MA-POMDP model moves beyond the common error-free communication assumption and learns cooperative policies together with robust channel coding. Experiments on binary symmetric, AWGN, and bursty noise channels show that joint learning performs better than separation-based methods. In joint channel coding, the learned schemes outperform Hamming codes by up to 3.70 dB in block error rate.

Pesce and Montana [48] propose MD-MADDPG, which uses a shared memory device as a learnable communication channel under the centralized training and decentralized execution paradigm. Agents learn gated read and write operations so they can update and interpret memory while also learning their policies. Tested on six coordination-heavy navigation tasks, MD-MADDPG performs better than MADDPG and other baselines, mainly in synchronization tasks. The memory visualizations show task-related usage patterns, and the ablation results confirm that the main architectural components are needed.

2.7. UAVs in Disaster SAR

In SAR missions, UAVs are highly important since they enable fast exploration of hazardous, damaged, or difficult-to-reach areas without risking the lives of the searchers. According to the recent literature, UAVs aid SAR operations with respect to rapid deployment, aerial surveillance, sensor-based victim identification, and operation in restricted areas [49]. Recent studies show how drone teams can coordinate effectively to search challenging mountain environments [50], while advanced visual and thermal sensors make it easier to locate, track, and evaluate victims in real time [51,52,53].

The use of multiple UAVs enhances the performance of SAR tasks through the parallel exploration of various regions. Yet, several issues arise when employing multi-UAVs, including mission allocation, avoiding collisions, limited battery capacity, and poor communication in disaster zones. According to Ghauri et al. [54], the problem of dynamic mission allocation remains one of the critical issues for multiple UAVs conducting SAR activities.

2.8. Research Gap

Existing MARL approaches focus on learning coordination through policy optimization or explicit communication, while stigmergy-based systems rely on fixed rules without adaptability. However, little work explores structured, directional representations of environmental memory that can guide decentralized policies without communication overhead.

In particular, current approaches either depend on learned communication strategies, which introduce additional system complexity, or rely on static stigmergic rules that lack adaptability in dynamic environments. This creates a gap in developing methods that retain the efficiency of stigmergy while enabling adaptive, learned coordination.

These particular gaps prompt this research in three ways. First, Salman et al. [35] perform automated stigmergy behavior design yet lack the learning mechanism whereby their system cannot prioritize adaptively. Second, Tan and Zhao [1] implement a deep RL strategy for multiple UAVs in SAR tasks using a centralized approach that falls short when the base station gets compromised. Third, Pham et al. [7] and MARVEL [8] illustrate the use of reinforcement learning techniques in multi-robot task accomplishment while excluding the use of pheromones to communicate past actions. This means that the robots have to learn how to coordinate with each other based solely on observation. To the best of our knowledge, this is among the first studies to incorporate directional pheromone-gradient observations as structured environmental-memory features within a CTDE-based MARL framework for decentralized UAV search-and-rescue operations.

3. Methodology

The methodology is designed to isolate the impact of directional pheromone gradient observations on decentralized coordination performance, ensuring that all other components of the system remain consistent across compared approaches.

3.1. Problem Formulation and Environment

Disaster search-and-rescue (SAR) operations are inherently stochastic and partially observable. The locations of victims are not known, there may be obstacles in the way, and communication between agents may be faulty. Hence, the problem is formulated as a Decentralized Partially Observable Markov Decision Process (Dec-POMDP), where each UAV makes decisions based on its own local observations. The Dec-POMDP is defined by the tuple $\mathcal{M}=\langle \mathcal{N},\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R},\mathsf{\Omega },\mathcal{O},\gamma \rangle$, as detailed in

Table 1. Multi-UAV SAR modeling Dec-POMDP tuple definitions.

Component	Description
$\mathcal{N}=\{1,\dots ,N\}$	Set of N autonomous drone agents ($N=10$).
$\mathcal{S}$	Global state space, including grid cell occupancy, static obstacle and hazard coordinates, victim locations, and four-layer virtual pheromone intensities across all cells.
$\mathcal{A}={\prod }_{i=1}^N{\mathcal{A}}_i$	Joint action space, where each agent’s discrete action space is ${\mathcal{A}}_i=\{\mathrm{North}, \mathrm{South}, \mathrm{East}, \mathrm{West}, \mathrm{Hover}\}$.
$\mathcal{P}(s^{\prime }\mid s,\mathbf{a})$	State transition probability function $\mathcal{S}\times \mathcal{A}\times \mathcal{S}\to [0,1]$ governing movement and environmental virtual pheromone updates like evaporation or diffusion.
${\mathcal{R}}_i(s,\mathbf{a})$	Individual scalar reward for agent i, $\mathcal{S}\times \mathcal{A}\to \mathbb{R}$, computed from that agent’s own local outcomes (collision, hazard entry, newly-scanned cells, victim detection, energy cost; Equation (3)).
$\mathsf{\Omega },\mathcal{O}(s,i)$	$\mathsf{\Omega }={\prod }_{i=1}^N{\mathsf{\Omega }}_i$ is the joint observation space, where agent i receives local observation vector $o_i\in {\mathsf{\Omega }}_i$ generated by observation function $\mathcal{O}(s,i):\mathcal{S}\times \mathcal{N}\to {\mathsf{\Omega }}_i$.
$\gamma$	Discount factor $\gamma \in [0,1)$ (${\gamma }_{RL}=0.93$).

.

The disaster site is modeled by a grid G, with each square $c\in G$ possibly being free, obstructed, dangerous, searched, or occupied by a victim. The environment spans a 40 × 40 grid (1600 cells in total), providing enough space for meaningful multi-drone exploration and coordination while remaining computationally tractable for tabular Q-learning. The grid has no physical units; cell size, drone speed, and mission duration are not mapped to anything real-world in this study. As a rough intuition only, a cell might correspond to a 5 m × 5 m patch of ground, roughly what a downward-facing camera would cover at low SAR altitude, putting the whole grid at about 200 m × 200 m. That mapping does not feed into any of the reported numbers. At each timestep t, UAV i has a local observation $o_i^t$, current location $p_i^t$, current battery life $b_i^t$, and an action $a_i^t$. Each agent can see within its own local window, with a sensing range of $r_s=2$.

The environment is generated procedurally using a fixed random seed per episode. Obstacles, hazard zones, and victim locations are sampled uniformly at random without replacement from all grid cells outside a $9\times 9$ central exclusion zone reserved for drone initialisation. Obstacles are impassable; hazard zones are traversable but penalize entry (reward $-\eta$) and deposit risk pheromone ${\tau }_r$. Both are static for the duration of each episode—neither obstacles nor hazard zones move. Victims are likewise stationary; their locations are unknown to agents until physically detected within sensing range ($r_s=2$). The drones are uniformly randomized in the central $7\times 7$ square. No two drones occupy the same grid point in the starting position. Each of the 30 iterations uses a distinct map so that the performance on different maps can be measured. The entire source code of the simulation is provided along with this paper.

3.2. System Architecture

The proposed system has five layers, shown in Figure 1. The architecture establishes a multi-tier data pipeline where solid downward arrows represent the primary feed-forward data flow, routing sensor streams down through mapping to policy execution. The sensing layercollects RGB, thermal, LiDAR, IMU, and communication-signal data. Figure 1’s sensing layer lists the full suite intended for real deployment, RGB, thermal, LiDAR, IMU, communication. What is actually simulated here is narrower: victim detection abstracts to the short-range thermal or visual sensor already described in Section 3.4 below, while obstacle avoidance and localization run on grid-level ground-truth occupancy rather than simulated LiDAR or IMU data.

The mapping layer builds local occupancy grids and victim-probability grids, then merges them across drones when communication is available, see Figure 2 for more details. The stigmergy layer maintains and spreads the four pheromone maps using Equation (1). Each pheromone layer is updated once per timestep across all drones acting in that step: newly sensed evidence is added via deposit, then evaporation reduces every cell’s existing value by a fixed fraction, and diffusion redistributes a portion of each cell’s value to its immediate neighbors (Equation (1)). This means information left by one drone becomes visible to a physically nearby drone within a few timesteps even without direct communication, since the pheromone field itself carries the information forward.

Crucially, the connection between the Stigmergy and Learning layers involves two distinct pathways in Figure 1:

• Directional Feature Extraction (solid arrow, Stigmergy to Learning): Rather than feeding raw scalar maps into the policy, the stigmergy layer extracts three-cell directional gradients (${\nabla }_v,{\nabla }_c$) across cardinal cones and passes these structured vectors into the learning layer’s observation space.
• Pheromone Feedback Updates (dashed arrow, Learning to Stigmergy): Following action execution, pheromone deposits ($\mathsf{\Delta }{\tau }_c,\mathsf{\Delta }{\tau }_v,\mathsf{\Delta }{\tau }_r$) are written back into the stigmergy layer to continuously evolve the shared environmental memory maps.

The learning layer runs the MARL policy. An optional mission-control layer tracks global progress and assigns area priorities. When it is disconnected, the system still runs in a fully decentralized way.

The proposed observation formulation is independent of the specific environment structure and can be applied to any decentralized MARL setting where spatial information is represented through local maps or distributed environmental fields.

3.3. Virtual Pheromone Model

To support indirect coordination, the proposed system uses four virtual pheromone layers inspired by stigmergy, see Figure 3 for more details. The coverage layer ${\tau }_c$ marks scanned cells, the victim-likelihood layer ${\tau }_v$ stores possible human-presence evidence, the risk layer ${\tau }_r$ marks hazardous regions, and the communication layer ${\tau }_m$ identifies useful relay areas.

Each pheromone layer is updated through evaporation, diffusion, and new deposits:

$${\tau }_k^{t+1}\left(c\right)=(1-{\rho }_k)\left[{\tau }_k^t\left(c\right)+D_k\sum _{c^{\prime }\in \mathcal{N}\left(c\right)}\left[{\tau }_k^t\left(c^{\prime }\right)-{\tau }_k^t\left(c\right)\right]\right],$$

(1)

where $k\in \{c,v,r,m\}$, ${\rho }_k$ is the evaporation rate, $D_k$ is the diffusion coefficient, and $\mathcal{N}\left(c\right)$ is the four-cell neighborhood, see Figure 4 for more details. New pheromone deposits $\mathsf{\Delta }{\tau }_k^t\left(c\right)$ are added separately at the cell a drone currently occupies, and are not decayed until the following timestep’s update (Equation (1)) is applied. The victim-likelihood layer uses the slowest evaporation rate, ${\rho }_v=0.02$, so possible victim evidence remains available for longer. The three pheromone-update terms in Equation (1) have the following operational meaning. Evaporation (${\rho }_k$) is a fixed fractional decay applied to the existing value at each cell every timestep, causing older evidence to fade over time. Diffusion ($D_k$) redistributes each cell’s value toward its four-connected neighbors, allowing information to spread spatially even to cells a drone has not directly visited. Deposit ($\mathsf{\Delta }{\tau }_k$) is an additive increment applied at the cell a drone currently occupies, representing new sensed evidence. All four pheromone layers are dimensionless scalar fields: their values represent relative signal strength rather than a physical quantity and are used only for comparison against one another (e.g., in the stigmergy-only scoring rule) and as discretized Q-table inputs.

3.4. Directional Pheromone Gradient Observation

The central design contribution is an enriched observation vector. Each drone i observes

$$o_i=\left({\tau }_c,{\tau }_v,{\tau }_r,{\tau }_m,b_{\mathrm{norm}},{\nabla }_v,{\nabla }_c,{\rho }_{\mathrm{neigh}},w_N,w_S,w_W,w_E\right)$$

(2)

where ${\nabla }_v=({\overline{\tau }}_v^N,{\overline{\tau }}_v^S,{\overline{\tau }}_v^W,{\overline{\tau }}_v^E)$ is the directional victim-pheromone gradient. It is computed as the mean ${\tau }_v$ over a three-cell cone in each cardinal direction. The term ${\nabla }_c$ is computed in the same way, but for the coverage pheromone. ${\rho }_{\mathrm{neigh}}$ gives the normalized number of drones within $r_c$, while $w_d$ are binary obstacle indicators for the four directions where $w_N,w_S,w_W,w_E\in \{0,1\}$ represent whether the immediate adjacent cell in the north, south, west, or east direction is blocked by an impassable obstacle (1) or clear (0). Neighbor density is taken from physical drone positions, so it gives short-range proximity awareness without active message exchange. It remains available even when communication is lost.

This directional representation reduces decision ambiguity. With only scalar pheromone values at the current cell, a drone knows what is happening where it stands but not where the useful signal is stronger. It leads to inefficient exploration or back-and-forth movement under partial observability conditions. Representation by directional trends ${\nabla }_v$ and ${\nabla }_c$ provides a straightforward hint to the policy about where more victim-likelihood evidence is stronger or where there is less coverage. In such a way, drones can explore more purposefully without the need for a global map or additional communication.

The detection of a victim takes place based on a deterministic binary scheme assuming perfect detection within range ($P(\mathrm{detection}\mid \mathrm{in}\mathrm{range})=1.0$) and zero false alarms ($P\left(\mathrm{false}\mathrm{alarm}\right)=0.0$). The presence of a victim at the grid cell $(r,c)$ is determined if the drone visits any cell $(r^{\prime },c^{\prime })$, for which $\max \left(\right|r^{\prime }-r|,|c^{\prime }-c\left|\right)\le r_s$. This means that the victim is located within a drone’s $5\times 5$ sensing window, where $r_s=2$. Upon detection, the value of $\mathsf{\Delta }{\tau }_v=1.5$ is added to the drone’s current position, and the victim is considered to be discovered. Positions of victims are unknown to all agents at the beginning of each episode; they are not included in the initial state or observation. The model abstracts a short-range thermal or visual sensor, consistent with the sensing layer shown in Figure 1 and discussed in Section 3.2. While this deterministic model isolates exploration performance from sensor noise, real-world SAR sensors inevitably produce false alarms and missed detections. In practice, missed detections which represent false negatives would require a slower pheromone decay rate (${\rho }_v$) so clues do not fade before nearby drones arrive. Conversely, false alarms representing false positives would briefly draw drones off-course until they can verify the location up close. Testing noisy, probabilistic sensor models is left for future work.

Overall, the added directional features make the observation more informative than scalar pheromone inputs alone. They let each agent infer local spatial gradients of coverage and victim likelihood, reduce ambiguity in action selection, and support more stable navigation in partially observable environments.

These features add more than scalar pheromone values alone. If a drone only observes ${\tau }_v$ at its current cell, it knows there is some victim-related signal nearby but not where that signal is coming from. With ${\nabla }_v$, it can move toward the strongest victim-likelihood direction without needing a global map or explicit message passing.

Unlike conventional methods based only on scalar local observations, the proposed representation encodes directional trends. This lets each agent infer spatial gradients of relevant environmental signals, reduces ambiguity in action selection and supports more purposeful navigation under partial observability.

Similarly, ${\nabla }_c$ encodes which direction has the least explored space, enabling learned dispersion without inter-agent negotiation. The neighbor density ${\rho }_{\mathrm{neigh}}$ allows the policy to avoid redundant clustering even when no explicit task-assignment protocol exists. An ablation study in Section 5 quantifies the individual contribution of each feature.

The proposed directional features introduce minimal computational overhead, as gradient values are computed from local pheromone maps using simple averaging operations over small spatial neighborhoods. A comparison between conventional scalar observations and the proposed directional representation is summarized in

Table 2. Comparison between scalar and directional observation representations. Coverage variability is the pooled standard deviation across all 150 evaluation runs; the coverage gain is the absolute increase from RL-only’s 81.8% to Hybrid’s 98.9%.

Property	Scalar (RL-Only)	Directional (Hybrid)
Observation dimensionality	9 features	18 features
Coverage variability (std)	10.8%	2.7%
Mean steps to first victim	5.8	3.2
Coverage gain	—	+17.1%

.

3.5. Hybrid Decision Process

Algorithm 1 formalizes the complete procedure under the Centralized Training with Decentralized Execution (CTDE) paradigm, explicitly differentiating the global training workflow from the fully decentralized execution phase.

Algorithm 1 operates across two distinct phases under the CTDE paradigm: Centralized Training (lines 1–17) and Decentralized Execution (lines 19–32). During the training phase, each drone extracts local observations $o_i^t$ using its sensing radius ($r_s=2$), three-cell directional gradients ${\nabla }_v,{\nabla }_c$, and neighbor density ${\rho }_{\mathrm{neigh}}$ (lines 6–7), selecting actions via an $\epsilon$-greedy policy on $Q_i$ (line 8). After executing joint actions and depositing local pheromones (lines 10–11), a scalar centralized team reward $R_t$ is computed from the global state $s_t$ (line 12) to update each individual action-value function $Q_i$ (line 14).

During the decentralized execution phase (lines 19–32), each drone operates autonomously onboard without access to global state or a central controller (lines 20–21). Every agent senses its local window ($r_s=2$), deposits corresponding coverage, victim, or risk pheromones ($\mathsf{\Delta }{\tau }_c,\mathsf{\Delta }{\tau }_v,\mathsf{\Delta }{\tau }_r$) into its local map (line 22), extracts directional gradients and neighbor density (line 23), and constructs $o_i^t$ (line 24). Drones select actions greedily based on $Q_i(o_i^t,·)$ (line 25) and execute them locally (line 26). If another drone falls within communication radius $r_c$, compressed pheromone updates are exchanged (lines 27–29), after which local evaporation and diffusion are applied across all pheromone layers (line 31). This execution loop runs independently across agents at every timestep, requiring no centralized coordinator.

Algorithm 1 Hybrid Stigmergy-MARL Framework (CTDE Paradigm).

Require: State space $\mathcal{S}$, sensing radius $r_s$, communication radius $r_c$, discount factor ${\gamma }_{\mathrm{RL}}$, learning rate ${\eta }_{\mathrm{RL}}$, decay schedule $\epsilon$

Ensure: Learned action-value functions $Q_i$ for each agent $i\in \mathcal{N}$

1: // Phase 1: Centralized Training Phase (CTDE Framework)   2: for episode $e=1\dots E$ do   3:    Reset grid environment $s_0$; clear/initialize virtual pheromone maps $\left\{{\tau }_k\right\}$   4:    for step $t=1\dots T$ do   5:      for each agent $i\in \mathcal{N}$ do   6:         Compute agent i’s individual scalar reward $R_t^i$ from its own local outcomes (Equation (3))   7:         Update action-value function: $Q_i(o_i^t,a_i^t)\leftarrow Q_i(o_i^t,a_i^t)+{\eta }_{\mathrm{RL}}\left[R_t^i+{\gamma }_{\mathrm{RL}}{\max }_{a^{\prime }}{Q}_i(o_i^{t+1},a^{\prime })-Q_i(o_i^t,a_i^t)\right]$   8:      end for   9:      Execute joint action ${\mathbf{a}}^t=(a_1^t,\dots ,a_N^t)$ in environment 10:      Deposit pheromones $(\mathsf{\Delta }{\tau }_c, \mathsf{\Delta }{\tau }_v, \mathsf{\Delta }{\tau }_r)$ and update map evolution (Equation (1)) 11:      Compute scalar centralized team reward $R_t$ from global state $s_t$ (Equation (3)) 12:      for each agent $i\in \mathcal{N}$ do 13:         Update action-value function: $Q_i(o_i^t,a_i^t)\leftarrow Q_i(o_i^t,a_i^t)+{\alpha }_{\mathrm{RL}}\left[R_t+{\gamma }_{\mathrm{RL}}{\max }_{a^{\prime }}{Q}_i(o_i^{t+1},a^{\prime })-Q_i(o_i^t,a_i^t)\right]$ 14:      end for 15:    end for 16: end for 17: $$ 18: // Phase 2: Decentralized Execution Phase (Evaluation/Deployment) 19: for step $t=1\dots T$ do 20:    for each agent $i\in \mathcal{N}$ in parallel (Onboard Execution) do 21:      Sense local window ($r_s=2$); deposit $\mathsf{\Delta }{\tau }_c$, $\mathsf{\Delta }{\tau }_v$, or $\mathsf{\Delta }{\tau }_r$ into local map 22:      Compute local directional gradients ${\nabla }_v,{\nabla }_c$ and neighbor density ${\rho }_{\mathrm{neigh}}$ 23:      Construct $o_i^t$ purely from local sensors (Equation (2)) without global state access 24:      Select greedy action $a_i^t=\arg {\max }_a{Q}_i(o_i^t,a)$ 25:      Execute action $a_i^t$ locally 26:      if drone j within communication radius $r_c$ then 27:         Broadcast compressed $\mathsf{\Delta }{\tau }_k$ updates to drone j 28:      end if 29:    end for 30:    Apply local evaporation-diffusion updates to pheromone fields (Equation (1)) 31: end for

3.6. Non-Learned Baselines

Three non-learned baselines are included for comparison. Random walkselects actions uniformly at random each step. Frontier search directs each drone to the nearest unvisited cell using Manhattan distance, with random tie-breaking. Stigmergy-only uses fixed hand-coded rules based solely on the shared pheromone field, with no learning: at each step, each drone scores the four cardinal moves using the linear combination $s=3{\tau }_v-2{\tau }_c-1.5{\tau }_r-2{\delta }_{\mathrm{crowd}}$, where ${\tau }_v$, ${\tau }_c$, ${\tau }_r$ are the pheromone values at the candidate cell and ${\delta }_{\mathrm{crowd}}$ penalizes cells occupied by other drones. A small uniform noise term $U(-0.05,0.05)$ breaks ties. The drone moves to the highest-scoring valid cell or hovers if all moves are blocked. This rule prioritizes victim-likelihood signals over coverage and avoids hazards and crowding, but the weights are fixed and cannot be adapted at runtime.

Obstacle handling is enforced at two levels. First, the environment itself treats any move into an obstacle cell as blocked for every method, including the learned policies: the drone’s position does not change, and a collision penalty $-\mu$ is applied (Equation (3)). Second, Frontier search and Stigmergy-only additionally avoid obstacles proactively: both methods exclude obstacle-occupied cells from their candidate move set before scoring, so they do not intentionally select a blocked cell as a target. Random walk performs no such filtering and selects uniformly among all five actions regardless of obstacles, relying solely on the environment’s collision handling; this is one contributor to its lower coverage and slower victim discovery relative to the other baselines.

3.7. Training Strategy

$$R_t=\alpha R_{\mathrm{victim}}+\beta R_{\mathrm{cov}}+\gamma R_{\mathrm{comm}}-\lambda R_{\mathrm{energy}}-\mu R_{\mathrm{coll}}-\eta R_{\mathrm{risk}}$$

(3)

Victim detection and cell creation are rewarded positively; energy consumption, collision, and entry into the danger zone are penalized. The victim coefficient $\alpha =5.0$ takes precedence over all others, guaranteeing that the drone algorithm favors victim detection above cell creation. This value was set by design rather than a formal hyperparameter search, chosen to lie well above the full range spanned by the remaining coefficients ($\lambda =0.1$ to $\mu =2.0$;

Table 3. Simulation parameters and reward coefficients.

Parameter	Description	Value
Grid/Drones/Victims	Environment setup	$40\times 40$/10/20
Obstacles/Hazards	Static obstacles/hazard zones	180/50
Sensing radius $r_s$	Per-drone view window	2 cells
Comm. radius $r_c$	Pheromone sharing range	6 cells
Battery budget	Steps per episode	500
${\rho }_c/{\rho }_v/{\rho }_r/{\rho }_m$	Evaporation rates	0.05/0.02/0.03/0.04
$D_c/D_v/D_r/D_m$	Diffusion coefficients	0.08/0.05/0.06/0.07
$\alpha$/$\beta$	Victim/coverage weight	5.0/1.0
$\gamma$/$\lambda$	Comm.energy weight	0.3/0.1
$\mu$/$\eta$	Collision/risk penalty	2.0/1.5
Learning rate ${\eta }_{\mathrm{RL}}$	Q-learning step	0.18
Discount ${\gamma }_{\mathrm{RL}}$	Future reward	0.93
$\epsilon$ schedule	Exp. decay (×0.9996/ep)	1.0 → 0.135 at ep 5000
Discretization bins	Per pheromone feature	6
Training episodes/runs	Per agent/eval seeds	5000/30

), ensuring victim detection dominates the reward signal regardless of how the other terms interact.

This weighting directly mirrors real-world search-and-rescue priorities: locate survivors first, explore the map second, and keep the drones safe throughout the mission. Setting $\alpha =5\times \beta$ ensures that whenever a drone picks up a victim trace, following that clue immediately takes priority over scanning empty ground ($\beta =1.0$). At the same time, penalties for collisions ($\mu =2.0$) and hazard zones ($\eta =1.5$) are set higher than coverage rewards ($\beta =1.0$), preventing the drones from taking reckless risks just to cover extra area. Finally, smaller terms for energy use ($\lambda =0.1$) and communication ($\gamma =0.3$) serve to prevent loitering and keep the swarm in relay range when no active victim leads are present.

Tabular Q-learning [55] is chosen deliberately for these experiments. It allows complete control of the observation space and makes it possible to separate the contribution of the suggested representation. It also helps avoid additional sources of noise caused by function approximation, making performance improvements associated solely with directional gradient representations easier to attribute.

The training regime adheres to the CTDE setup. During training, the simulator generates the global state used to calculate the reward, while during evaluation, each drone selects actions based only on its individual observation. The Q-tables have a six-bin quantization for each pheromone-relevant observation feature and are initialized in an optimistic way ($+2.0$) to promote exploration early in the training process. To avoid conflict with the victim reward weight $\alpha$ in Equation (3) the Q-learning step size is written as ${\eta }_{\mathrm{RL}}=0.18$ in Table 3. Exploration uses multiplicative decay, ${\epsilon }_{t+1}={\epsilon }_t\times 0.9996$, applied once per episode with a minimum value of 0.05. Across 5000 training episodes, $\epsilon$ drops from 1.0 to about 0.135 so exploration does not vanish. Each learned method is trained with five independent random seeds and results are reported as mean ± std across seeds to separate training-seed variance from evaluation variance. Statistical significance is tested using Welch’s t-test on pooled evaluation runs ($n=150$) along with Cohen’s d effect size and bootstrap 95% confidence intervals. For a fair comparison, all models use the same reward function, training process and environment conditions. However, only the observation representation changes.

The RL-only baseline uses the same learning rate, discount factor, exploration schedule, training length and evaluation protocol as the Hybrid model. It differs only in the observation or policy state.

Table 4. Policy observation state: Hybrid vs. RL-Only.

Feature	Encoding	Hybrid	RL-Only
Local coverage pheromone ${\tau }_c$	6 bins	✓	✓
Local victim pheromone ${\tau }_v$	6 bins	✓	✓
Local risk ${\tau }_r$	binary flag	✓	✓
Normalized battery $b_{\mathrm{norm}}$	4 bins	✓	✓
Obstacle indicators $w_{N,S,W,E}$	4 binary	✓	✓
Victim-gradient direction ${\nabla }_v$	N/S/W/E/none	✓	—
Coverage-gradient direction ${\nabla }_c$	N/S/W/E	✓	—
Neighbor density ${\rho }_{\mathrm{neigh}}$	3 bins	✓	—
Total state dimension		18	9

gives the exact features for each model. Both models share five base features: local coverage pheromone ${\tau }_c$ with six bins, local victim pheromone ${\tau }_v$ with six bins, a binary risk flag from ${\tau }_r$, normalized battery with four bins, and four binary obstacle indicators. The Hybrid model adds three more features: victim-gradient direction ${\nabla }_v$, defined as the dominant cardinal direction of ${\tau }_v$; coverage-gradient direction ${\nabla }_c$, defined as the direction of least-explored space; and neighbor density ${\rho }_{\mathrm{neigh}}$ with three bins. The reported improvements are therefore attributable solely to these three added observation features. The RL-only baseline is intentionally matched to the Hybrid in architecture, reward function, and training procedure, differing only in observation features (Table 4), so that any performance difference is attributable to the three added gradient features rather than to architectural differences.

The state space increases from 9 to 18 dimensions; however, due to discretization, the resulting Q-table remains computationally tractable within the evaluated environment size.

4. Results and Discussion

4.1. Training Convergence

Figure 5 shows the training reward smoothed with a 100-episode uniform moving average over 5000 episodes. The Hybrid model shows faster and more stable reward improvement than the RL-only baseline, indicating that directional pheromone-gradient observations provide useful structure for learning coordinated search behavior. This supports the role of pheromone-guided observations in reducing sample complexity compared with learning from local state features alone.

This Hybrid approach not only achieves better performance at the end but also converges much faster. Looking at the learning curve, we see that it reaches stable behavior sooner compared to the RL-only baseline, which implies that the knowledge of directional gradients helps the agent learn more efficiently.

Both models are trained under an identical, fixed budget of 5000 episodes to keep the comparison matched (Section 3.7); within this budget, RL-only has not yet reached a stable asymptote, while the Hybrid model’s reward curve is already close to flat. Characterizing RL-only’s eventual asymptotic performance would require training well beyond this budget and is left for future work.

4.2. Coverage and Victim Detection

Figure 6 depicts area coverage over time for all five approaches in 30 trials with $\pm 1\sigma$ confidence intervals. Hybrid agent achieves 98.9% area coverage and 93.3% victim detection, improving coverage from 81.8% to 98.9% and victim detection from 71.7% to 93.3% over the pure RL approach. Welch’s t-test shows that these gains are significant for both coverage ($t=18.88$, $df=168$, $p<0.001$, Cohen’s $d=2.18$) and victim detection ($t=16.49$, $df=218$, $p<0.001$, Cohen’s $d=1.90$) (

Table 5. Statistical significance: Hybrid vs. RL-only (Welch’s t-test, pooled $n=150$ each).

Metric	t	df	p	Cohen’s d
Coverage	18.88	168	<0.001	2.18
Victim detection	16.49	218	<0.001	1.90

). In addition, the coverage variance reduces from $\pm 10.8\%$ with RL-only approach to $\pm 2.7\%$ with the Hybrid one indicating increased stability in decentralized execution due to observations of pheromone gradients, see Table 5 for more details.

Random walk’s coverage curve has not fully plateaued within the 500-step evaluation window. With more time, unstructured search would likely continue moving toward full coverage. The 500-step limit reflects the drone battery constraint (Table 3) rather than a training or compute budget, so the comparison is intentionally about performance under this shared, realistic time constraint rather than eventual coverage given unlimited time.

For each of the 150 evaluation configurations (5 seeds × 30 runs), every method is evaluated on an identical map, presenting the same obstacle layout, hazard zones, victim locations, and drone starting positions, ensuring a like-for-like comparison across methods.

4.3. Comparative Analysis

Benchmark results are shown in Figure 7. The Hybrid algorithm provides 98.9% coverage and 93.3% victim detection, greatly outperforming the RL-based-only algorithm, providing 81.8% coverage and 71.7% victim detection. This corresponds to an increase from 81.8% to 98.9% in coverage and from 71.7% to 93.3% in victim detection.

The stigmergy-only approach is also quite good, achieving 86.5% coverage and 77.7% victim detection. Its fixed exhaustive exploration rules help it achieve better coverage than both random walk and RL-only, although it still remains below the Hybrid model overall. However, unlike the learned Hybrid model, it cannot adapt its behavior at runtime and therefore cannot flexibly prioritize changing mission objectives, energy constraints, or hazard conditions. The Hybrid model’s advantage over the non-learned baselines follows from combining structured environmental memory with an adaptable policy: Frontier search and Stigmergy-only can exploit the pheromone field but cannot adjust their fixed scoring rules to changing conditions, while random walk has no structure at all. Its advantage over RL-only follows from the ablation study (

Table 6. Ablation study: mean ± Std across five Seeds (30 Runs Each, Pooled $n=150$). RMSE vs. Hybrid is the root-mean-square deviation of each variant from the full Hybrid model, computed jointly across the coverage and victim-detection columns.

Variant	Coverage (%)	Victims (%)	RMSE vs. Hybrid (%)
Hybrid Full (ours)	$98.9\pm 2.7$	$93.3\pm 7.1$	—
No directional gradients	$82.3\pm 13.9$	$71.7\pm 16.9$	19.26
No neighbor density	$99.2\pm 2.6$	$94.3\pm 7.1$	0.74

): providing directional gradients removes the ambiguity of scalar-only observations, letting each drone move toward likely victim locations or unexplored regions directly rather than discovering them through undirected trial and error, which both raises mean performance and substantially reduces its variance across training seeds (Section 4.4).

4.4. Ablation Study

To quantify the contribution of each proposed feature, we train and evaluate three variants: the full Hybrid model, an ablated model without directional gradients (${\nabla }_v,{\nabla }_c$ removed), and an ablated model without neighbor density (${\rho }_{\mathrm{neigh}}$ removed). Results are shown in Figure 8 and Table 6.

Table 6 additionally reports the root-mean-square deviation of each variant from the full Hybrid model, computed jointly across coverage and victim-detection percentages, as a single combined measure of each ablation’s overall effect size.

Removing directional gradients causes a significant performance collapse, reducing coverage by 16.7 percentage points ($98.9\%\to 82.3\%$) and victim detection by 21.6 percentage points ($93.3\%\to 71.7\%$). From a behavioral standpoint, eliminating ${\nabla }_v$ and ${\nabla }_c$ forces the policy to regress to local scalar sampling. When situated in a cell with non-zero victim or coverage pheromones, an agent without directional gradients cannot deduce the spatial vector toward the source peak or unexplored regions. This ambiguity also explains the variant’s far larger run-to-run variance: pooled across all 150 evaluation runs, coverage varies by $\pm 13.9\%$ for this variant versus $\pm 2.7\%$ for the full Hybrid model (Table 6); without a consistent directional signal to follow, outcomes depend more on which unstructured moves happen to succeed in a given run, rather than on a repeatable strategy.

Conversely, removing neighbor density (${\rho }_{\mathrm{neigh}}$) produces only a small shift in the reported means (+0.3% coverage, +1.0% victim detection) that is not statistically significant (coverage: $t=0.75$, $p=0.45$; victims: $t=1.25$, $p=0.21$; both Cohen’s $d<0.15$), so it should not be read as evidence that neighbor density helps or hurts performance in either direction. Its role instead lies outside these two metrics: it discourages redundant clustering via the crowding term in the reward (Equation (3)), and unlike pheromone-map sharing, it remains available even under total communication loss since it is derived from physical drone positions rather than message exchange. The directional gradient components remain the dominant drivers of the performance gains reported above.

4.5. Robustness Under Communication Loss

Training, the main comparison (

Table 7. Evaluation results: mean ± Std across five seeds (30 runs each, pooled $n=150$; $40\times 40$ grid, 10 drones, 20 victims, 500 steps).

Method	Coverage (%)	Victims (%)	1st Victim (Steps)
Random walk	$80.5\pm 8.2$	$69.9\pm 13.1$	$10.0\pm 12.3$
Frontier search	$85.8\pm 12.9$	$77.8\pm 17.4$	$2.7\pm 0.2$
Stigmergy only	$86.5\pm 11.7$	$77.7\pm 11.2$	$2.8\pm 0.3$
RL-only	$81.8\pm 10.8$	$71.7\pm 14.3$	$5.8\pm 1.4$
Hybrid (ours)	$\mathbf{98.9}\pm \mathbf{2.7}$	$\mathbf{93.3}\pm \mathbf{7.1}$	$\mathbf{3.2}\pm \mathbf{0.0}$

), and the ablation study (Table 6) all use a single pheromone field shared instantly by every drone, without restriction by communication radius $r_c$; this radius governs only the neighbor-density feature and the communication reward term in those experiments. The experiments in this subsection instead switch to per-drone pheromone maps, evaluating policies that were trained under the shared-field assumption against a degraded, communication-limited deployment condition. Two properties of this design explain why coverage remains high even at 100% link failure. First, because the policy is trained under a fully shared field, it already encodes broadly useful exploration habits rather than habits that depend on continuously receiving updates from other drones. Second, even when merging stops entirely, each drone continues accumulating its own local pheromone map from its own sensing, and neighbor density remains available regardless of dropout level since it is computed from physical drone positions rather than message exchange (Section 3.4). Together, these mean a drone’s own exploration degrades gracefully rather than collapsing when pheromone updates from other drones stop arriving.

To evaluate robustness under realistic degraded-communication conditions, each drone maintains its own local pheromone map during evaluation, merging (element-wise maximum) with neighbors whose link is currently up. Both models were evaluated across a sweep of communication-link failure probabilities: 0%, 25%, 50%, 75%, and 100%. At each level, each active pairwise link is severed independently with the given probability, preventing map merging between the affected drones and forcing each to rely solely on its own locally accumulated pheromone evidence. Note that ${\rho }_{\mathrm{neigh}}$ is computed from physical drone positions rather than message exchange and therefore remains available at all dropout levels; only pheromone-map sharing is gated by the dropout probability. Results are shown in Figure 9 and

Table 8. Communication-dropout sweep: mean coverage and victims, 150 runs.

Dropout (%)	Hybrid Cov. (%)	Hybrid Vic. (%)	RL Cov. (%)	RL Vic. (%)
0	99.3	93.7	84.4	73.7
25	99.0	93.3	84.6	73.3
50	99.3	93.6	85.0	73.9
75	98.9	93.2	84.9	74.6
100	99.1	91.8	87.5	76.9

.

The Hybrid model demonstrates strong robustness across the full sweep: coverage remains at 99.1% even at 100% link failure, a degradation of only 0.2% from the fully connected baseline (99.3%). The RL-only agent similarly maintains stable performance, with both models relying on locally available pheromone information and learned policies that do not require active inter-agent coordination to function effectively. The Hybrid model keeps a consistent $\approx 14$–15% coverage lead over RL-only at every dropout level. This suggests that pheromone-based stigmergy gives a structural coordination advantage and that advantage remains even when communication is fully lost. While this experiment proves the policy can handle sudden signal loss, real-world impairments like transmission delays and data corruption are detailed in Section 4.6.

4.6. Parameter Sensitivity Analysis

To evaluate whether the performance of the proposed Hybrid framework depends on sensitive hyperparameter tuning, we conducted a parameter sensitivity analysis across three core parameters: the virtual pheromone evaporation rate ($\rho$), the diffusion coefficient (D), and the Q-learning rate (LR). Each parameter was varied across scaling factors of $0.5\times , 1.0\times , 1.5\times ,$ and $2.0\times$ relative to baseline configuration (${\rho }_v=0.02$, $D_c=0.08$, $\mathrm{LR}=0.18$).

As shown in

Table 9. Parameter sensitivity analysis across evaporation ($\rho$), diffusion (D), and learning rate (LR). Metrics represent mean performance across evaluation seeds.

Parameter	Scale/Value	Coverage (%)	Victims (%)	1st Victim (Steps)
Evaporation ($\rho$)	$0.5\times$	97.9	90.2	2.9
	$1.0\times$ (base)	99.8	94.0	2.9
	$1.5\times$	99.8	93.8	3.3
	$2.0\times$	99.3	94.3	3.1
Diffusion (D)	$0.5\times$	98.5	93.0	2.8
	$1.0\times$ (base)	99.8	94.0	2.9
	$1.5\times$	99.0	93.3	2.7
	$2.0\times$	99.2	93.7	2.8
Learning Rate (LR)	$0.5\times$ (0.09)	99.8	94.0	2.9
	$1.0\times$ (0.18)	99.8	94.0	2.9
	$1.5\times$ (0.27)	99.8	94.0	2.9
	$2.0\times$ (0.36)	99.8	94.0	2.9

, the framework maintained high performance across all parameter sweeps, keeping area coverage above $97.9\%$ and victim detection above $90.2\%$. Cutting the evaporation rate in half ($0.5\times$) caused a minor drop in victim detection ($90.2\%$) because older pheromone traces took longer to fade, occasionally drawing drones back to already-searched areas. However, varying the diffusion rate and learning step size across a $4\times$ range had almost no impact on overall performance. These results confirm that the framework’s success comes from the spatial guidance provided by directional gradient observations, rather than fine-tuned hyperparameters.

4.7. Limitations

Seven limitations are acknowledged. First, although the parameter sensitivity analysis in Section 4.6 demonstrates that the proposed framework remains robust across a broad range of evaporation, diffusion, and learning-rate settings, the current implementation still relies on manually selected parameter values. Adaptive online tuning via meta-learning may further reduce manual calibration requirements and enable automatic adjustment of pheromone dynamics when operating across significantly different environments, mission objectives, or swarm sizes. Second, the sim-to-real gap, GPS denial, sensor noise, aerodynamic disturbances, and irregular 3D terrain must be addressed before physical deployment; AirSim or Webots validation is the recommended next step. Third, tabular Q-learning may underperform deep function approximators (QMIX, MAPPO) on larger or more complex environments, and the absolute performance ceiling of both the Hybrid and RL-only models is therefore lower than what deep MARL would achieve on this task. The RL-only baseline is intentionally matched in architecture to the Hybrid to isolate the contribution of the directional gradient observation features; comparison against deep MARL baselines (QMIX, MAPPO, independent DQN) is deferred to future work. Fourth, safety-critical outdoor UAV operations require geofencing, fail-safe landing protocols, and regulatory compliance beyond the scope of this work. Fifth, the communication-dropout evaluation considers only independent random link failures. It does not model realistic physical channel impairments such as transmission delays, packet loss, partial map corruption, or dynamic signal attenuation, which remain critical directions for future network-level modeling.

Sixth, as discussed in Section 3.2 and Section 3.4, victim and obstacle detection rely on deterministic, distance-based rules operating on ground-truth grid occupancy, rather than on simulated sensor data (RGB, thermal, LiDAR, or IMU) with realistic noise, false positives, or false negatives. Incorporating such a sensor model is necessary before assessing real-world deployability. Seventh, as noted in Section 3.1, drone kinematics are not physically modeled: each timestep corresponds to one discrete grid move regardless of real-world speed, acceleration, or turning constraints, and the 500-step budget represents a fixed mission duration rather than a physically grounded battery or endurance model. Incorporating realistic drone dynamics and flight-time constraints is left for future work.

Additionally, the proposed representation is evaluated within a discretized state space using tabular reinforcement learning. While this setup allows controlled evaluation of the representation itself, further validation with deep function approximators and continuous observation spaces is required to assess scalability in more complex environments.

5. Conclusions

The proposed work investigated whether directional pheromone-gradient observations can improve decentralized coordination in multi-UAV search operations under partial observability and limited communication. To address this question, we proposed a Hybrid stigmergy–MARL framework that combines virtual pheromone fields with reinforcement learning under the CTDE paradigm.

Experimental results showed that the proposed approach achieved 98.9% area coverage and 93.3% victim detection, outperforming the RL-only baseline by increasing coverage from 81.8% to 98.9% and victim detection from 71.7% to 93.3%. Ablation analysis further demonstrated that directional gradient features are the primary contributor to these gains, while communication-loss experiments confirmed robust performance even under complete communication failure.

The main contribution of this research is the introduction of directional pheromone-gradient observations as a structured form of environmental memory that guides decentralized decision-making without requiring centralized control or continuous communication. The results indicate that combining stigmergic information with learned policies can improve both coordination effectiveness and operational robustness in swarm-based search missions.

Future work will investigate integrating the proposed observation design with advanced MARL frameworks such as QMIX and MAPPO. These methods can model inter-agent dependencies more effectively than tabular Q-learning and may improve coordination quality, scalability, and learning efficiency in larger swarm deployments. We also plan to explore neural pheromone encoders that can learn continuous representations of pheromone fields directly from data, potentially reducing discretization limitations and enabling richer spatial reasoning in large-scale environments. In addition, we plan to extend the framework beyond search and detection by incorporating autonomous medical-package delivery and victim-assistance capabilities, enabling a more complete end-to-end search-and-rescue system.