RLFS-OR: Reinforcement Learning-Based Forwarder Selection for Opportunistic Routing in Wireless Sensor Networks

Abstract

This paper introduces RLFS-OR, a reinforcement learning-based opportunistic routing protocol designed for energy-constrained and duty-cycled wireless sensor networks (WSNs). Unlike traditional opportunistic routing, which either relies on static metrics or requires nodes to remain continuously active, RLFS-OR integrates a Deep Q-Network (DQN) to dynamically select the most energy-efficient forwarder based on residual energy, hop distance, wake-up timing, and link quality. A realistic Castalia-derived radio model is incorporated, accounting for transmission, reception, idle listening, and path loss-dependent energy consumption. Through coordinated learning and asynchronous duty-cycle integration, RLFS-OR minimizes overhearing and unnecessary wake-ups. Simulation results demonstrate that RLFS-OR significantly outperforms two established protocols—ORW and FCM-OR—achieving 10–30% lower energy consumption and 10–45% longer network lifetime under diverse network densities and traffic loads. RLFS-OR also provides smoother node-death dynamics and optimal performance at low duty cycles. The findings confirm RLFS-OR as an efficient and scalable solution for long-lived WSN deployments.

1. Introduction

Wireless Sensor Networks (WSNs) are composed of spatially distributed autonomous sensor nodes that collaboratively monitor environmental or physical phenomena such as temperature, humidity, and motion. These nodes are typically constrained in terms of power, processing, and memory resources. Among these, energy consumption is the most critical concern, as battery replacement or recharging is often infeasible, particularly in remote or hazardous environments. Consequently, optimizing energy usage has been a key design objective for communication protocols in WSNs, especially in the routing layer [1].

To reduce idle listening and redundant transmissions, Asynchronous Duty-Cycled WSNs (ADC-WSNs) have been proposed [2,3]. In ADC-WSNs, each node alternates independently between active and sleep states, significantly lowering idle energy drain. However, this asynchronous behavior creates challenges in routing whereby a sender node may have to wait for its intended receiver to become active. This delay leads to energy waste and higher end-to-end latency [4].

To address this, Opportunistic Routing (OR) strategies such as ExOR [1] and ORW [4] allow sender nodes to broadcast packets to a set of candidate forwarders. Any node that wakes up and successfully receives the packet can forward it, eliminating strict reliance on fixed next-hop paths. This increases delivery probability and reduces delay, making OR well-suited for ADC-WSNs. However, it also introduces drawbacks, including redundant forwarding, collisions, and the challenge of selecting the optimal forwarder from a dynamic candidate set [5].

Recent efforts have attempted to improve opportunistic routing by introducing forwarder selection mechanisms based on metrics such as the Expected Duty Cycle (EDC) [4], link quality (e.g., RSSI, SNR), and residual energy [6]. However, these heuristic or static methods often fall short in dynamically changing environments, leading to sub-optimal energy usage and network performance.

To overcome these limitations, researchers have begun applying Reinforcement Learning (RL) in WSN routing [7,8,9,10], enabling nodes to learn optimal forwarding policies via interaction with the network environment. A widely used RL technique in WSNs is Q-learning, which is valued for its simplicity and model-free operation. However, Q-learning faces scalability issues in large state–action spaces, which is common in WSNs where nodes must consider multiple neighbors, energy levels, and link states simultaneously.

To enhance scalability and learning performance in such complex and dynamic environments, we adopt Deep Q-Network (DQN), a deep reinforcement learning algorithm that uses a neural network to approximate the Q-function [11]. DQN alleviates the curse of dimensionality and enables better generalization across similar states, which is crucial in WSNs where each node may experience a wide range of operating conditions. Unlike traditional Q-learning [10], which requires a Q-table for every possible state–action pair (which becomes infeasible as the network scales), DQN uses deep learning to approximate the optimal Q-values from features like link quality, residual energy, EDC, and wake-up history.

DQN has been successfully applied in robotics, networking, and autonomous systems and is particularly suited for partially observable, non-stationary environments—as is the case with ADC-WSNs. It enables each node to learn a compact yet expressive policy for forwarder selection without relying on handcrafted rules or exhaustive state enumeration.

1.1. Contributions

To overcome the challenges of opportunistic routing with ADC-WSNs, in this work, we propose RLFS-OR, a Deep Q-Network-integrated reinforcement learning-based opportunistic routing protocol that offers the following key contributions:

• A novel reinforcement learning-based opportunistic routing protocol (RLFS-OR) is proposed for asynchronous duty-cycled wireless sensor networks. The protocol leverages a Deep Q-Network (DQN) to select energy-efficient forwarders based on local observations, including residual energy, hop distance, wake-up timing, and collision probability.
• A realistic radio energy model based on the widely adopted CC2420 IEEE 802.15.4 transceiver is incorporated into the routing process. The CC2420 transceiver has been extensively used in wireless sensor network (WSN) platforms, and its transmit, receive, and idle power consumption characteristics are well documented in the official technical datasheet and prior energy-aware WSN studies [12,13]. The radio energy parameters employed in this work are derived from the CC2420 datasheet [12] and are consistent with established CC2420-based energy modeling practices [13]. This integration enables accurate modeling of transmission, reception, idle listening, and path loss-dependent power costs, thereby reflecting realistic sensor-node behavior.
• A duty cycle-aware forwarding mechanism is introduced to reduce unnecessary wake-ups and overhearing, enabling RLFS-OR to operate efficiently under low-duty-cycle conditions, where most traditional protocols experience significant performance degradation.
• A multi-objective reward function combining remaining energy, hop count, EDC and collision probability is introduced to guide DQN learning toward globally efficient routing.
• A comprehensive comparative evaluation is performed against two state-of-the-art schemes—ORW (with duty-cycled sensors) [4] and FCM-OR (cluster-based opportunistic routing with duty cycled sensors) [14]. The experiments cover varying node densities, traffic loads, transmission ranges, and duty cycles.
• Significant performance improvements are demonstrated, including 10–30% lower energy consumption, 10–45% longer network lifetime, smoother node-death dynamics, and superior scalability across diverse deployment conditions.
• A full parameter sensitivity and duty-cycle optimization study is conducted, showing that RLFS-OR achieves its highest efficiency in the 0.1–0.3 duty-cycle range, confirming its suitability for long-lived, low-power WSN applications.

Deep reinforcement learning has recently shown strong potential in domains where routing decisions depend on dynamic, high-dimensional environmental states. The Deep Q-Network (DQN) algorithm, first introduced by Mnih et al. [11] for stable Q-learning with neural function approximation, demonstrated that deep learning can outperform conventional decision-making schemes in complex stochastic environments. Although reinforcement learning has been applied in wireless sensor networks, existing works primarily rely on tabular Q-learning or lightweight heuristics, which are unable to generalize efficiently in large-scale, duty-cycled, or highly dynamic network conditions [10,15]. The use of DQN in WSNs remains extremely limited, and—critically—no prior study has integrated DQN with opportunistic routing under asynchronous duty-cycle operation. Therefore, the proposed RLFS-OR protocol represents the first attempt to employ DQN-based forwarder selection in a duty-cycled opportunistic routing setting. By leveraging DQN’s ability to extract meaningful patterns from local state features, RLFS-OR learns an adaptive, energy-aware routing policy that significantly reduces redundant transmissions, balances energy usage, and enhances the overall lifetime of WSN deployments.

1.2. Paper Organization

The remainder of this paper is structured as follows: Section 2 provides an overview of related work. Section 3 outlines the preliminaries and system model underlying the proposed method. Section 4 details the RLFS-OR protocol and its methodology. Section 5 presents simulation results and performance evaluations. Section 6 and Section 7 discuss key findings and future research directions, respectively. Finally, Section 8 concludes the paper with insights and contributions.

2. Related Works

Opportunistic routing (OR) has emerged as an effective strategy for improving the reliability of data delivery in wireless sensor networks (WSNs), particularly in duty-cycled environments characterized by intermittent connectivity and asynchronous wake-up schedules. Traditional routing protocols that rely on predetermined routes—such as tree-based forwarding [16] or fixed-path collection mechanisms—often experience significant degradation when nodes frequently sleep, wake unpredictably, or suffer from link instability. As nodes may not be awake when a scheduled transmission occurs, these approaches incur high latency, packet loss, and unnecessary retransmissions, ultimately leading to excessive energy consumption [5].

Existing energy-efficient communication strategies in WSNs can generally be classified along two orthogonal dimensions: (i) clustering-based versus routing-based approaches and (ii) synchronous versus asynchronous duty-cycling mechanisms. Clustering-based methods organize nodes into hierarchical groups to reduce redundant transmissions and balance energy consumption [17], whereas routing-based schemes focus on selecting optimal next-hop forwarders without forming explicit cluster structures [18]. In parallel, synchronous duty-cycled protocols rely on coordinated wake-up schedules across nodes [19], while asynchronous approaches allow for independent sleep–wake patterns to minimize coordination overhead [5]. Opportunistic routing techniques, including reinforcement learning-based solutions [18], primarily belong to the routing-based and asynchronous category, where forwarder selection must dynamically adapt to uncertain wake-up behavior and fluctuating link conditions.

OR addresses these limitations by enabling a sender to broadcast a packet to a set of candidate forwarders rather than a single predetermined next hop [4]. Any awake and suitable candidate may opportunistically forward the packet, thereby improving reliability and reducing retransmission overhead. Furthermore, OR naturally adapts to real-time variations in link quality, residual energy distribution, and node wake-up behavior—critical factors that influence routing efficiency in duty-cycled sensor networks [4,6,20,21,22,23,24,25,26].

One of the seminal opportunistic routing protocols is ORW (“Opportunistic Routing in Wireless Sensor Networks”) [4], which introduced the Expected Duty Cycle (EDC) metric to evaluate both forwarding cost and the wake-up frequency of neighboring nodes. By integrating duty-cycle awareness into the metric, ORW improved packet forwarding efficiency compared to static metrics used by CTP-WUR [27] and B-MAC [28]. However, ORW and similar protocols still depend on static, heuristic evaluations and do not fully capture the dynamic nature of asynchronous duty-cycled networks—where link quality, congestion, residual energy, and forwarder wake-up patterns fluctuate significantly over time [5].

To overcome these limitations, recent studies have explored reinforcement learning (RL) as a mechanism for adaptive opportunistic routing. In [29], the authors used Q-learning to estimate the forwarding value of neighbors based on long-term rewards, such as successful packet delivery or energy consumption reduction. EMOR (Energy Efficient Mixture Opportunistic Routing) [30] further improved energy performance by combining mixture models with RL to balance the delivery ratio and energy usage. RL-based opportunistic protocols such as EBR-RL [10], EER-RL [15], and RCAR [31] introduced adaptive mechanisms that account for network conditions such as congestion, local cluster density, and heterogeneous link behavior. These studies demonstrated that RL can significantly enhance routing performance by continuously learning optimal forwarding behaviors from feedback signals rather than relying on rigid metrics.

More advanced RL-based protocols such as DRL-OR [32] applied Deep Q-Networks (DQN) to opportunistic routing, leveraging neural approximators to generalize forwarding strategies across large state spaces. DeepHop [33] extended this concept by applying deep reinforcement learning to dynamically adapt hop selection in asynchronous duty-cycled environments. These protocols highlight the power of deep RL techniques to model complex interactions involving duty-cycle variability, residual energy levels, and dynamic neighbor sets.

However, while DRL-OR [32] and DeepHop [33] demonstrate the effectiveness of deep reinforcement learning for routing decisions, their design objectives differ from those of RLFS-OR. DRL-OR focuses on multi-type service requirements and large state–action spaces in general network environments, whereas DeepHop emphasizes distributed deep learning for hop-by-hop routing in edge or high-performance network settings. In contrast, RLFS-OR is specifically designed for asynchronous duty-cycled wireless sensor networks (WSNs) operating under IEEE 802.15.4 constraints.

Table 1. Summary of previous related works in the area of opportunistic routing in WSNs.

Protocol	Routing Metric	Addressed Issues	Advantages	Drawbacks
ORW [4]	EDC (Expected Duty Cycle)	Asynchronous wake up, idle listening	Energy aware; simple metric	Static selection; no learning component
CTP-WUR [27]	Link quality + wake-up rate	Wake-up radio scheduling, low-power forwarding	High delivery ratio in scheduled networks	Hardware specific; limited to synchronized WSNs
EBR-RL [10]	RL-based residual energy, hop distance	Energy balancing, adaptive routing	Dynamic adaptation; improved network lifetime	Focus on clustering; duty-cycle integration lacking
EER-RL [15]	RL reward: delivery success + energy cost	Energy-efficient routing with RL	Good energy savings; adaptable	RL only in the routing layer; sleep/wake costs omitted
RCAR [31]	RL-based clustering + routing	Cluster formation + opportunistic forwarding	Reduced control overhead; hybrid RL	Complexity; slower convergence in large-scale networks
EMOR [30]	Reward-based mixture (Energy + Link)	Sparse networks, energy optimization	Highly energy efficient; hybrid strategy	Designed for lunar networks; high complexity
DRL-OR [32]	DQN-based forwarder Q-value	Large state–action spaces, dynamic neighbor sets	Scalable RL; better generalization	Higher computational demand; training overhead
DeepHop [33]	Deep RL hop selection + duty-cycle adaptation	Duty-cycled WSNs with fluctuating connectivity	Integrates duty-cycle and routing decisions	Deep model complexity; memory/processing overhead
FCM-OR [14]	FR + EDC + FS + residual energy + link Quality	Sender wait time, uneven local density, multiple forwarders, redundant transmission	Balanced load; improved lifetime	Extra clustering overhead; no RL adaptation
RLFS-OR (Proposed)	Hop count + RSSI-aware EDC + DQN + residual energy + collision prob.	Realistic energy model, MAC duty cycle, adaptive forwarder selection	Integrated power model; dynamic selection, longer lifetime	Parameter tuning required; RL training overhead

presents a concise summary of previous related works within the scope of our protocol research area and provides a brief comparison with the proposed protocol. Unlike these approaches, RLFS-OR explicitly integrates MAC-layer duty-cycle awareness through the Expected Duty Cycle (EDC) metric and incorporates a realistic CC2420-based radio energy model that accounts for transmission, reception, idle listening, and collision-related costs. Furthermore, RLFS-OR adopts a lightweight DQN implementation in which only the output-layer parameters are updated during training, reducing computational complexity and making the approach more suitable for energy-constrained WSN environments. These distinctions highlight that RLFS-OR is not merely a direct application of a DQN to opportunistic routing but a cross-layer, energy-aware adaptation tailored to low-power duty-cycled sensor networks.

Despite the advances in this research area, most existing RL-based OR protocols suffer from one or more of the following limitations:

• MAC-layer awareness is missing: RL decisions ignore real duty-cycle behavior; wake-up schedules; and MAC-layer energy costs such as idle listening, sleep-to-wake transitions, and collision overheads.
• Energy modeling is oversimplified: Many works do not incorporate realistic radio energy models, resulting in inaccurate evaluation of RL-based decisions.
• Cluster-based approaches introduce overhead: RL + clustering protocols (e.g., RCAR [31]) increase complexity and require global density information, making them unsuitable for lightweight WSN deployments.
• Limited adaptation to dynamic traffic: Most RL-based approaches do not jointly adapt to fluctuating traffic loads, congestion, and varying energy profiles.

FCM-OR [14] addresses redundant transmissions by integrating soft fuzzy clustering with opportunistic routing. However, while effective in mitigating multiple-receiver problems, FCM-OR does not incorporate RL, lacks MAC-layer duty-cycle adaptation, and relies heavily on local cluster formation, which can be computationally demanding in large-scale networks.

To address these gaps, our proposed RLFS-OR leverages DQN because it offers several advantages uniquely suited for asynchronous, duty-cycled sensor networks:

• High-dimensional state modeling: RLFS-OR considers multiple MAC-aware features—residual energy, hop count, expected duty cycle, and collision probability. DQN handles this multidimensional state representation better than tabular Q-learning, which suffers from state-space explosion.
• Continuous adaptation to volatile network conditions: Duty-cycled WSNs have rapidly changing neighbor sets and stochastic wake-up behavior. DQN’s neural approximator generalizes learned patterns, allowing for fast adaptation, even when network topology changes frequently.
• Stable learning via replay memory and target networks: These mechanisms prevent divergence and stabilize learning in non-stationary environments—critical for WSNs where nodes die, sleep, or change roles unpredictably.
• Elimination of handcrafted routing metrics: Traditional OR protocols depend on manually designed metrics like EDC [4], ETX [1,34,35,36,37], hop count [38,39,40,41,42], geographical distance [43,44,45,46,47], expected transmission time (ETT) [48,49,50] or coding gain [51] bias. DQN automatically learns an optimal forwarding strategy that balances energy efficiency and delivery progress without such manual tuning.
• Scalable to dense networks: DQN avoids per-state storage overhead and scales efficiently to larger node populations, where tabular RL would become infeasible.

Thus, DQN is selected as the core learning mechanism because it allows RLFS-OR to operate effectively in asynchronous, energy-constrained, and dynamically changing wireless sensor networks, where classical RL or heuristic protocols fall short.

To the best of our knowledge, RLFS-OR is the first opportunistic routing protocol to tightly couple DQN with a duty-cycled MAC environment and a complete radio-energy model, enabling far more accurate learning and a longer network lifetime.

3. System Model

This section describes the network assumptions, MAC behavior, energy model, and routing states used in the proposed RLFS-OR protocol.

3.1. Network Topology and Deployment Assumptions

We consider a wireless sensor network (WSN) composed of N static and homogeneous sensor nodes, each equipped with a CC2420 radio transceiver and a unique node identifier. Nodes are randomly deployed in a two-dimensional sensing field of size L × L square meters, and a single sink node is positioned at one corner of the field. The sink is assumed to have unlimited energy and operates with a duty cycle of 1.0 (always awake), whereas all other nodes follow independent, asynchronous, duty-cycled schedules.

All nodes have a fixed transmission range of 50 m, resulting in a multi-hop topology whenever the Euclidean distance between a node and the sink exceeds this range. Since nodes remain static after deployment, neighbor discovery is performed only once; however, routing metrics such as the Expected Duty Cycle (EDC) and residual energy are periodically recomputed to reflect dynamic channel and energy conditions. A visual representation of the network topology has been shown in Figure 1. The green dotted lines between the sensor nodes indicate the connection between each other, and the gray dotted lines represent the direct connection between the one-hop distance nodes and the sink.

Each node generates data events when active and forwards sensed packets using opportunistic routing. Nodes are assumed to know: (i) their own coordinates (via initial deployment), (ii) the sink location, and (iii) the total network size (N). This information does not require global synchronization or continuous exchange, minimizing control overhead.

Node failure in the network primarily occurs due to energy depletion, which is explicitly modeled in our simulations. When a node’s residual energy reaches zero, it is considered inactive and is excluded from subsequent forwarding operations. Since RLFS-OR employs a fully distributed opportunistic routing mechanism, failed nodes are automatically removed from candidate forwarder sets during local neighbor updates (Every time a node overhears any packet in the neighborhood, it will update the necessary information related to the relative neighbor). Each node periodically recomputes routing metrics such as the residual energy and Expected Duty Cycle (EDC), ensuring that only active neighbors participate in forwarding decisions. Consequently, the protocol does not require any centralized recovery or reconfiguration mechanism. The learning process naturally adapts to topology changes by updating Q-values based on the current set of available neighbors, thereby maintaining routing continuity, even under progressive node failures.

3.2. MAC-Layer Behavior and Duty-Cycle Model

Nodes employ the BoX-MAC2 protocol [3], a low-power, asynchronous MAC designed for preamble sampling. Each node alternates between active and sleep states within a fixed 610 ms frame.

A node’s duty cycle is defined as

$$DC={\displaystyle \frac{T_{active}}{T_{frame}}},$$

(1)

$$T_{frame}=T_{active}+T_{sleep}.$$

(2)

Thus, for a node with $DC=0.5$, the active period is $T_{active}=305$ ms, while the remaining $T_{sleep}=305$ ms. Nodes operate independently, causing sender–receiver desynchronization, which is one of the primary challenges of ADC-WSNs.

In the MAC layer, nodes broadcast packets to all neighbors within range, enabling multiple forwarders to opportunistically receive the transmission. Only the forwarder that satisfies routing conditions (e.g., highest Q-value, and awake state) proceeds to forward the packet.

Collisions occur when multiple forwarders attempt to transmit simultaneously. The collision probability ($P_c$) is used by RL to discourage high-contention paths.

3.3. Reinforcement Learning-Based Forwarder Selection

Unlike clustering-based approaches (e.g., FCM-OR), RLFS-OR integrates Deep Q-Network (DQN)-based decision making directly into the routing layer. Each node runs a lightweight RL agent that evaluates candidate neighbors according to a local state vector:

$$s=[E^{res},H{C}^{sink},EDC,P^c],$$

(3)

where $E^{res}$ represents residual energy, $H{C}^{sink}$ indicates the hop distance to the sink, $EDC$ is the expected duty cycle (neighbor wake likelihood), and $P^c$ is the estimated collision probability in the neighborhood.

The hop-distance-to-sink value ($H{C}^{sink}$) is obtained through a lightweight initialization phase prior to data transmission. During this phase, the sink broadcasts a control message that propagates throughout the network. Nodes located within one-hop distance from the sink assign their hop count as 1 and rebroadcast the control packet with the updated hop value. As the control message propagates further, each receiving node increments the hop count by one and forwards the packet, thereby establishing a minimum-hop gradient toward the sink.

This initialization procedure is executed only once at the time of deployment and does not constitute a separate routing protocol. During network operation, $H{C}^{sink}$ may be periodically refreshed to accommodate topology changes resulting from node failures or energy depletion. Since nodes are static after deployment, the associated overhead is minimal and does not require global synchronization.

Using this state, the DQN outputs a Q-value for each neighbor. The neighbor with the highest Q-value (or $\epsilon$-$greedy$ exploration) is selected as the next-hop forwarder.

The number of neighbors does not increase the dimensionality of the DQN output layer. Instead of assigning one output neuron per neighbor, RLFS-OR evaluates candidate neighbors individually using the same fixed-size state representation and computes their Q-values sequentially. The forwarder with the maximum Q-value is selected. Therefore, even when the number of neighbors increases (e.g., 30 or 50), the neural network structure remains unchanged, and computational complexity increases linearly with the size of the candidate set rather than exponentially. This design ensures scalability in dense deployments.

The reinforcement signal is received after forwarding, and the reward is computed as follows:

$$r=+{\alpha }_1{E}^{\mathrm{res}}-{\alpha }_{2}H{C}^{\mathrm{sink}}-{\alpha }_3\mathrm{EDC}-{\alpha }_4{P}^c,$$

(4)

The design of the reward encourages energy balancing, shorter forwarding paths, low-duty-cycle neighbors, and avoidance of congested or collision-prone areas.

3.4. RSSI-Based Link Quality Model

The Received Signal Strength Indicator (RSSI) for a link between nodes i and j is modeled using the log-distance path-loss model with shadowing:

$$RSS{I}_{ij}=P_t-PL\left(d\right),$$

(5)

$$PL\left(d\right)=PL\left(d_0\right)+10\eta {log}_{10}\left({\displaystyle \frac{d}{d_0}}\right)+X_{\sigma },$$

(6)

where $P_t$ is the transmission power (dBm), d is the distance between nodes, $d_0$ represents a reference distance, $\eta$ is the path-loss exponent, and $X_{\sigma }$∼$\mathcal{N}(0,{\sigma }^2)$ indicates a shadowing term.

Reception probability increases with higher RSSI values and decreases below a threshold:

$$P_{\mathrm{recv}}=\left\{\begin{array}{cc}0,\hfill & RSSI<RSS{I}_{min},\hfill \\ f\left(RSSI\right)={\displaystyle \frac{RSSI-RSS{I}_{min}}{RSS{I}_{max}-RSS{I}_{min}}},\hfill & RSS{I}_{min}\le RSSI\le RSS{I}_{max},\hfill \\ 1,\hfill & RSSI>RSS{I}_{max}.\hfill \end{array}\right.$$

(7)

The $f\left(RSSI\right)$ function is defined as a normalized linear mapping between the minimum reception threshold ($RSS{I}_{min}$) and the maximum reliable signal strength ($RSS{I}_{max}$). When the received signal strength falls below $RSS{I}_{min}$, packet reception is assumed to be unsuccessful. For signal strengths between $RSS{I}_{min}$ and $RSS{I}_{max}$, the reception probability increases linearly. Signals above $RSS{I}_{max}$ are assumed to be reliably received. This simplified mapping enables computationally efficient estimation of link reliability while remaining consistent with the log-distance path-loss model.

3.5. Expected Duty Cycle (EDC)

The EDC estimates how frequently a neighbor is expected to be awake:

$$EDC\left(i\right)={\displaystyle \frac{1+{\sum }_{j\in forwarders\left(i\right)}L{Q}_{ij}·EDC\left(j\right)}{{\sum }_{j\in forwarders\left(i\right)}L{Q}_{ij}}},$$

(8)

where $L{Q}_{ij}$ indicates the link quality between nodes i and j. A lower EDC indicates more frequent awakenings and, therefore, preferable routing candidates.

3.6. Energy Consumption Model

The RLFS-OR protocol adopts a Castalia-aligned energy model [52] that explicitly accounts for per-frame power consumption, radio communication costs, and state-transition overheads. All energy values are computed in joules, and power terms originate from CC2420-class sensor radios. Each sensor node begins with an initial energy budget of $E_0=100$ J and consumes energy during idle/listen operations, sleep periods, packet transmissions, and packet receptions.

3.6.1. Per-Frame Baseline Power Integration

Every simulation round corresponds to a time frame of 610 ms. For each node, the idle/listen power depends on its duty cycle ($dc$) and local neighborhood density:

$$P_{\mathrm{idle}}\left(d\right)=P_{\mathrm{sleep}}+\left(P_{\mathrm{listen}}-P_{\mathrm{sleep}}\right)d^{1.5},$$

(9)

where $P_{sleep}$ and $P_{listen}$ are 1.0 mW and 56.0 mW, respectively. The exponential 1.5 is a non-linear scaling factor used to indicate that as duty cycle increases, idle/listen energy grows faster than linearly due to overhearing, contention, and channel activity.

Each node also consumes a baseline CPU/sensing cost as $P_{\mathrm{base}}=0.6$ mW. Thus, the per-frame baseline energy consumption is expressed as follows:

$$E_{\mathrm{baseline}}={\displaystyle \frac{\left(P_{\mathrm{base}}+P_{\mathrm{idle}}\left(d\right)\right)T_{\mathrm{frame}}}{1000}},$$

(10)

3.6.2. Transmission and Reception Energy

The radio transmission and reception energy model follows CC2420 standards [12,13]:

$$E_{\mathrm{tx}}(k,d)=k\left(E_{\mathrm{elec}}+E_{\mathrm{amp}}{d}^n\right),$$

(11)

where

• $E_{\mathrm{tx}}(k,d)$ is the energy usage for transmitting k bits through d distance;
• k is the packet length (4000 bits);
• d is the transmission distance between the sender and forwarder;
• n is the free-space path-loss exponent (2);
• $E_{\mathrm{elec}}$ = $50\mathrm{nJ}/\mathrm{bit}$; and
• $E_{\mathrm{amp}}$ = $100\mathrm{pJ}/\mathrm{bit}/{\mathrm{m}}^2$.

$$E_{\mathrm{rx}}\left(k\right)=k{E}_{\mathrm{elec}},$$

(12)

where $E_{\mathrm{rx}}\left(k\right)$ is the energy usage for receiving k bits.

3.6.3. Packet Collision Probability

The collision probability depends on the duty cycle and node density:

$$P_c\left(i\right)=clip\left((0.04+0.22dc)·\left(1+\kappa \left({\displaystyle \frac{deg\left(i\right)}{\overline{deg}}}-1\right)\right),0,0.60\right),$$

(13)

where $\kappa$ is 0.5 and $deg\left(i\right)$ is the number of neighbors of node i.

Equation (13) is a semi-empirical collision probability model designed to capture the combined effects of the duty cycle and local node density under asynchronous medium access. The baseline term $(0.04+0.22dc)$ reflects the observation that higher duty cycles increase channel occupancy time and, consequently, collision likelihood. The density-dependent adjustment term $(1+\kappa ({\displaystyle \frac{deg(i)}{\overline{deg}}}-1))$ scales the collision probability according to the relative neighborhood density of node i, where $deg\left(i\right)$ denotes the local degree and $\overline{deg}$ represents the average network degree. The $\kappa$ parameter controls sensitivity to density variation and was selected based on preliminary simulation calibration to ensure stable behavior across sparse and dense deployments. The clipping operation bounds the collision probability between 0 and 0.60 to avoid unrealistic values in high-density scenarios. This formulation balances modeling simplicity and computational efficiency while preserving realistic MAC-layer contention behavior.

3.6.4. Total Per-Round Energy Consumption

The overall energy consumed by node i during a round is expressed a follows:

$$E_i=E_{baseline}+E_{tx}+E_{rx},$$

(14)

This forms the basis for network lifetime estimation and reward evaluation.

4. Proposed Methods

The proposed RLFS-OR (Reinforcement Learning-based Forwarder Selection for Opportunistic Routing) is a cross-layer design that integrates DQN-based learning with opportunistic routing and duty-cycled MAC behavior.

4.1. Protocol Overview

RLFS-OR aims to overcome the limitations of static metrics (e.g., EDC and hop count) and non-scalable Q-learning by enabling each node to dynamically evaluate its neighbors with a DQN model. The key innovations include the following:

• DQN-based Next-Hop Selection: Each active node runs a DQN that outputs Q-values for all awake neighbors. The top-ranked neighbor is selected as the forwarder.
• MAC-Aware State Representation: The state includes metrics that reflect radio behavior (collision probability), distance to sink (hop count), duty-cycling behavior (EDC), and energy status (residual energy).
• Reward Linked to Realistic Energy Model: The reward penalizes energy loss, long paths, high EDC, and collisions—forcing the agent to learn energy-efficient and reliable forwarding behavior.
• Opportunistic Reception: Multiple neighbors may receive a broadcast, but only the one satisfying RLFS-OR’s policy wakes to forward the packet.

4.2. DQN Architecture and Action Selection

Each node hosts a lightweight neural network:

• Input: Four-dimensional state vector;
• Layers: Two hidden layers (ReLU activation)
• Output: Q-values for each available neighbor.

The policy follows $\epsilon -greedy$ exploration:

$$a=\left\{\begin{array}{cc}\mathrm{random}\mathrm{action},\hfill & \mathrm{with}\mathrm{probability}\epsilon ,\hfill \\ {\displaystyle arg\underset{a}{max}Q(s,a),}\hfill & \mathrm{otherwise},\hfill \end{array}\right.$$

(15)

Experience replay and target network updates ensure stability.

The DQN architecture consists of a fully connected feedforward network with two hidden layers of 64 neurons each and ReLU activation functions. The input dimension is four, corresponding to the state vector $[E^{res},H{C}^{sink},EDC,P^c]$, and the output layer contains two neurons representing candidate forwarding decisions. The learning rate is set to 0.001, and the discount factor is $\gamma =0.98$. Experience replay is employed with a buffer size of 10,000 transitions, and a separate target network is maintained for stable Q-learning.

The network weights are initialized using Xavier initialization. To ensure computational feasibility in resource-constrained WSN environments, only the output-layer parameters are updated during training, while the hidden-layer weights remain fixed after initialization. This significantly reduces gradient computation complexity and memory overhead compared to full backpropagation. Learning is performed entirely online and in a distributed manner, with each node maintaining its own lightweight DQN instance. No offline pre-training or centralized computation is required.

4.3. Packet Forwarding Procedure

Each routing round executes the following:

1.

Node wakes up according to its duty cycle.

2.

Node forms state $\epsilon$.

3.

DQN evaluates Q-values for neighbors.

4.

Next-hop forwarder selected $(\epsilon$-$greedy)$.

5.

If forwarder is awake:

• Transmit packet; update energy.

6.

If forwarder is asleep:

• Node waits; consumes idle energy.

7.

Forwarder calculates reward, updates DQN.

8.

Routing round ends and repeats until depletion.

The process-flow diagram of the proposed protocol is shown in Figure 2, and the DQN-based opportunistic forwarder selection algorithm’s pseudo-code is shown in Algorithm 1. Although Algorithm 1 is presented in a global form for clarity of exposition, the RLFS-OR protocol operates in a fully distributed manner. Specifically, each sensor node independently executes the described procedure using its locally observed state information and neighbor set. The outer loop over routing rounds reflects the progression of network operation in simulation, whereas the inner logic (Lines 9–34) represents the per-node decision-making and learning process executed locally at each active node. No centralized controller or global state information is required.

Algorithm 1 RLFS-OR: Online DQN-Based Opportunistic Forwarder Selection

Require: Wireless sensor network with N nodes; asynchronous duty-cycled MAC; radio energy model; lightweight DQN implemented using Eigen; learning rate $\alpha$; discount factor $\gamma$; exploration rate $\epsilon$ Ensure: Online learned forwarding policy $\pi \left(s\right)$

1: Initialization:   2: Initialize node parameters (residual energy, duty-cycle schedules, neighbor lists)   3: Initialize lightweight DQN using Eigen   4: Fix hidden-layer parameters and initialize outer-layer weights only   5: Initialize replay buffer $\mathcal{B}$   6: Set exploration rate $\epsilon$   7: for each routing round do   8:        Nodes wake up asynchronously according to duty-cycle schedules   9:        for each sender node i do 10:               Sense channel and construct state vector 11:               $s_i\leftarrow [E_i^{res},H{C}_i^{sink},ED{C}_i,P_i^c]$ 12:               Evaluate $Q(s_i,a)$ for all candidate forwarders 13:               if random $<\epsilon$ then 14:                    Select random forwarder a 15:               else 16:                    $a\leftarrow arg{max}_{a}Q(s_i,a)$ 17:               end if 18:               if selected forwarder is awake then 19:                    Transmit packet (TX energy consumption) 20:                    Forwarder receives packet and updates residual energy 21:               else 22:                    Other eligible forwarders wait for random backoff (idle energy loss) 23:               end if 24:               Compute reward 25:               $r\leftarrow {\alpha }_1{E}_i^{res}-{\alpha }_{2}H{C}_i^{sink}-{\alpha }_{3}ED{C}_i-{\alpha }_4{P}_i^c$ 26:               Observe next state $s_i^{\prime }$ 27:               Store transition $(s_i,a,r,s_i^{\prime })$ in $\mathcal{B}$ 28:               if $\mathcal{B}$ is not empty then 29:                    Sample transition $(s,a,r,s^{\prime })$ from $\mathcal{B}$ 30:                    Compute target value 31:                    $y\leftarrow r+\gamma {max}_{a^{\prime }}Q(s^{\prime },a^{\prime })$ 32:                    Update outer-layer weights only using Eigen 33:                    ${\theta }_{\mathrm{out}}\leftarrow {\theta }_{\mathrm{out}}+\alpha (y-Q(s,a)){\nabla }_{{\theta }_{\mathrm{out}}}Q$ 34:               end if 35:        end for 36:        Anneal exploration rate 37:        $\epsilon \leftarrow max({\epsilon }_{min},\epsilon ·\lambda )$ 38: end for 39: return learned policy $\pi \left(s\right)$ (online exploitation)

5. Performance Evaluation

To show the performance of our proposed RLFS-OR protocol, this section presents a precise and concise analysis of the experimental results, their interpretation, and the conclusions drawn from them. Subheadings are used to organize the results by specific aspects or categories of the experiments. The analysis focuses on identifying key patterns, trends, and insights revealed by the data. Furthermore, the conclusions highlight the practical implications of the findings, as well as their relevance to the objectives of the study.

5.1. Simulation Setup

To evaluate the performance of the proposed methods, we used Castalia, an OMNET++-based simulator that is well-suited for WSN simulations. For our simulations, we configured the CC2420 radio module, a built-in parameter setup available in Castalia, to ensure accurate wireless communication modeling.

In this setup, a randomly selected node generates data packets at specified intervals following a given packet rate. All sensor nodes in the network communicated using the BoX-MAC [3] protocol with duty cycles to conserve energy. Because of its compatibility with asynchronous low-duty-cycle WSNs, BoX-MAC was chosen. Our proposed method, RLFS-OR, was implemented on this MAC protocol alongside two existing OR protocols—FCM-OR [14] (with DC) and ORW [4] (with DC)—to compare their performance, primarily in terms of energy consumption.

Initially, we deployed 100 to 1000 sensor nodes non-uniformly in a 1000 m × 1000 m area and placed a sink node at the top-right corner of the sensor field. Except for the sink node, all nodes used a duty cycle ranging from 0.2 to 0.9, and each time frame was fixed at 610 ms. Hence, every 610 ms, the radio of a node changed its state at least once. Each node had a transmission range of 50 m and a carrier sensing range of 100 m. This estimated the minimum hop distance from any corner of the field by approximately 10, and in the worst case, the hop distance could be increased by up to 15.

The transmission range was set to 50 m to reflect a realistic and conservative operating condition of the CC2420 IEEE 802.15.4 transceiver under typical indoor or moderate path-loss environments [12]. Although the CC2420 chip can achieve transmission distances of up to approximately 75–100 m in open-space conditions, depending on the antenna configuration and path-loss characteristics, a 50 m range was selected to ensure multi-hop communication and avoid trivial single-hop connectivity in the 2D deployment field.

The sensing range (100 m) refers to the application-level event detection radius and does not represent radio communication capability. It was intentionally set larger than the transmission range to model scenarios where sensed data must be forwarded via multi-hop routing toward the sink. This separation between sensing and transmission ranges is common in WSN simulations to emulate realistic environmental monitoring deployments.

Lifetime Evaluation

Each sensor node was initialized with an initial energy of $E_{initial}=\mathrm{18,720}\mathrm{J}$ (default value corresponds to two AA batteries). A node was considered dead when its residual energy reached zero.

Network lifetime was evaluated using FND standard metrics:

• FND (First Node Dies): The simulation round at which the first node depletes its energy.

The network lifetime curves were obtained by tracking residual energy of all nodes during simulation and recording the corresponding rounds when these thresholds were reached. Each represented data point in the performance graphs represents the average of 10 independent simulation runs, each using different random deployment seeds. This averaging reduces stochastic bias and ensures statistical reliability.

5.2. Simulation Results

5.2.1. Average Energy Consumption per Node vs. Duty Cycle

Figure 3 shows the total consumed energy per node under varying duty-cycle ratios for three routing protocols: ORW, FCM-OR, and the proposed RLFS-OR. The number of deployed nodes was fixed at 400, and all simulations were conducted with a 100 m transmission range. Across all duty-cycle settings (0.2–0.9), RLFS-OR consistently demonstrates the lowest energy consumption per node, followed by FCM-OR, while ORW shows the highest consumption among the three protocols. As the duty cycle increases, nodes remain awake for longer periods, causing increased idle listening and greater forwarding activity. Consequently, all protocols exhibit a linear rise in per-node energy consumption. However, the rate of this increase differs significantly between protocols. ORW shows the steepest increase, indicating high sensitivity to duty-cycle expansion. FCM-OR performs better than ORW due to fuzzy metric-based candidate selection, which reduces unnecessary transmissions. RLFS-OR rises most slowly with increasing duty cycle, demonstrating strong robustness against higher active ratios. The significant reduction in energy usage for RLFS-OR can be attributed to: (i) Q-value-based forwarder selection, where nodes with higher remaining energy, lower expected delay cost (EDC), lower hop count, and lower collision probability are prioritized; (ii) learning-driven suppression of unnecessary forwarding, preventing high-energy waste in dense deployments; (iii) a backoff time proportional to the Q-value, which reduces redundant transmissions and overhearing; and (iv) balanced energy distribution, since RLFS-OR avoids repeatedly selecting the same small set of forwarders. These mechanisms collectively minimize channel contention, idle listening, and retransmission costs—dominant factors in dense 400-node topologies.

5.2.2. Estimated Network Lifetime vs. Duty Cycle

The graph in Figure 4 shows that the estimated network lifetime decreases as the duty cycle increases. RLFS-OR achieves the longest lifetime at every operating point. At DC = 0.2, RLFS-OR extends the network lifetime by 35–45% over ORW and FCM-OR and maintains a 20–30% improvement even at DC = 0.9. This is due to the protocol’s ability to balance the forwarding load and reduce unnecessary retransmissions, resulting in slower energy depletion across the network.

5.2.3. Average Consumed Energy vs. Density

The graph in Figure 5 displays how the average energy consumption varies as node density increases. For this setup, we used 100 m as the transmission range for nodes and a fixed duty cycle of 0.5 for all nodes. Higher density provides more forwarding choices; hence, in ORW, more redundant transmissions lead to more energy waste. In FCM-OR, fuzzy selection helps, but performance is still not optimal, whereas, in RLFS-OR, DQN identifies low-cost forwarders and reduces traffic overhead. As a result, our proposed RLFS-OR maintains the lowest energy usage across all node densities, showing strong scalability and adaptability to dense network environments.

5.2.4. Estimated Network Lifetime vs. Density

Figure 6 illustrates how the estimated network lifetime varies as node density increases. For this setup, we used the same parameters as in Figure 5. RLFS-OR delivers the highest network lifetime across all densities. ORW shows the shortest lifetime. FCM-OR performs slightly better than ORW but still achieves performance below that of RLFS-OR. As node density increases, the lifetime for RLFS-OR decreases slightly but remains higher than the others. DQN in RLFS-OR learns optimal forwarders and reduces collisions, leading to less wasted energy. Higher density increases contention for ORW/FCM-OR, but RLFS-OR handles contention better via learned Q-value-based decisions. RLFS-OR increases lifetime by up to 30% compared to ORW and 10–15% compared to FCM-OR across all node densities, showing robust scalability.

5.2.5. Average Consumed Energy vs. Traffic Load

Figure 7 compares how the average energy consumption varies as the packet transmission rate increases. Here, a packet rate of 0.1 means a random sender sends 1 packet per 10 s in the running simulation. As the packet rate increases (0.1 to 0.5 packets/10 s), all protocols show increased energy consumption. RLFS-OR again consumes the least energy consistently. ORW has the highest energy cost due to the frequency of transmissions and collisions. FCM-OR is stable, but its transmission cost is still higher than that of RLFS-OR. Because higher packet rates increase channel utilization, in ORW, more collisions and retransmissions occur. FCM-OR performs better but still lacks adaptive learning. On the other hand, RLFS-OR’s smart schedule with Q-based forwarder ranking reduces channel congestion. Therefore, RLFS-OR demonstrates 10–20% lower energy consumption compared to FCM-OR and 15–30% compared to ORW under increasing traffic loads, validating RLFS-OR’s efficiency in heavy-load environments.

5.2.6. End-to-End Packet Delivery Latency vs. Duty Cycle

Figure 8 presents the average end-to-end delay across different duty cycles. At low duty cycles, all protocols experience longerdelays due to limited wake-up availability. As the duty cycle increases, delay generally decreases, since nodes remain active for longer durations. RLFS-OR achieves consistently lower or comparable delay performance compared to ORW and FCM-OR. In particular, at moderate-to-high duty cycles, RLFS-OR demonstrates significantly reduced delay. This improvement can be attributed to learning-driven forwarder selection, which prioritizes neighbors with shorter hop distances and favorable duty-cycle characteristics while avoiding collision-prone nodes.

Although ORW may achieve competitive delay in some mid-range duty cycles, its performance deteriorates at higher duty cycles due to increased contention and redundant transmissions. FCM-OR performs moderately but lacks dynamic adaptation to congestion and energy imbalance. Overall, RLFS-OR achieves improved delay performance while simultaneously reducing retransmission overhead, demonstrating a balanced trade-off between latency and energy efficiency.

5.2.7. Packet Reception Rate vs. Duty Cycle

Figure 9 illustrates the total packet reception rate under varying duty cycles for ORW, FCM-OR, and RLFS-OR. It is important to note that the packet delivery ratio (PDR) remains equal to 1 for all three protocols, since opportunistic routing ensures that packets are eventually delivered through cooperative forwarding and retransmission mechanisms. Therefore, the objective of this analysis is not to evaluate reliability but to assess retransmission efficiency.

As observed in the figure, ORW exhibits a significantly higher packet reception count as the duty cycle increases. This behavior indicates frequent retransmissions and redundant forwarding, particularly in dense wake-up conditions. FCM-OR reduces this redundancy through fuzzy-based candidate filtering; however, it still demonstrates moderate growth in the reception rate at higher duty cycles.

In contrast, RLFS-OR consistently maintains a lower packet reception rate across most duty-cycle settings. This reduction indicates controlled retransmission behavior and improved suppression of unnecessary forwarding. By prioritizing forwarders based on Q-value evaluation—including residual energy, hop count, duty-cycle awareness, and collision probability—RLFS-OR minimizes redundant receptions and overhearing. A lower reception rate, in this context, reflects better transmission discipline and improved energy efficiency rather than reduced reliability. Excessive packet receptions increase idle listening, channel contention, and cumulative radio energy consumption, directly impacting network lifetime. Both Figure 8 and Figure 9 show the performance of the protocols in a high-density node environment.

6. Analysis and Discussion

The experimental results across all scenarios—including varying duty cycles, node densities, and traffic loads—collectively demonstrate that RLFS-OR provides consistently superior performance compared to ORW and FCM-OR. The observed improvements stem directly from the protocol’s reinforcement learning-driven decision-making mechanism, which allows each node to autonomously infer the most energy-efficient, low-collision forwarding behavior in a dynamic multi-hop WSN environment.

First, energy consumption trends reveal clear distinctions in protocol efficiency. Regardless of the duty-cycle ratio or network density, RLFS-OR exhibits markedly lower energy usage per node. This trend is especially evident in dense deployments (400–1000 nodes), where traditional opportunistic protocols such as ORW suffer from severe contention and redundant transmissions, resulting in higher channel activity and increased overhearing. FCM-OR reduces this effect through fuzzy-based ranking but still cannot match the adaptability of RLFS-OR. The DQN-based forwarder selection in RLFS-OR enables nodes to learn long-term energy consequences by incorporating residual energy, hop count, EDC, and collision probability into the Q-value. As a result, RLFS-OR avoids repeatedly selecting the same set of nodes, effectively balancing the load and prolonging the viability of the entire network. This behavior is reflected in the significantly lower energy consumption slopes as duty cycle and traffic load increase.

Second, network lifetime results reinforce the advantage of RL-based adaptability. RLFS-OR extends network lifetime by 20–45% across nearly all evaluated conditions. The improvement is most substantial at low duty-cycle settings, where nodes sleep frequently and poor routing decisions in conventional protocols quickly drain specific bottleneck nodes. RLFS-OR mitigates this issue by learning forwarding patterns that reduce retransmissions and distribute forwarding responsibility more evenly. Even as node density increases—a key factor that typically exacerbates contention—RLFS-OR maintains a longer network lifetime than its counterparts. This indicates that RLFS-OR not only minimizes direct transmission energy but also indirectly reduces energy waste caused by collisions, idle listening, and redundant forwarding attempts.

Third, increased-packet-rate experiments further validate the robustness of RLFS-OR under higher traffic loads. While all protocols experience rising energy consumption as traffic intensifies, RLFS-OR demonstrates the slowestrate of increase. This behavior is attributed to RL’s learned suppression of low-utility transmissions and its Q-value-driven prioritization of energetically favorable forwarders. ORW and FCM-OR, lacking such adaptive behavior, generate more congestion and collisions at high packet rates, leading to significant retransmission overhead. Meanwhile, RLFS-OR maintains stable energy usage by minimizing channel contention and efficiently regulating forwarding attempts.

Finally, the combined effect of the duty cycle, as well as density-based and traffic-based experiments, highlights a general pattern: RLFS-OR scales more gracefully than ORW and FCM-OR across diverse operating conditions. Its learning-based mechanism enables it to adapt to the local topology, traffic conditions, energy states, and collision environments without requiring global coordination or pre-defined heuristics. This adaptability is crucial in real-world WSN deployments, where environmental dynamics and network conditions cannot be predicted in advance. The strong convergence behavior observed in the training reward curve further confirms that the reinforcement learning environment is stable and the learned policies are reliable.

Overall, the results indicate that RLFS-OR effectively overcomes the limitations inherent in both opportunistic routing (ORW) and fuzzy metric-based routing (FCM-OR). By incorporating machine learning into forwarder selection, RLFS-OR achieves a holistic optimization of energy consumption, lifetime extension, and collision reduction. These benefits position RLFS-OR as a practical and scalable next-generation routing solution for large-scale, asynchronous, duty-cycled wireless sensor networks.

7. Future Research Directions

Although RLFS-OR demonstrates strong improvements in energy efficiency, collision reduction, and network lifetime compared to ORW and FCM-OR, several opportunities remain for further advancement. A first direction involves extending the learning model beyond single-agent DQNs. While RLFS-OR already benefits from node-level reinforcement learning, multi-agent RL techniques—such as decentralized actor–critic models or cooperative learning frameworks—may allow nodes to share partial state information and jointly improve forwarding decisions. This could reduce the convergence time, improve stability under dynamic traffic loads, and better exploit spatial diversity in dense WSN deployments.

Another promising direction is the integration of adaptive duty-cycling mechanisms. RLFS-OR currently operates under fixed duty-cycle ratios, but an RL-based wake–sleep controller could dynamically adjust duty cycles according to network congestion, residual energy, and application latency constraints. Such co-design of routing and duty cycling may further reduce idle listening and extend lifetime beyond what routing improvements alone can achieve.

The protocol may also benefit from the incorporation of link-layer awareness, such as through real hardware-derived RSSI modeling, interference profiling, and bursty traffic characterization. More realistic channel models including fading and shadowing would improve the applicability of RLFS-OR to practical outdoor and industrial WSNs. Additionally, integrating advanced collision-avoidance mechanisms—for example, RL-optimized CSMA/CA parameters or probabilistic forwarding—could further reduce the high-density contention observed at larger node counts.

Scalability to larger deployments and mobile settings represents another research dimension. Extending RLFS-OR to handle mobile sinks, mobile sensor nodes, or UAV-assisted data collection would introduce temporal variability, requiring continuous online learning rather than offline training. Similarly, federated or distributed learning strategies could enable large-scale WSNs to share model updates while minimizing communication overhead.

Finally, future work may explore how RLFS-OR interacts with heterogeneous sensor environments, where nodes vary in hardware capabilities, energy budgets, and sensing roles. Tailoring Q-value features or reward functions to individual node classes could yield more specialized and efficient forwarding behaviors. Implementing RLFS-OR on physical hardware or IoT testbeds would also help validate its performance under real network conditions, providing insights into computational overhead, memory constraints, and synchronization challenges.

Overall, the results of this work highlight the potential of reinforcement learning to improve opportunistic routing in asynchronous duty-cycled WSNs, and the future directions outlined above offer pathways to extend RLFS-OR into more adaptive, scalable, and real-world-capable routing solutions.

8. Conclusions

This study investigated the integration of deep reinforcement learning with opportunistic routing under realistic duty-cycled wireless sensor network (WSN) constraints. We proposed RLFS-OR, a Deep Q-Network (DQN)-based opportunistic routing protocol tailored for asynchronous IEEE 802.15.4 environments. By incorporating residual energy, hop count, expected duty-cycle behavior, and collision probability into a unified reward-driven framework, RLFS-OR learns adaptive and energy-efficient forwarding strategies that respond to dynamic wake-up patterns and local contention. Extensive simulations demonstrate that RLFS-OR consistently improves energy consumption, network lifetime, and load balancing compared to ORW and FCM-OR. The protocol effectively suppresses redundant retransmissions, achieves smoother node-death dynamics, and maintains competitive latency performance under varying duty-cycle conditions. The observed stable convergence behavior further confirms the suitability of reinforcement learning for energy-constrained WSN routing. Overall, the results indicate that lightweight DQN-based decision making can significantly enhance opportunistic routing performance in asynchronous duty-cycled sensor networks.