A Lightweight Double-Deep Q-Network for Energy Efficiency Optimization of Industrial IoT Devices in Thermal Power Plants

Abstract

Industrial Internet of Things (IIoT) deployments in thermal power plants face significant energy efficiency challenges due to harsh operating conditions and device resource constraints. This paper presents gradient memory double-deep Q-network (GM-DDQN), a lightweight reinforcement learning approach for energy optimization on resource-constrained IIoT devices. At its core, GM-DDQN introduces the gradient memory mechanism, a novel memory-efficient alternative to experience replay. This core innovation, combined with a simplified neural network architecture and efficient parameter quantization, collectively reduces memory requirements by 99% and computation time by 85–90% compared to standard methods. Experimental evaluations across three realistic simulated thermal power plant scenarios demonstrate that GM-DDQN improves energy efficiency by 42% compared to fixed policies and 27% compared to threshold-based approaches, extending battery lifetime from 8–9 months to 14–15 months while maintaining 96–97% PSR. The method enables sophisticated reinforcement learning directly on IIoT edge devices without requiring cloud connectivity, reducing maintenance costs and improving monitoring reliability in industrial environments.

1. Introduction

The rapid evolution of the Industrial Internet of Things (IIoT) is revolutionizing modern industrial operations, especially within critical infrastructures like thermal power plants [1,2,3,4,5,6,7,8,9]. These complex environments increasingly rely on the IIoT for continuous monitoring, predictive maintenance, and real-time operational optimization [10,11,12,13]. However, deploying wireless sensor networks (WSNs) in such demanding settings introduces significant hurdles, primarily in achieving energy efficiency without compromising reliable communication [14,15,16]. To address this critical gap, this paper introduces a novel resource-aware deep reinforcement learning (DRL) framework designed to optimize energy consumption for IIoT devices in thermal power plants.

1.1. Background

The advent of Industry 4.0 has positioned IIoT technology as a cornerstone for modern thermal power plant operations, extensively utilized for equipment condition-monitoring, predictive maintenance, and real-time control [17,18,19,20,21]. Within these critical infrastructures, the operational status of core machinery, such as boilers and turbines, directly dictates power generation efficiency, systemic safety, and economic viability. IIoT sensors, acting as crucial conduits between physical assets and digital analytical systems, are thus indispensable for the performance and reliability of the entire power generation ecosystem.

Nevertheless, sensors deployed in these extreme environments—characterized by high temperatures often ranging from 100 to 500 °C, high pressures up to 10 to 30 MPa, strong electromagnetic interference (EMI) from power equipment, and arduous installation conditions—face substantial operational challenges [22]. The energy efficiency of battery-powered sensors in such settings is a critical determinant of device lifespan, maintenance expenditure, and overall system reliability. For example, a modest 10% reduction in battery life due to inefficient energy management in a plant with several hundred sensors can escalate annual replacement costs by tens of thousands of dollars [3,23]. More critically, energy depletion leading to monitoring failures can prevent the timely detection of faults in essential equipment. Such failures can precipitate equipment shutdowns, incurring hourly losses that can range from tens to hundreds of thousands of dollars, and, in severe cases, trigger power grid instabilities with potential economic impacts in the millions [24].

Conventional energy management techniques, including fixed power allocation and rudimentary threshold-based sleep strategies, exhibit marked limitations within the dynamic and complex milieu of thermal power plants. Fixed power strategies, for instance, are inherently unable to adapt to fluctuating wireless channel conditions. This rigidity can lead to an estimated 30% increase in energy consumption due to packet retransmissions in high-interference scenarios, or, conversely, energy wastage under benign channel conditions. Similarly, static sleep strategies are ill-equipped to predict and react to sporadic critical events, such as abrupt increases in equipment vibration. Consequently, they risk missing vital data transmissions or incurring unnecessary energy expenditure, potentially up to 50% of standby power, from premature wake-ups during quiescent periods [25,26]. Such inefficiencies culminate in curtailed battery lifespans: for example, a reduction from 12 to 9 months, inflated maintenance overheads, and compromised reliability and safety of the power generation system.

Consequently, maximizing the energy efficiency of IIoT devices—while concurrently ensuring reliable data transmission and prompt responsiveness—emerges as a pivotal technical challenge for the intelligent and sustainable operation of thermal power plants. Addressing this challenge transcends mere engineering concerns of extending device operational lifespans; it is intrinsically linked to enhancing the resilience and economic performance of the entire power system, thereby holding substantial practical and theoretical importance.

1.2. Key Challenges in Energy Efficiency Optimization

Optimizing the energy efficiency of IIoT deployments in thermal power plants is confronted by several formidable challenges:

• Environmental Complexity and Dynamics: Thermal power plant environments are inherently non-stationary. IIoT devices must contend with pronounced wireless channel variations stemming from EMI generated by power equipment, signal reflections from metallic structures, and fluctuating operational loads that might vary, for instance from 50% to 100%. These factors can cause the signal-to-noise ratio (SNR) to plummet from a stable 25 dB to as low as 5 dB within brief intervals. Concurrently, critical parameters such as boiler vibrations or turbine speeds can exhibit erratic changes: for example, the vibrations might escalate from a baseline 20 Hz to 60 Hz during an anomaly, demanding real-time adaptive capabilities from sensor nodes. Such pronounced dynamism renders static optimization strategies largely ineffective in maintaining peak performance [27,28,29].
• Multi-Objective Trade-offs: Critical monitoring applications in thermal power plants necessitate a delicate balance between energy efficiency, system reliability exemplified by data delivery success rates, and responsiveness characterized by low latency. An overemphasis on energy conservation can inadvertently degrade data transmission success rates or protract response times. In high-stakes scenarios, such as the early detection of equipment faults, these compromises can lead to severe consequences; for instance, the delayed detection of a turbine bearing failure could result in hourly losses amounting to hundreds of thousands of dollars. Optimization algorithms must therefore navigate these competing objectives to achieve a holistically optimal operational balance [30,31,32,33].
• Resource-Constrained Edge Devices: IIoT sensors in power plants are typically embedded systems with stringent resource limitations, often featuring microcontrollers (MCUs) like the STM32 series with modest processing power around 48 MHz and limited memory of approximately 32 KB of RAM and 256 KB of flash. In stark contrast, conventional DRL algorithms often demand considerable computational and memory resources, rendering them unsuitable for direct deployment on such edge devices. Consequently, a pivotal challenge lies in designing energy efficiency optimization algorithms that are not only effective but also sufficiently lightweight to operate within these constraints without significant performance degradation [34,35].
• Perceptual and Decisional Uncertainty: Sensor-derived perceptions of the operating environment, including channel quality and equipment vibration levels, are often subject to noise and inherent latencies. Decisions predicated on such imperfect information can, in turn, influence future state observations, creating intricate feedback loops. Furthermore, the unpredictable nature of power plant load variations and dynamic interference patterns introduces a high degree of uncertainty into the decision-making process, demanding robust algorithmic solutions [36,37].

1.3. Our Contributions

To address the aforementioned challenges in optimizing energy efficiency for IIoT devices within thermal power plants, this paper proposes a novel resource-aware double-deep Q-network (GM-DDQN) method. The primary contributions of this work are as follows:

• A Novel Memory-Efficient Learning Mechanism: Gradient Memory. Our primary contribution is the proposal and formalization of the gradient memory mechanism, a conceptual alternative to experience replay designed specifically for memory-constrained edge learning. By storing and reusing a compact history of loss gradients, it fundamentally addresses the memory bottleneck of on-device DRL, paving the way for enhanced on-device intelligence in IIoT networks.
• A Holistic Lightweight DRL Framework for Edge Deployment. Building upon our core innovation, we design and implement the complete GM-DDQN framework. It integrates the gradient memory mechanism with complementary techniques, including a streamlined neural network architecture, parameter quantization, and efficient parameter update strategies. This results in a system that is not only theoretically novel but also practically deployable, drastically cutting computational overhead and inference time while preserving robust learning and decision-making capabilities.
• Joint Adaptive Wireless Communication and Sleep Scheduling. We introduce a novel approach that formulates transmission power control and sleep mode management as a unified joint decision-making problem. The GM-DDQN agent learns to dynamically coordinate these actions based on real-time environmental states, including SNR, equipment vibration intensity, and network traffic. This holistic management demonstrably achieves substantial energy savings, experimentally shown to be between 22% and 42%, directly enhancing device longevity.
• Multi-Objective Optimization Tailored for Industrial Needs. To meet the multifaceted operational requirements of thermal power plants, we designed a novel multi-objective reward function. This function intrinsically balances three critical performance indicators: energy efficiency, data transmission reliability (success rate), and communication latency. Through carefully assigned weighting factors, the GM-DDQN agent is trained to maximize energy efficiency while stringently maintaining data success rates above 95% and communication delays below a 400 ms threshold.
• Rigorous Validation in Realistic Simulated Scenarios. The efficacy of the proposed GM-DDQN framework is demonstrated through extensive and rigorous experimental evaluations. We systematically benchmark GM-DDQN against standard DRL algorithms (DDQN, DDPG, and PPO) and conventional strategies across diverse and realistic simulated thermal power plant scenarios. The results substantiate that GM-DDQN achieves performance comparable to state-of-the-art DRL methods in terms of energy efficiency and data reliability while operating with only a fraction of their computational resource footprint. This comprehensive validation underscores the practical viability of GM-DDQN for large-scale IIoT deployments.

2. Related Work

This section reviews the energy efficiency optimization of IIoT devices, applications of reinforcement learning in resource management, and lightweight deep reinforcement learning techniques for resource-constrained devices. We analyze the limitations in the existing research and explain how our proposed GM-DDQN method addresses these gaps.

2.1. Energy Efficiency Optimization in IIoT

The Industrial Internet of Things, as a core technology of Industry 4.0, is transforming traditional manufacturing. Hu et al. [6] proposed a five-layer architecture for IIoT intelligence empowering smart manufacturing, indicating that energy optimization is one of the important contributions of IIoT intelligence. In this context, energy efficiency of IIoT devices has become a core challenge affecting long-term reliable system operation.

Yadav et al. [38] reviewed various industrial energy optimization methods, including optimization algorithms based on simulated annealing, genetic algorithms, and SARSA-based sleep mode management, which effectively balance energy conservation and service quality. Hou et al. [39] developed a thermal energy harvesting WSN node that converts industrial waste heat into electrical energy using thermoelectric generators, determining a minimum sleep cycle with a 5.4% duty cycle to achieve indefinite operation. Farné et al. [40] demonstrated how to utilize industrial Ethernet infrastructure for real-time monitoring of industrial equipment efficiency without additional monitoring systems.

In wireless communication energy efficiency, Pradhan et al. [41] proposed a joint optimization method for packet length and power allocation to maximize secure energy efficiency in URLLC-oriented IIoT environments. Solati et al. [42] studied energy efficiency in UAV-assisted IIoT networks, combining beamforming and NOMA techniques to solve non-convex optimization problems. Zhou et al. [43] proposed an energy efficiency optimization framework for power line inspection using IIoUAVs in smart grids, jointly considering variables at large and small time scales, designing a two-stage algorithm combining dynamic programming, auction theory, and matching theory. Jiang et al. [44] proposed an energy-efficient networking approach for IIoT cloud services, jointly optimizing distributed data centers and cloud network energy efficiency with a heuristic algorithm combining a niche genetic algorithm and random depth-first search.

In energy system optimization, Mamaghani et al. [45] conducted multi-objective optimization of HT-PEM fuel cell-based micro combined heat and power systems, analyzing the trade-offs between different performance indicators under full- and partial-load conditions. Ma et al. [46] studied multi-objective performance optimization for gas turbine part-load operation, and Kumar [47] conducted a comprehensive 4-E analysis of thermal power plants, identifying key energy loss locations. Qu et al. [48] proposed a dual-region topology optimization method to improve thermal cycle efficiency in nuclear power plants, achieving approximately a 240% efficiency improvement. Liu et al. [49] conducted thermal economy analysis and multi-objective optimization of a CO₂ transcritical pumped thermal electricity storage system, increasing the energy utilization rate to 97.66%. Cacciali et al. [50] investigated thermal energy integration and optimization in compressed air energy storage systems, proposing solutions for solid and liquid TES systems.

While these traditional optimization methods are effective under specific conditions, they often rely on predefined rules or simplified models and struggle to adapt to dynamic changes in complex industrial environments like thermal power plants, prompting researchers to explore more advanced methods.

2.2. Reinforcement Learning-Based Energy and Resource Management

Deep reinforcement learning, with its ability to learn optimal strategies through interaction without requiring precise environment models, provides a powerful tool for optimizing complex dynamic systems. Chen et al. [36] comprehensively reviewed DRL applications in IoT, highlighting its potential in addressing communication, computation, caching, and control challenges, especially emphasizing its value in smart grid energy management.

Zhang et al. [51] proposed an intelligent resource adaptation scheme for diversified service requirements in IIoT, designing a KODDQN algorithm to solve the optimization problem of maximizing long-term key quality indicators (KQIs). Dridi et al. [52] proposed a DRL method based on an RNN architecture for microgrid distributed energy management, demonstrating superior performance and learning speed compared to traditional methods. Dolatabadi et al. [53] proposed a completely model-free DDPG framework for optimizing PV-integrated energy hub scheduling, analyzing DQN’s limitations in handling continuous action spaces and DDPG’s advantages.

Furthermore, deep reinforcement learning and related machine learning techniques have been widely applied to solve diverse resource and energy optimization problems, such as in mobile edge computing for joint task offloading and resource allocation with energy harvesting [54], in virtualized radio access networks for optimizing network function placement using graph neural networks [55], and even in logistics for enhancing time and energy efficiency in machine learning-driven truck–drone collaborative deliveries [56]. Similarly, deep Q-learning has been utilized to develop efficient UAV detouring algorithms for energy-minimized data collection and sensor recharging within constrained wireless sensor network environments [57].

While these studies demonstrate DRL’s potential in energy and resource management, their high computational and memory requirements make direct deployment on resource-constrained IIoT devices challenging, highlighting the necessity of developing lightweight DRL methods.

2.3. Lightweight DRL Methods and Edge Intelligence

Traditional DRL methods’ high resource demands fundamentally conflict with IIoT devices’ resource constraints. Chen et al. [36] emphasized the concept of edge intelligence—deploying AI at network edges—highlighting benefits including reduced latency, enhanced security, and improved energy efficiency while acknowledging the significant challenges in implementing complex DRL algorithms on resource-constrained edge devices.

Singh et al. [58] proposed a compact IIoT system for remote monitoring of industrial facilities, integrating real-time data analysis, predictive maintenance, and energy optimization algorithms, emphasizing the importance of local data processing in edge computing environments. Dridi et al. [52] also identified computational and data requirements of reinforcement learning, suggesting further simplification and optimization for resource-constrained devices.

The existing lightweight DRL research primarily focuses on model compression, knowledge distillation, and efficient network architectures but lacks optimization specifically for industrial scenarios like thermal power plants, particularly in multi-objective optimization problems involving transmission power adjustment and sleep mode management.

2.4. Research Gaps and Contributions

Based on a comprehensive analysis of the existing research, we identify several key research gaps: First, the existing IIoT energy efficiency optimization methods often rely on predefined rules or computationally intensive algorithms, struggling to balance adaptability and resource constraints. The traditional methods by Yadav et al. [38] lack environmental adaptability; Ma et al. [46] focused on industrial equipment rather than monitoring devices; Hou et al. [39]’s fixed sleep cycle strategy lacks adaptability; Farné et al. [40] primarily optimized monitored equipment rather than monitoring devices themselves; and the methods by Pradhan et al. [41] and Zhou et al. [43] have high computational complexity, making implementation on resource-constrained devices difficult. Second, although Dridi et al. [52], Dolatabadi et al. [53], and Zhang et al. [51] demonstrated DRL’s potential in energy management, their high computational requirements limit application on resource-constrained IIoT devices. Singh et al. [58], while considering resource constraints, lacked adaptive learning mechanisms for dynamic environments. Third, the existing lightweight DRL methods lack optimization specifically for industrial scenarios like thermal power plants, unable to effectively address their complex environmental characteristics and multi-objective optimization needs.

To address these gaps, our proposed GM-DDQN method offers the following innovative contributions:

Lightweight Deep Reinforcement Learning Framework: We propose a GM-DDQN approach that is suitable for embedded systems, significantly reducing the resource requirements while maintaining decision-making performance through a simplified network structure, a novel gradient memory mechanism that replaces experience replay, and streamlined update strategies.

Adaptive Wireless Communication Strategy: We innovatively treat transmission power adjustment and sleep mode management as a joint decision problem, dynamically coordinating both based on the environmental states—increasing power and maintaining activity during high-interference and high-load periods while reducing power and entering deep sleep during stable low-load periods.

Multi-Objective Optimization for Industrial IoT: We design a multi-objective reward function specifically for thermal power plants, organically combining energy efficiency, data success rate, and communication delay, allowing weight adjustments to maximize energy efficiency while ensuring high reliability and low latency.

The experimental results show that GM-DDQN performs comparably to more complex algorithms while consuming only about one-twentieth of the computational resources, providing a practical solution for the large-scale deployment of IIoT systems in thermal power plants, demonstrating significant engineering application value.

3. Problem Modeling and Environment Definition

This section formalizes the energy efficiency optimization problem for IIoT devices operating within thermal power plants. We first detail the system model and delineate the specific environmental characteristics of these industrial settings. Subsequently, we formulate the optimization task as a Markov Decision Process (MDP), providing a framework for applying reinforcement learning.

3.1. System Model

We consider an IIoT network deployed within a thermal power plant, comprising multiple battery-powered sensor nodes. These nodes are responsible for collecting environmental and equipment data and transmitting it to a central gateway for further processing or decision-making. The overall system architecture, illustrating the interaction between sensor nodes, the IIoT environment, and the decision optimization agent, is depicted in Figure 1.

Each sensor node $i\in \mathcal{N}=\{1,2,...,N\}$ is defined by several key attributes:

• Maximum battery capacity ($B_i^{max}$ in mAh) and current battery level ($B_i\left(t\right)$).
• A discrete set of available transmission power levels (${\mathcal{P}}_i=\{p_i^1,...,p_i^K\}$ in dBm).
• A discrete set of available sleep modes (${\mathcal{S}}_i=\{s_i^1,...,s_i^L\}$), each mode $s_i^l$ having a specific power consumption rate ($e_i^l$ in mW) and wake-up delay ($d_i^l$ in ms).
• An average data generation rate (${\lambda }_i$ in packets/s) and processing capability ($C_i$ in MIPS).

The wireless communication channel between node i and the gateway is influenced by time-varying path loss $P{L}_i\left(t\right)$, constant channel noise power $N_0$, and interference $I_i\left(t\right)$. The signal-to-noise-plus-interference ratio (SINR), ${\gamma }_i\left(t\right)$, is crucial for communication quality and is given by

$${\gamma }_i\left(t\right)={\displaystyle \frac{p_i\left(t\right)·P{L}_i\left(t\right)}{N_0+I_i\left(t\right)}}$$

(1)

where $p_i\left(t\right)\in {\mathcal{P}}_i$ is the selected transmission power. A detailed temperature-influenced model for $P{L}_i\left(t\right)$ is presented in Section 3.2.3.

The Packet Success Rate (PSR), $PS{R}_i\left(t\right)$, representing the probability of successful data transmission, is modeled as a function of ${\gamma }_i\left(t\right)$:

$$PS{R}_i\left(t\right)=\left\{\begin{array}{cc}1-exp(-k·({\gamma }_i\left(t\right)-{\gamma }_{th})),\hfill & \mathrm{if}{\gamma }_i\left(t\right)>{\gamma }_{th}\hfill \\ 0,\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(2)

Here, ${\gamma }_{th}$ is the minimum SINR for successful reception, and k is a constant related to the modulation and coding scheme.

3.2. Thermal Power Plant Environment Characteristics

The IIoT operational environment in thermal power plants is characterized by harsh and dynamic conditions impacting sensor performance and energy consumption.

3.2.1. Electromagnetic Interference Patterns

Electromagnetic interference (EMI), $I_i\left(t\right)$, is a significant factor composed of three distinct components: a stable background level, predictable cyclic noise, and sporadic bursts. This can be modeled as

$$I_i\left(t\right)=I_{\mathrm{base}}+I_{\mathrm{var}}\left(t\right)+I_{\mathrm{burst}}\left(t\right)$$

(3)

Here, $I_{\mathrm{base}}$ represents the persistent baseline EMI from distant power lines and general industrial noise. $I_{\mathrm{burst}}\left(t\right)$ captures sporadic high-intensity interference from transient events like large motor startups or circuit breaker switching.

The term $I_{\mathrm{var}}\left(t\right)$ models the predictable time-varying interference linked to the cyclical operation of heavy machinery, such as turbines and generators. It can be represented as a superposition of sinusoidal waves:

$$I_{\mathrm{var}}\left(t\right)=A·sin\left(2\pi f_{1}t\right)+B·sin(2\pi f_{2}t+\varphi )$$

(4)

In this model, amplitudes A and B and frequencies $f_1$ and $f_2$ correspond to the interference strength and fundamental operating frequencies of different major equipment (e.g., a 50/60 Hz generator and a variable-frequency drive motor), while $\varphi$ represents a phase offset.

3.2.2. Equipment Vibration and Monitoring Needs

Equipment vibration intensity, $V_i\left(t\right)$, is a key indicator of machinery health. We model it as the sum of a normal baseline, predictable fluctuations, and potential anomaly signals:

$$V_i\left(t\right)=V_{\mathrm{normal}}+V_{\mathrm{fluctuation}}\left(t\right)+V_{\mathrm{anomaly}}\left(t\right)$$

(5)

Here, $V_{\mathrm{normal}}$ represents the baseline vibration of a machine operating in a healthy steady state. $V_{\mathrm{fluctuation}}\left(t\right)$ models the regular load-dependent variations in vibration that occur as the power plant’s output changes throughout the day. Critically, $V_{\mathrm{anomaly}}\left(t\right)$ signifies sharp, irregular, and significant deviations from the norm, which could indicate an impending equipment fault, such as bearing wear or blade imbalance, requiring immediate attention.

3.2.3. Temperature and Path Loss Variations

Ambient temperature $T_i\left(t\right)$ at a sensor’s location is dynamic, affected by machinery and diurnal cycles:

$$T_i\left(t\right)=T_{base}+\mathrm{\Delta }{T}_{operation}\left(t\right)+\mathrm{\Delta }{T}_{daily}\left(t\right)$$

(6)

Temperature influences wireless path loss $P{L}_i\left(t\right)$, which is modeled as Equation (7):

$$P{L}_i\left(t\right)=P{L}_0·{\left({\displaystyle \frac{d_i}{d_0}}\right)}^{-\alpha }·(1+\beta ·(T_i\left(t\right)-T_{ref}))$$

(7)

where $P{L}_0$ is reference path loss at distance $d_0$, $d_i$ is node–gateway distance, $\alpha$ is path loss exponent, $\beta$ is temperature coefficient, and $T_{ref}$ is reference temperature.

These environmental factors (EMI, vibration, and temperature) are often interconnected, creating complex coupled effects on sensor performance and energy use. For instance, increased plant load can simultaneously elevate EMI, vibration, and local temperatures.

3.3. Energy Consumption Model

The total energy consumed by sensor node i during a time step t, $E_i\left(t\right)$, is the sum of:

$$E_i\left(t\right)=E_i^{tx}\left(t\right)+E_i^{rx}\left(t\right)+E_i^{proc}\left(t\right)+E_i^{sleep}\left(t\right)$$

(8)

representing energy for transmission, reception, processing, and sleep modes, respectively.

Transmission energy $E_i^{tx}\left(t\right)$ for $N_i^{tx}\left(t\right)$ packets of size L bits, at data rate $R_i\left(t\right)$ and power $p_i\left(t\right)$, is

$$E_i^{tx}\left(t\right)=p_i\left(t\right)·{\displaystyle \frac{L·N_i^{tx}\left(t\right)}{R_i\left(t\right)}}$$

(9)

This is the energy for transmission attempts; effective energy per successfully delivered bit increases if $PS{R}_i\left(t\right)<1$ due to retransmissions.

Reception energy $E_i^{rx}\left(t\right)$ and processing energy $E_i^{proc}\left(t\right)$ (dependent on $C_i$ and processing tasks) are considered. For modeling simplicity in this work, these are assumed to be relatively constant or subsumed into an average active-mode power consumption when not transmitting or in deep sleep, allowing focus on the dominant controllable terms $E_i^{tx}\left(t\right)$ and $E_i^{sleep}\left(t\right)$.

Sleep mode energy $E_i^{sleep}\left(t\right)$ for a chosen mode $s_i\left(t\right)\in {\mathcal{S}}_i$ is

$$E_i^{sleep}\left(t\right)=e_i^{s_i\left(t\right)}·T_i^{sleep}\left(t\right)$$

(10)

where $e_i^{s_i\left(t\right)}$ is the mode’s power rate and $T_i^{sleep}\left(t\right)$ is its duration.

3.4. MDP Formulation

The energy efficiency optimization is formulated as an MDP, defined by $(\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R},\gamma )$, where the IIoT device (agent) learns to minimize long-term energy use while meeting performance criteria.

3.4.1. State Space ($\mathcal{S}$)

The state $s_t\in \mathcal{S}$ at time t provides the agent with essential context for decision-making:

$$s_t=\langle {\gamma }_i\left(t\right),B_i\left(t\right),V_i\left(t\right),Q_i\left(t\right),T_i\left(t\right)\rangle$$

(11)

This vector comprises current SINR (${\gamma }_i\left(t\right)$), remaining battery ($B_i\left(t\right)$), equipment vibration ($V_i\left(t\right)$), data queue length ($Q_i\left(t\right)$), and ambient temperature ($T_i\left(t\right)$). These variables are selected for their direct influence on communication, energy, and data relevance.

3.4.2. Action Space ($\mathcal{A}$)

The action $a_t\in A$ at time t is a single choice from a discrete action space. This space is constructed as the Cartesian product of the set of available transmission power levels $P_i$ and the set of available sleep modes $S_i$. Each action is therefore a composite tuple that jointly determines both control variables:

$$a_t=\langle p_i\left(t\right),s_i\left(t\right)\rangle ,\mathrm{where}{p}_i\left(t\right)\in P_i\mathrm{and}{s}_i\left(t\right)\in S_i$$

(12)

For instance, with the 4 power levels and 3 sleep modes defined in our experiments, the agent selects one action from a total of $\left|A\right|=4\times 3=12$ possible discrete actions at each time step. This formulation transforms the multi-dimensional control problem into a unified single-dimensional decision task. Consequently, the action space remains small and computationally tractable, directly avoiding the combinatorial explosion that would arise from treating each control variable independently.

3.4.3. Transition Probability ($\mathcal{P}$)

The transition probability $\mathcal{P}\left(s_{t+1}\right|s_t,a_t)$ dictates the likelihood of moving from state $s_t$ to $s_{t+1}$ given action $a_t$. It is governed by environmental dynamics (Section 3.2) and action consequences. In our model-free DRL approach, $\mathcal{P}$ is learned implicitly through environmental interaction.

3.4.4. Reward Function ($\mathcal{R}$)

The reward function $\mathcal{R}(s_t,a_t)$ guides the agent by quantifying the desirability of actions, balancing multiple objectives:

$$\mathcal{R}(s_t,a_t)=w_1{R}_{energy}+w_2{R}_{reliability}+w_3{R}_{latency}$$

(13)

Components (for state $s_t$ and action $a_t$):

• Energy: $R_{energy}=-E_i\left(t\right)/E_{max}$, penalizing normalized energy consumption.
• Reliability: $R_{reliability}=PS{R}_i\left(t\right)$, rewarding successful transmissions.
• Latency: $R_{latency}$ is defined as

  $$R_{latency}=\left\{\begin{array}{cc}1,\hfill & \mathrm{if}{D}_i\left(t\right)\le D_{th}\hfill \\ 1-{\displaystyle \frac{D_i\left(t\right)-D_{th}}{D_{max}-D_{th}}},\hfill & \mathrm{if}{D}_{th}<D_i\left(t\right)<D_{max}\hfill \\ 0,\hfill & \mathrm{if}{D}_i\left(t\right)\ge D_{max}\hfill \end{array}\right.$$

  (14)

  rewarding timely data delivery relative to thresholds $D_{th}$ and $D_{max}$.

Weights $w_1=0.5,w_2=0.4,w_3=0.1$ (summing to 1) prioritize energy saving and reliability, reflecting typical industrial monitoring needs. These were set based on preliminary experiments and domain knowledge; their sensitivity is a subject for future work.

3.4.5. Discount Factor ($\gamma$)

A discount factor $\gamma =0.95$ is used, valuing future rewards to encourage long-term energy-efficient behavior.

3.5. Optimization Objective

The goal is to find an optimal policy ${\pi }^{\ast }:\mathcal{S}\to \mathcal{A}$ that maximizes the expected sum of discounted future rewards:

$${\pi }^{\ast }=arg\underset{\pi }{max}\mathbb{E}\left[\sum _{k=0}^{\infty }{\gamma }^k\mathcal{R}(s_{t+k},a_{t+k})|s_t,a_{t+k}=\pi \left(s_{t+k}\right)\right]$$

(15)

This policy guides the IIoT node to make optimal power and sleep mode choices to balance energy, reliability, and latency.

3.6. Problem Complexity Analysis

Optimizing energy efficiency in this context is challenging due to the following:

• High-Dimensional and Continuous States: The state space is vast, making tabular RL methods impractical.
• Non-Stationary Environment: Plant dynamics change, requiring adaptive policies.
• Delayed Rewards and Temporal Credit Assignment: The long-term effects of actions are hard to attribute.
• Complex Objective Trade-offs: Balancing energy, reliability, and latency is non-trivial.
• IIoT Device Resource Constraints: Algorithms must be lightweight and computationally efficient.

These complexities necessitate an intelligent, adaptive, and resource-efficient approach like the proposed GM-DDQN, capable of learning effective policies from direct interaction with the dynamic environment.

4. Lightweight Double-Deep Q-Network (GM-DDQN) Method

This section details our proposed gradient memory double-deep Q-network (GM-DDQN), an approach specifically engineered for energy efficiency optimization on resource-constrained IIoT devices. We begin with an overview of the GM-DDQN architecture, followed by a discussion of its core lightweighting components, state and action representations, training methodology, practical implementation considerations for IIoT deployment, and a brief theoretical analysis.

4.1. GM-DDQN Architecture Overview

The GM-DDQN framework adapts the conventional double-deep Q-network (DDQN) by incorporating several key modifications aimed at drastically reducing computational and memory demands without substantial performance degradation. The overall architecture, as illustrated in Figure 2, highlights the interaction between the input state, the lightweight Q-networks (main and target), and the novel gradient memory mechanism.

Distinct from standard DDQN, which typically employs two identically structured, often large, neural networks, GM-DDQN utilizes a significantly more compact network design. Furthermore, it replaces the conventional large experience replay buffer with a “gradient memory” mechanism, designed for efficient learning from a smaller processed representation of recent experiences.

4.2. The Core Innovation: The Gradient Memory Mechanism

The conceptual cornerstone of GM-DDQN is the gradient memory mechanism, a novel and memory-efficient alternative to the conventional experience replay buffer that forms the primary memory bottleneck in standard DRL algorithms. Instead of storing full $(s,a,r,s^{\prime })$ transitions, which is infeasible on resource-constrained devices, our approach maintains a compact cyclical buffer of the most recent loss gradients.

The loss function $L\left({\theta }_t\right)$ at time t is the standard DDQN loss:

$$L\left({\theta }_t\right)={\mathbb{E}}_{(s,a,r,s^{\prime })\sim D_t}\left[{(y_t-Q(s,a;{\theta }_t))}^2\right]$$

(16)

where $y_t=r+\gamma {max}_{a^{\prime }}Q(s^{\prime },a^{\prime };{\theta }_t^{-})$ is the target value, with ${\theta }_t^{-}$ being the parameters of the target network. The gradient $g_t={\nabla }_{\theta }L\left({\theta }_t\right)$ is computed.

A gradient memory matrix $G\in {\mathbb{R}}^{m\times \left|\theta \right|}$ stores the last m gradients (typically $m=8$ in our setup, where $\left|\theta \right|=572$ is the number of parameters). G is updated cyclically: $G[tmodm]=g_t$.

The Q-network parameters $\theta$ are then updated using a weighted average of these stored gradients:

$${\theta }_{t+1}={\theta }_t-\alpha ·\sum _{j=0}^{m-1}{w}_j·G\left[j\right]$$

(17)

where $\alpha$ is the learning rate. The weights $w_j$ prioritize more recent gradients, calculated as

$$w_j={\displaystyle \frac{exp(-\lambda ·((tmodm)-jmodm\left)\right)}{{\sum }_{k=0}^{m-1}exp(-\lambda ·((tmodm)-kmodm))}}$$

(18)

with a decay parameter $\lambda =0.5$. This mechanism drastically reduces memory requirements compared to experience replay, as shown in

Table 1. Memory footprint: standard DDQN with experience replay vs. GM-DDQN.

Memory Component	Std. DDQN (10k Samples)	GM-DDQN ($\mathit{m}=8$)
Experience/Gradient Storage	∼480 KB (for transitions)	∼4.5 KB (for gradients)
Network Parameters (Quantized)	∼21.3 KB (float32)	∼0.56 KB (int8)
Approx. Total On-Device	∼501.3 KB	∼5.06 KB

.

The total memory reduction is approximately 99%, rendering GM-DDQN highly suitable for memory-constrained IIoT devices.

4.3. Supporting Lightweighting Techniques

To fully realize the benefits of the gradient memory mechanism and create a deployable on-device framework, we complement our core innovation with two established lightweighting strategies.

4.3.1. Compact Neural Network Architecture and Quantization

The neural network in GM-DDQN is a multi-layer perceptron (MLP) with a reduced number of layers and neurons compared to typical DRL models. For a given state $s_t$, the Q-value $Q(s_t,a)$ for each action a is computed as

$$\begin{array}{cc}\hfill h_1& =\sigma (W_1{s}_t+b_1)\hfill \\ \hfill h_2& =\sigma (W_2{h}_1+b_2)\hfill \\ \hfill Q(s_t,a)& =W_3{h}_2+b_3\hfill \end{array}$$

(19)

where $s_t\in {\mathbb{R}}^{n_s}$ is the state vector (with $n_s=5$ as per Section 3.4.1), $\sigma$ is the ReLU activation function, $h_1,h_2\in {\mathbb{R}}^{n_h}$ are hidden layers (with $n_h=16$), and $W_k,b_k$ are the corresponding weight matrices and bias vectors. The output layer has $\left|A\right|$ neurons, where $\left|A\right|=12$ (4 power levels × 3 sleep modes) is the dimension of our discrete action space.

This streamlined structure leads to a significant parameter reduction.

Table 2. Parameter count comparison: standard DDQN vs. GM-DDQN.

Network Layer	Std. DDQN (${\mathit{n}}_{\mathit{h}}=64$)	GM-DDQN (${\mathit{n}}_{\mathit{h}}=16$)	Reduction
Input to Hidden 1	$5\times 64+64=384$	$5\times 16+16=96$	75.0%
Hidden 1 to Hidden 2	$64\times 64+64=$ 4160	$16\times 16+16=272$	93.5%
Hidden 2 to Output	$64\times 12+12=780$	$16\times 12+12=204$	73.8%
Total Parameters	5324	572	89.3%

compares the parameter count of a representative standard DDQN (e.g., with $n_h=64$) against our GM-DDQN.

To further minimize the computational and memory footprint, 8-bit quantization is applied to the network parameters ($W_k,b_k$). The quantized weight $W_q$ is obtained from W using

$$W_q=\mathrm{round}\left({\displaystyle \frac{W-min\left(W\right)}{max\left(W\right)-min\left(W\right)}}\times (2^8-1)\right)$$

(20)

Inference then utilizes fixed-point arithmetic: for instance, calculating a quantized hidden layer activation $h_q$:

$$h_q=\mathrm{clip}\left(\mathrm{round}\left({\displaystyle \frac{W_q·s_q+b_q}{S}}\right),0,255\right)$$

(21)

where $s_q,b_q$ are quantized inputs and biases, and S is a scaling factor determined during quantization. This 89.3% parameter reduction, coupled with quantization, significantly lowers memory needs and accelerates inference, crucial for IIoT deployment.

4.3.2. Efficient Target Network Updates

Standard DDQN involves periodically copying all parameters from the main Q-network to the target network ${\theta }^{-}$. GM-DDQN employs a more computationally frugal soft update mechanism:

$${\theta }_t^{-}=\beta ·{\theta }_t+(1-\beta )·{\theta }_{t-1}^{-}$$

(22)

where $\beta$ is a small update coefficient (e.g., $\beta =0.05$), gradually blending the main network’s parameters into the target network.

4.4. State and Action Representation

Effective learning requires appropriate representation of states and actions.

4.4.1. State Normalization

Input state components (SINR ${\gamma }_i\left(t\right)$, battery $B_i\left(t\right)$, vibration $V_i\left(t\right)$, queue $Q_i\left(t\right)$, and temperature $T_i\left(t\right)$) are normalized to the range $[0,1]$ to enhance learning stability and efficiency. For example, for SINR,

$${\gamma }_i^{norm}\left(t\right)={\displaystyle \frac{{\gamma }_i\left(t\right)-{\gamma }_{min}}{{\gamma }_{max}-{\gamma }_{min}}}$$

(23)

Similar normalization is applied to other components using their respective predefined min/max values or thresholds (e.g., $B_i^{max}$ for battery and $Q_{th}$ for queue length).

4.4.2. Action Discretization

As established in the problem formulation (Section 3.4.2), the agent’s actions are composite choices of transmission power and sleep mode. To make this tractable for our DRL algorithm, these choices are handled within a single flat action space. We define a set of four power levels (e.g., $\{0,5,10,15\}$ dBm) and three sleep modes (e.g., active, light sleep, and deep sleep), resulting in a cardinality of $\left|A\right|=4\times 3=12$ discrete actions. For instance, action index 0 could map to the pair {0 dBm, deep sleep}, while action index 11 could map to {15 dBm, active}. At each time step, the agent’s neural network outputs Q-values for all 12 actions, and the agent selects the single action with the highest value. This approach ensures the action space remains computationally tractable while providing sufficient control granularity.

4.5. Training Procedure

The GM-DDQN is trained offline using the procedure outlined in Algorithm 1. Key parameters include a learning rate $\alpha =0.001$, discount factor $\gamma =0.95$, and an exploration probability $ϵ$ that anneals from 1.0 to 0.05 during training to balance exploration and exploitation.

4.6. Implementation Considerations for IIoT Devices

Deploying GM-DDQN on IIoT devices necessitates careful optimization of memory and computation:

• Memory Footprint Optimization: Quantized weights are stored in flash memory (often more plentiful than RAM). In-place computations for activations and direct matrix operations on quantized values further reduce RAM usage by avoiding intermediate dequantized copies.
• Computational Efficiency: Activation functions are implemented via lookup tables to bypass costly floating-point operations on MCUs lacking hardware FPUs. “Batch-free” inference (one sample at a time) minimizes peak RAM. Early stopping heuristics during inference can prune unnecessary computations if an action can be determined with partial evaluation.
• Energy-Aware Execution: The inference frequency itself is adapted based on battery level and environmental stability. Inference is coordinated with data transmission to utilize active processor states. Between decisions, the device enters optimized low-power modes, selected based on the predicted time to the next decision.

These strategies ensure that the GM-DDQN agent not only learns energy-efficient policies but also executes efficiently.

Algorithm 1 GM-DDQN Training Procedure

1: Input: Learning rate $\alpha$, discount factor $\gamma$, target update rate $\beta$, gradient memory size m   2: Output: Trained and quantized Q-network parameters $\theta$   3: Initialize main Q-network $\theta$ and target Q-network ${\theta }^{-}=\theta$ randomly.   4: Initialize gradient memory $G={\mathbf{0}}^{m\times \left|\theta \right|}$.   5: Initialize time step $t=0$.   6: while not converged do   7:    Observe current state $s_t$.   8:    Select action $a_t$: with probability $ϵ$ choose random action, else $a_t=arg{max}_{a}Q(s_t,a;\theta )$.   9:    Execute $a_t$: observe reward $r_t$ and next state $s_{t+1}$. 10:    Compute target Q-value $y_t=r_t+\gamma {max}_{a^{\prime }}Q(s_{t+1},a^{\prime };{\theta }^{-})$. 11:    Compute loss $L_t={(y_t-Q(s_t,a_t;\theta ))}^2$. 12:    Compute gradient $g_t={\nabla }_{\theta }{L}_t$. 13:    Store gradient: $G[tmodm]\leftarrow g_t$. 14:    Compute weighted average gradient ${\overline{g}}_t={\sum }_{j=0}^{m-1}{w}_j·G\left[j\right]$ using Equation (18). 15:    Update main network: $\theta \leftarrow \theta -\alpha ·{\overline{g}}_t$. 16:    Update target network: ${\theta }^{-}\leftarrow \beta ·\theta +(1-\beta )·{\theta }^{-}$. 17:    $t\leftarrow t+1$. 18:    Anneal $ϵ$. 19: end while 20: Quantize final parameters $\theta$ for deployment. 21: Return$\theta$.

4.7. Theoretical Analysis

Despite its modifications, GM-DDQN retains desirable theoretical properties while achieving its lightweight goals.

Convergence and Stability through Smoothed Gradient Estimation: The gradient memory mechanism, which uses a weighted sum of recent gradients for updates, can be viewed as a form of smoothed gradient estimation. By averaging over the last m gradients, it reduces the variance of the update signal, which can lead to more stable convergence, analogous to the role of momentum in optimization algorithms. This smoothing effect is particularly beneficial in noisy and non-stationary environments as it prevents the learning process from overreacting to transient fluctuations.

Approximating Recent Experience Sampling: Furthermore, the gradient memory serves as a computationally efficient proxy for experience replay. Instead of storing memory-intensive state-transition tuples $(s,a,r,s^{\prime })$, it stores a compact history of their resulting learning signals (gradients). The update rule in Equation (17) effectively approximates the expected gradient over a mini-batch of experiences drawn from an implicit non-uniform distribution that heavily favors recent policy-relevant interactions. This allows the agent to learn from a diverse yet temporally focused set of experiences, retaining the stabilization benefits of experience replay while being orders of magnitude more memory-efficient.

Computational Complexity: Inference complexity is dominated by matrix multiplications in the small network, roughly $O(n_s{n}_h+n_h^2+n_h|A\left|\right)$, significantly lower than larger networks. Training update complexity per step is primarily driven by gradient computation and the weighted sum over the gradient memory, $O\left(m\right|\theta \left|\right)$, which is efficient due to small m and $\left|\theta \right|$.

Approximation Error: Simplification and quantization introduce approximation errors. If $f^{\ast }$ is the true optimal Q-function, $f_{\mathrm{DDQN}}$ the approximation by a standard DDQN, and $f_{\mathrm{GM}\text{-}\mathrm{DDQN}}$ by our method, then $\parallel f^{\ast }-f_{\mathrm{GM}\text{-}\mathrm{DDQN}}{\parallel }_{\infty }\le \parallel f^{\ast }-f_{\mathrm{DDQN}}{\parallel }_{\infty }+{\parallel f_{\mathrm{DDQN}}-f_{\mathrm{GM}\text{-}\mathrm{DDQN}}\parallel }_{\infty }$. The second term, due to architectural simplification and quantization, can be bounded, with the architectural part related to $O(1/\sqrt{n_h})$. Careful design aims to keep this additional error small.

In essence, GM-DDQN is tailored for resource-constrained settings by strategically replacing the large memory-intensive experience replay buffer with the compact computationally efficient gradient memory. This novel mechanism aims to preserve the core learning capabilities and stability of the DDQN framework through smoothed history-aware gradient updates.

5. Experiments

This section describes the experimental framework developed to evaluate the proposed GM-DDQN method, including the simulation environment, test scenarios, implementation specifics, comparative baselines, evaluation metrics, and a detailed analysis of the results. We conduct comprehensive experiments to assess the energy efficiency, communication reliability, computational resource requirements, and overall performance of our approach in various simulated thermal power plant scenarios.

5.1. Experimental Setup

5.1.1. Simulation Environment

To evaluate the performance of our proposed method in environments that closely mimic real thermal power plants, we developed a comprehensive simulation platform. A critical aspect of our experimental design is to distinguish between the learning agent and the overall network environment. Our simulation focuses on the decision-making process of a single IIoT node, which acts as the learning agent. This methodological choice is deliberate, allowing us to isolate and rigorously evaluate the learning capability of the GM-DDQN algorithm itself before addressing the more complex multi-agent learning problem.

To ensure the simulation remains representative of a dense network deployment, the presence of other concurrently operating IIoT devices is modeled as a component of the ambient interference. Specifically, the interference term $I_i\left(t\right)$ in our model (Section 3.2.1) accounts not only for EMI from industrial machinery ($I_{var}\left(t\right)$ and $I_{burst}\left(t\right)$) but also for the aggregate stochastic interference generated by a population of peer devices. This approach allows us to test our agent’s robustness in a realistic interference-rich setting that reflects a multi-device reality without introducing the non-stationarity of a full multi-agent learning environment.

To assess the policy’s performance under varying channel conditions, we evaluated the agent at three representative distances from the gateway: a short distance (e.g., 20 m), a medium distance (e.g., 50 m), and a long distance (e.g., 80 m). The final results presented in this paper are averaged across these distance scenarios to ensure statistical robustness.

This simulation environment includes the following:

• Thermal Power Plant Environment Model: Incorporating electromagnetic interference patterns (baseline, variable, and burst), equipment vibration dynamics (normal, fluctuating, and anomalous), temperature variations, and their impact on path loss characteristics.
• Wireless Network Simulator: Modeling packet transmission success/failure based on SINR, channel dynamics (path loss influenced by temperature, as per Section 3.2.3), and interference effects.
• IIoT Device Energy Consumption Model: Accounting for energy consumed during transmission, reception, data processing, and different sleep modes, as detailed in Section 3.3.
• Hardware Resource Simulator: Tracking the computational (e.g., CPU cycles or equivalent time) and memory (ROM/RAM) footprint during algorithm execution on a target IIoT node.

The simulation parameters are listed in

Table 3. Simulation parameters used in the experimental evaluation.

Category	Parameter	Value
Environment	Baseline interference ($I_{base}$)	−90 dBm
	Interference burst amplitude ($I_{burst}$)	−80 to −60 dBm
	Normal vibration level ($V_{normal}$)	20 Hz
	Abnormal vibration level ($V_{anomaly}$)	60–80 Hz
	Temperature range ($T_{base}\pm \mathrm{\Delta }T$)	40 °C to 120 °C
Network	Path loss exponent ($\alpha$)	3.5
	Path loss at reference distance ($P{L}_0$)	−40 dBm
	Temperature coefficient ($\beta$)	0.002
	Noise floor ($N_0$)	−100 dBm
	Data packet size (L)	128 bytes
IIoT Device	Battery capacity ($B^{max}$)	1000 mAh
	Tx power levels ($\mathcal{P}$)	$\{0,5,10,15\}$ dBm
	Sleep mode power ($e^s$)	$\{50,5,0.1\}$ mW
	Wake-up delay ($d^s$)	$\{0,10,100\}$ ms
	Processor frequency	48 MHz
Algorithm (GM-DDQN)	Learning rate ($\alpha$)	0.001
	Discount factor ($\gamma$)	0.95
	Target network update rate ($\beta$)	0.05
	Gradient memory size (m)	8

. These values are derived from a combination of actual measurements in thermal power plants, specifications of commercial IIoT devices (e.g., for battery capacity, processor frequency, and radio characteristics), and established models from wireless communication literature.

5.1.2. Test Scenarios

We evaluate the algorithm performance across three distinct scenarios, representative of typical operational challenges in thermal power plants:

• Scenario 1: High-Interference Environment. This scenario simulates operations near high-power electrical equipment, characterized by strong electromagnetic interference. The SINR fluctuates significantly, typically between 5 and 15 dB, due to frequent and intense interference bursts, demanding robust adaptive transmission strategies from the algorithm.
• Scenario 2: Variable Vibration Monitoring. This scenario focuses on applications like turbine health monitoring, where equipment vibration patterns frequently switch between normal operational levels and abnormal states (indicative of potential faults). This requires the algorithm to dynamically adjust its data sampling (implicitly linked to data generation for transmission) and transmission policies to capture critical events while conserving energy during quiescent periods.
• Scenario 3: Temperature-Constrained Deployment. This scenario models the dual impact of high ambient temperatures on IIoT nodes: (a) temperature-dependent variations in wireless channel path loss (as described in Section 3.2.3), and (b) accelerated battery discharge rates at elevated temperatures.

For each scenario, simulations are run for an equivalent of 30 days of device operation, with key metrics recorded at 1 min intervals. This duration and granularity allow for the assessment of both short-term adaptive responses and long-term energy efficiency trends.

5.1.3. Implementation Details

The proposed GM-DDQN algorithm and baseline RL methods were implemented for simulation. For evaluating on-device performance metrics (memory and computation time), the GM-DDQN was profiled as if deployed on a target hardware platform characterized by

• Processor: ARM Cortex-M4F operating at 48 MHz.
• Memory: 32 KB RAM and 256 KB flash.
• Radio: IEEE 802.15.4-compliant [59].

Deployment optimizations for GM-DDQN included 8-bit quantization for all network parameters and the use of fixed-point arithmetic during inference. The memory optimization techniques (as discussed in Section 4.6) and computational acceleration strategies (also from Section 4.6) were factored into these profiles.

5.2. Baseline Methods for Comparison

We compare GM-DDQN against a diverse set of baseline methods:

• Fixed Policy (FP): A non-adaptive strategy using a fixed transmission power (10 dBm) and a predetermined periodic sleep schedule. This represents a basic IIoT deployment.
• Threshold-Based Adaptive (TBA): Adjusts transmission power based on observed PSR and sleep intervals based on vibration thresholds. This reflects common adaptive practices in current industrial IoT.
• Standard DDQN: A full-scale DDQN with 64 neurons per hidden layer and an experience replay buffer of 10,000 transitions, serving as a performance benchmark without strict resource constraints.
• Deep Deterministic Policy Gradient (DDPG): An advanced actor–critic RL method for continuous control problems.
• Proximal Policy Optimization (PPO): A stable policy gradient RL method widely used in various tasks.
• Q-Learning with Function Approximation (Q-FA): A lightweight RL method using linear function approximation instead of deep networks, offering low computational cost but limited representational power.

To ensure a fair comparison, all reinforcement learning methods (standard DDQN, DDPG, PPO, Q-FA, and GM-DDQN) utilize the same state space, action space, and reward function definitions, as detailed in Section 3.4. We note that a comparison with an exhaustive search algorithm, which would yield the theoretical optimal policy, was not included. An exhaustive search would require evaluating every possible sequence of actions over the simulation horizon. Given an action space of size $\left|A\right|=12$ and a long time horizon T, the computational complexity is on the order of $O\left(\right|A{|}^T)$, rendering it computationally intractable for any practical scenario. This intractability is precisely the motivation for employing intelligent approximation methods like DRL.

5.3. Evaluation Metrics

We employ a comprehensive suite of metrics, categorized as follows, to evaluate the performance of each method:

• Energy Efficiency:
  • Energy consumption per successfully transmitted packet (mJ/packet).
  • Average power consumption (mW).
  • Estimated battery lifetime (months).
  • Distribution of energy consumption across components (e.g., transmission, processing, and sleep), which will be detailed with the results where applicable.
• Communication Performance:
  • PSR (%).
  • End-to-end latency (ms).
  • Throughput (kbps).
  • Adaptability to interference bursts (e.g., time for PSR to stabilize after a burst).
• Resource Utilization (for on-device deployment profile):
  • Memory footprint (ROM and RAM in KB).
  • Computation time per decision-making step (ms).
  • Energy overhead of algorithm execution per decision (mJ/decision).
  • Initialization time (s).
• Learning Performance (for RL methods):
  • Convergence speed (number of training episodes required).
  • Cumulative reward achieved during training and evaluation.
  • Policy stability (e.g., consistency of actions in similar states, assessed qualitatively and by observing reward variance).
  • Adaptation time to significant environmental changes (recovery duration).

This multi-dimensional evaluation aims to provide a holistic view, capturing not only direct energy savings but also potential trade-offs in communication, resource usage, and learning dynamics.

5.4. Experimental Results and Analysis

The comprehensive experimental results are summarized in Figure 3, which evaluates the proposed GM-DDQN against baseline methods in terms of energy efficiency, communication performance, and learning capability.

5.4.1. Energy Efficiency Performance

The primary goal of GM-DDQN is to enhance energy efficiency. Figure 3a visually presents the estimated battery lifetime for all the evaluated methods across the three test scenarios, highlighting GM-DDQN’s effectiveness. The precise values are also listed in

Table 4. Estimated battery lifetime (months) across different scenarios.

Method	Scenario 1	Scenario 2	Scenario 3
Fixed Policy (FP)	8.2	9.1	7.5
Threshold-Based Adaptation (TBA)	10.3	11.7	9.6
Q-FA	11.8	12.5	10.2
GM-DDQN (Proposed)	14.2	15.1	13.6
Standard DDQN	14.8	15.7	14.1
DDPG	15.0	15.9	14.3
PPO	14.7	15.6	14.0

. GM-DDQN extends battery lifetime by up to 73% compared to the fixed policy (FP) and by up to 38% compared to Threshold-Based Adaptation (TBA). For an IIoT deployment with hundreds of sensors, this translates to extending the battery replacement cycle by several months, significantly lowering operational and maintenance costs.

In terms of energy consumption per successfully transmitted packet, GM-DDQN achieved an average reduction of 42% compared to FP and 27% compared to TBA across all the scenarios. When compared with other RL algorithms, GM-DDQN’s energy consumption per packet was only marginally (5–8%) higher than that of standard DDQN, DDPG, and PPO. This is a significant finding as it indicates that GM-DDQN’s lightweight design incurs minimal loss in optimization capability while drastically reducing resource requirements.

5.4.2. Communication Performance

Figure 3b illustrates the communication performance. The left panel shows the PSR for each method across the different scenarios, while the right panel compares the average PSR and average end-to-end latency. GM-DDQN consistently maintains a high PSR of 96–97% across all the scenarios. This performance is comparable to standard DDQN, DDPG, and PPO (97–98%), significantly higher than the fixed policy (88–92%), and slightly better than Threshold-Based Adaptation (94–95%). Q-FA achieves an intermediate PSR of 95–96% but exhibits greater variability.

Regarding end-to-end latency (Figure 3b, right panel), GM-DDQN achieves an average of 320 ms. This is well within the acceptable limits for most industrial monitoring applications, considerably better than the fixed policy (average 480 ms), and comparable to the other advanced RL methods, which range between 290 and 310 ms. The Q-FA method resulted in an average latency of 350 ms.

5.4.3. Resource Utilization

Table 5. Resource utilization comparison for on-device deployment profile.

Method	ROM (KB)	RAM (KB)	Comp. Time (ms)	Energy Overhead (mJ/decision)
Fixed Policy (FP)	0.5	0.1	0.02	0.001
Threshold-Based Adaptation (TBA)	1.2	0.3	0.05	0.002
Q-FA	2.3	1.2	0.8	0.04
GM-DDQN (Proposed)	4.8	2.4	12	0.6
Standard DDQN	42	520	85	4.2
DDPG	78	620	120	6.0
PPO	65	580	95	4.7

compares the on-device resource utilization metrics. GM-DDQN demonstrates a significant reduction in memory footprint (ROM and RAM) by over 99% compared to standard DDQN, DDPG, and PPO, making it viable for typical IIoT devices (e.g., 32 KB RAM). The computation time per decision is reduced from 85–120 ms for the larger models to just 12 ms for GM-DDQN (an 85–90% reduction). Consequently, the energy overhead per decision for GM-DDQN (0.6 mJ) is also substantially lower (85–90% reduction) than that of the standard deep RL methods (4.2–6.0 mJ), rendering its own operational energy cost negligible relative to the savings it provides.

5.4.4. Learning Performance

The learning performance of the RL-based methods is evaluated from two complementary perspectives, as shown in Figure 3c,d.

First, Figure 3c illustrates the raw dynamics and overall convergence behavior of the training process. The smoothed trend line shows that GM-DDQN converges in approximately 3500 episodes. While this is slower than standard DDQN (around 2200 episodes), it is still well within the practical limits for pre-deployment training. Importantly, the final cumulative reward achieved by GM-DDQN is approximately 94% of that achieved by standard DDQN, indicating that its lightweight design preserves the vast majority of the learning capability.

Second, to specifically address the stability of the learning process, a dedicated statistical analysis is presented in Figure 3d. This plot provides a clearer noise-free comparison of the algorithms’ robustness. The shaded area around each curve, representing the standard deviation across 10 independent runs, serves as a direct visual indicator of performance consistency. The narrow shaded region associated with GM-DDQN’s learning curve signifies low variance, indicating that the algorithm’s performance is highly stable and reliably repeatable across different training initializations. This degree of stability is comparable to that of the more complex standard DDQN and PPO methods, and notably superior to the wider variance observed for the simpler Q-FA method. This result validates the robustness of our proposed lightweight approach.

In tests involving sudden changes to interference patterns or vibration levels, GM-DDQN demonstrated a recovery time (adaptation to the new conditions) of 15–20 min. This compares to 10–15 min for standard DDQN and 5–10 min for DDPG/PPO. While slightly slower in adaptation, GM-DDQN’s response time is still adequate for industrial scenarios, where significant environmental shifts typically occur over longer timescales.

5.4.5. Performance Analysis in Specific Scenarios

GM-DDQN exhibited robust and adaptive performance across the diverse test scenarios:

• High-Interference Environment (Scenario 1): GM-DDQN effectively modulated its transmission power in response to dynamic interference levels. It increased power during interference peaks to maintain high PSR and reduced power during lulls to conserve energy. This outperformed the fixed policy, which wasted energy, and the Threshold-Based Adaptive strategy, which responded more slowly to sudden interference bursts.
• Variable Vibration Monitoring (Scenario 2): GM-DDQN adeptly managed sleep modes according to vibration intensity. It remained active during high vibration, transitioned to light sleep for moderate vibration, and entered deep sleep during normal (low) vibration periods. This strategy ensured the capture of critical data while maximizing energy savings, proving superior to the sub-optimal switching of the TBA strategy and the non-adaptive fixed policy.
• Temperature-Constrained Deployment (Scenario 3): GM-DDQN demonstrated the best resilience to temperature variations. Its energy consumption increased by only 32% as temperatures rose from 40 °C to 120 °C. This minimal increase, compared to the other algorithms, highlights its superior ability to learn and adapt to the complex interplay between environmental factors (like temperature-induced path loss changes) and optimal operational strategies.

5.4.6. Ablation Study and Component Analysis

To assess the individual contributions of GM-DDQN’s key components, an ablation study was performed.

Table 6. Ablation study results (averaged across scenarios).

Method Variant	Energy Cons. (mJ/packet)	PSR (%)	Comp. Time (ms)
Full GM-DDQN	0.28	96.5	12
Without Gradient Memory	0.30	95.8	15
Without Quantization	0.27	96.7	38
Single Hidden Layer	0.32	94.2	8
Without Target Network	0.35	92.3	10

summarizes the average performance across scenarios when specific components are removed or altered.

The ablation results, presented in Table 6, provide critical insights into the specific contribution of each component within the GM-DDQN framework. The following analysis delves into the underlying reasons for the observed performance changes.

• Without Gradient Memory: Removing the gradient memory and reverting to a simple single-sample update leads to a noticeable increase in energy consumption. This result validates the efficacy of our core innovation. As theorized in Section 4.7, the gradient memory provides a smoothed history-aware gradient estimate. This approach mitigates the high variance inherent in updates based on single, potentially noisy, state transitions, leading to a more stable learning process and a more consistently energy-efficient policy.
• Without Quantization: This variant reveals a crucial performance–resource trade-off. While forgoing quantization yields a marginal improvement in energy efficiency (a 3.7% reduction in mJ/packet), it causes a prohibitive 217% increase in computation time (from 12 ms to 38 ms). The performance impact of quantization is minimal because the learned Q-function is robust; the optimal actions in this problem space are determined by significant differences in Q-values, not by fine-grained distinctions that would be lost to 8-bit precision. This finding is consistent with the established literature, demonstrating that robust neural networks are highly amenable to quantization. Therefore, the immense computational gain, which is essential for on-device deployment, far outweighs the negligible loss in policy optimality.
• Single Hidden Layer: Using only a single hidden layer results in a significant 14% degradation in energy efficiency. This is attributed to insufficient representational capacity. A single-layer network struggles to model the complex non-linear relationships between the diverse state variables (e.g., SINR, vibration, and temperature) and the optimal action. The second hidden layer is crucial for creating hierarchical feature representations, allowing the model to capture the intricate trade-offs demanded by our multi-objective reward function, a task for which a flatter architecture is ill-equipped.
• Without Target Network: The removal of the target network causes the most substantial performance degradation, with a 25% decrease in energy efficiency. This instability arises because, without a stable separate target network, the learning target, $y_t$, becomes highly non-stationary. The same parameters (${\theta }_t$) are used to both estimate the current Q-value and calculate the target value, creating a destructive feedback loop that leads to severe learning instability and oscillations. The use of a slowly updated target network is therefore indispensable for decoupling the updates and ensuring stable convergence to a high-quality policy, a finding consistent with the foundational principles of deep Q-learning.

In summary, this in-depth analysis confirms that each component of GM-DDQN is a deliberate design choice. The framework’s success stems from the synergistic interplay between the core innovation of gradient memory and carefully selected lightweighting techniques, which collectively achieve a robust balance between high performance and resource efficiency.

6. Discussion

6.1. Summary of Experimental Results and Their Significance

The experimental findings validate the exceptional performance of the GM-DDQN framework in optimizing energy efficiency for industrial IoT devices within thermal power plants, underscoring its substantial potential for real-world deployment. Across three distinct testing scenarios—high-interference environments, vibration-monitoring fluctuations, and high-temperature constrained deployments—GM-DDQN demonstrates notable energy efficiency gains. Specifically, it reduces per-packet transmission energy consumption by 42% compared to the fixed policy and 27% compared to the threshold-based adaptive strategy. When benchmarked against advanced reinforcement learning methods like standard DDQN, DDPG, and PPO, GM-DDQN achieves comparable energy efficiency (differing by only 5–8%) while drastically lowering computational resource demands. This reflects the effectiveness of its lightweight design in maintaining optimization capabilities with minimal compromise.

Battery life evaluations further reveal that GM-DDQN extends device lifespan by up to 73% over the fixed policy and 38% over the threshold-based adaptive approach. For power plants with hundreds of sensors, this translates to significant cost savings by extending maintenance cycles by months. Moreover, GM-DDQN sustains a high PSR of 96–97%, closely aligning with the 97–98% achieved by other reinforcement learning methods, and far surpassing the fixed policy’s 88–92% and the threshold-based adaptive’s 94–95%. This reliability is achieved while maintaining an average end-to-end latency of 320 milliseconds, which meets typical industrial monitoring requirements.

In terms of learning performance, GM-DDQN converges after approximately 3500 episodes. While this is slower than standard DDQN’s 2200 episodes, its final cumulative reward reaches 94% of that achieved by standard DDQN. This affirms that the lightweight design has a negligible impact on learning capability, positioning GM-DDQN as a practical solution for resource-constrained environments.

6.2. Research Contributions

This study’s contributions, validated through rigorous experimentation, offer significant theoretical and practical advancements.

First, the core innovation of the gradient memory mechanism was empirically proven to be a viable memory-efficient alternative to experience replay for on-device learning. As quantified in Table 5, this mechanism, combined with a compact network architecture and quantization, reduces the total on-device memory footprint by approximately 99% compared to standard DDQN, making deployment on resource-constrained IIoT devices (32 KB RAM) feasible. The ablation study (Table 6) further confirmed its importance, showing that its removal led to noticeable degradation in energy efficiency, thus validating its role in stabilizing the learning process.

Second, the holistic lightweight design of the GM-DDQN framework demonstrated that substantial resource reduction does not necessitate a major compromise in performance. Figure 3d shows that GM-DDQN achieves a final cumulative reward that is 94% that of the full standard DDQN, with comparable learning stability. This addresses a critical trade-off in edge intelligence: achieving high performance with minimal overhead.

Third, the practical value of GM-DDQN was substantiated through scenario-based evaluations, which showed significant improvements in operational metrics. The joint optimization of the power and sleep modes extended battery lifetime by up to 73% compared to fixed policies (Figure 3a and Table 4), translating to a tangible reduction in maintenance costs from an 8–9 month cycle to a 14–15 month cycle. This was achieved while maintaining a high PSR of 96–97% (Figure 3b), demonstrating the framework’s ability to balance energy savings with communication reliability, a key requirement for industrial applications. The adaptive strategies learned by the agent proved effective across diverse challenges, from high-interference environments (Scenario 1) to variable-load conditions (Scenario 2), as detailed in Section 5.4.5.

Collectively, these validated contributions provide a robust and practical solution for deploying edge intelligence in demanding industrial environments, offering a clear pathway to more efficient, reliable, and intelligent monitoring systems.

6.3. Limitations and Future Work

Despite the promising outcomes, this study has several limitations that, when properly contextualized, define a clear and logical roadmap for future research.

First, the validation of our framework relies on simulation. A potential simulation-to-reality gap exists, a crucial point concerning real-world applicability. While our simulation platform was designed to be highly realistic (as detailed in Section 5.1.1), real-world deployments may present unmodeled challenges, such as complex multipath fading or specific RF interference signatures. To bridge this gap, a crucial next step involves validating the simulation model against data collected from a real-world testbed. Our ongoing efforts are focused on deploying a small number of sensor nodes within an operational power plant to gather baseline data, which will be used to calibrate and refine our simulation parameters, thereby enhancing its fidelity.

Second, and intrinsically linked to our research methodology, is the use of a single-agent learning framework. This was a deliberate methodological choice designed to isolate and rigorously evaluate the primary contribution of this paper: the novel GM-DDQN algorithm and its core gradient memory mechanism. By simplifying the environment to a single agent, we could analyze the algorithm’s performance without the confounding variables of multi-agent dynamics, such as non-stationarity and credit assignment. As justified in Section 5.1.1, the influence of other nodes was modeled as realistic environmental interference, ensuring the robustness of our evaluation. Having now established the efficacy of the core algorithm in this controlled setting, the clear and vital future direction is to extend this framework to a multi-agent reinforcement learning (MARL) setting to optimize the network-wide performance.

Third, following these steps, the subsequent phase would be empirical testing on physical IIoT hardware. Migrating the validated agent to an embedded system like an ARM Cortex-M4 introduces practical challenges, including RTOS task scheduling, precise energy accounting for memory access, and handling sensor noise. Overcoming these hardware-specific issues is the final step to fully substantiate the framework’s on-device viability.

Finally, other areas for future work include developing mechanisms for online or continual learning to adapt to entirely new scenarios, investigating the framework’s security vulnerabilities under adversarial conditions, and incorporating automated hyperparameter tuning methods to further enhance the practical deployment of GM-DDQN.

7. Conclusions

The proposed GM-DDQN framework effectively optimizes energy efficiency in industrial IoT devices, particularly for applications within thermal power plants. Experimental validation demonstrates its capability to achieve significant energy savings, leading to extended battery life, while maintaining reliable communication performance with minimal computational demands. These attributes make GM-DDQN a promising solution for resource-constrained IIoT deployments. Future work should focus on enhancing its convergence speed, further improving its adaptability to real-world dynamic conditions through mechanisms for online learning or transfer learning, and rigorously testing its performance and robustness in operational environments. Addressing these aspects will pave the way for broader practical application of lightweight deep reinforcement learning in creating sustainable and intelligent industrial systems.