# A Cluster-Based Optimal Computation Offloading Decision Mechanism Using RL in the IIoT Field

^{*}

*Applied Sciences*: Invited Papers in Computing and Artificial Intelligence Section)

## Abstract

**:**

## 1. Introduction

## 2. System Model

## 3. Problem Definition and Proposed Algorithm

**Agent**: The CH($c{h}_{i}$), $\forall c{h}_{i}\in \left\{c{h}_{1},\dots ,c{h}_{n}\right\}$**State**: ${s}^{i}\left(t\right)=\left({c}_{d}\left(t\right),{q}_{d}\left(t\right),{q}_{m}\left(t\right),{q}_{j}\left(t\right),{l}_{d}\left(t\right),{l}_{m}\left(t\right),{l}_{j}\left(t\right)\right),$ being the state of the created task at time slot t in cluster i.-${c}_{d}\left(t\right)$: the task created by device d at time slot t, $\forall d\in D$-${q}_{d}\left(t\right),{q}_{m}\left(t\right),{q}_{j}\left(t\right)$: the load of device d, serving MECS m, and serving D2D device j at time slot t, $\forall d,j\in D$, $\forall m\in M$-${l}_{d}\left(t\right),{l}_{m}\left(t\right),{l}_{j}\left(t\right)$: the location of device d, serving MECS m, and serving D2D device j at time slot t, $\forall d,j\in D$, $\forall m\in M$**Action**: ${a}^{i}\left(t\right)\in \left\{0,1,2\right\},$ being the offloading strategy of a task at time t in cluster i.**Reward (Penalty)**: $R\left({s}^{i}\left(t\right),{a}^{i}\left(t\right)\right)=1/\left\{\left(1-\alpha \right)\xb7{E}_{nor}\left(t\right)+\alpha \xb7{L}_{nor}\left(t\right)\right\},$ where $\alpha $ is the weighting factor between 0 and 1. ${E}_{nor}\left(t\right)$ indicates a normalized value of the total of computing and transmission energy consumed by the device when executing the task in a time slot t. ${L}_{nor}\left(t\right)$ denotes the normalized value of the execution delay of the task from the time slot t where it occurs until the job execution is completed.

## 4. Numerical Results

## 5. Conclusions

## Author Contributions

## Funding

## Institutional Review Board Statement

## Informed Consent Statement

## Data Availability Statement

## Conflicts of Interest

## References

- Sisinni, E.; Saifullah, A.; Han, S.; Jennehag, U.; Gidlund, M. Industrial Internet of Things: Challenges, Opportunities, and Directions. IEEE Trans. Ind. Inform.
**2018**, 14, 4724–4734. [Google Scholar] [CrossRef] - Sun, W.; Liu, J.; Yue, Y. AI-Enhanced Offloading in Edge Computing: When Machine Learning Meets Industrial IoT. IEEE Netw.
**2019**, 33, 68–74. [Google Scholar] [CrossRef] - Li, X.; Wan, J.; Dai, H.N.; Imran, M.; Xia, M.; Celesti, A. A Hybrid Computing Solution and Resource Scheduling Strategy for Edge Computing in Smart Manufacturing. IEEE Trans. Ind. Inform.
**2019**, 15, 4225–4234. [Google Scholar] [CrossRef] - Lin, C.; Deng, D.; Chih, Y.; Chiu, H. Smart Manufacturing Scheduling with Edge Computing using Multiclass Deep Q Network. IEEE Trans. Ind. Inform.
**2019**, 15, 4276–4284. [Google Scholar] [CrossRef] - Mao, Y.; You, C.; Zhang, J.; Huang, K.; Letaief, K.B. A Survey on Mobile Edge Computing: The Communication Perspective. IEEE Commun. Surv. Tutor.
**2017**, 19, 2322–2358. [Google Scholar] [CrossRef] [Green Version] - Hong, Z.; Chen, W.; Huang, H.; Guo, S.; Zheng, Z. Multi-Hop Cooperative Computation Offloading for Industrial IoT–Edge–Cloud Computing Environments. IEEE Trans. Parallel Distrib. Syst.
**2019**, 30, 2759–2774. [Google Scholar] [CrossRef] - Xie, J.; Jia, Y.; Chen, Z.; Nan, Z.; Liang, L. D2D Computation Offloading Optimization for Precedence-Constrained Tasks in Information-Centric IoT. IEEE Access
**2019**, 7, 94888–94898. [Google Scholar] [CrossRef] - Mehrabi, M.; You, D.; Latzko, V.; Salah, H.; Reisslein, M.; Fitzek, F.H.P. Device-Enhanced MEC: Multi-Access Edge Computing (MEC) Aided by End Device Computation and Caching: A Survey. IEEE Access
**2019**, 7, 166079–166108. [Google Scholar] [CrossRef] - Zhi, L.; Zhu, Q. Genetic Algorithm-Based Optimization of Offloading and Resource Allocation in Mobile-Edge Computing. Information
**2020**, 11, 83. [Google Scholar] [CrossRef] [Green Version] - Yang, G.; Hou, L.; He, X.; He, D.; Chan, S.; Guizani, M. Offloading Time Optimization via Markov Decision Process in Mobile-Edge Computing. IEEE Internet Things J.
**2021**, 8, 2483–2493. [Google Scholar] [CrossRef] - Yang, Y.; Long, C.; Wu, J.; Peng, S.; Li, B. D2D-Enabled Mobile-Edge Computation Offloading for Multiuser IoT Network. IEEE Internet Things J.
**2021**, 8, 12490–12504. [Google Scholar] [CrossRef] - Hossain, M.S.; Nwakanma, C.I.; Lee, J.M.; Kim, D.S. Edge Computational Task Offloading Scheme using Reinforcement Learning for IIoT Scenario. ICT Express
**2020**, 6, 291–299. [Google Scholar] [CrossRef] - Liu, H.; Cao, L.; Pei, T.; Deng, Q.; Zhu, J. A Fast Algorithm for Energy-saving Offloading with Reliability and Latency Requirements in Multi-Access Edge Computing. IEEE Access
**2020**, 8, 151–161. [Google Scholar] [CrossRef] - Wang, D.; Tian, X.; Cui, H.; Liu, Z. Reinforcement Learning-based Joint Task Offloading and Migration Schemes Optimization in Mobility-aware MEC Network. China Commun.
**2020**, 17, 31–44. [Google Scholar] [CrossRef] - Yu, B.; Zhang, X.; You, I.; Khan, U.S. Efficient Computation Offloading in Edge Computing Enabled Smart Home. IEEE Access
**2021**, 9, 48631–48639. [Google Scholar] [CrossRef] - Fan, Y.; Zhai, L.; Wang, H. Cost-Efficient Dependent Task Offloading for Multiusers. IEEE Access
**2019**, 7, 115843–115856. [Google Scholar] [CrossRef] - Qian, Y.; Wu, J.; Wang, R.; Zhu, F.; Zhang, W. Survey on Reinforcement Learning Applications in Communication Networks. J. Commun. Inf. Netw.
**2019**, 4, 30–39. [Google Scholar] - Liao, Z.; Peng, J.; Xiong, B.; Huang, J. Adaptive Offloading in Mobile-Edge Computing for Ultra-dense Cellular Networks based on Genetic Algorithm. J. Cloud Comput.
**2021**, 10, 15. [Google Scholar] [CrossRef] - Hu, G.; Jia, Y.; Chen, Z. Multi-User Computation Offloading with D2D for Mobile Edge Computing. In Proceedings of the 2018 IEEE Global Communications Conference (GLOBECOM), Abu Dhabi, United Arab Emirates, 9–13 December 2018. [Google Scholar]

**Figure 2.**Performance comparison according to task arrival rate per device and the system architecture: (

**a**) task blocking rate; (

**b**) task completion rate within delay constraints; (

**c**) total energy consumption (J); (

**d**) throughput.

**Figure 3.**Performance comparison according to task arrival rate per device and the method of selecting the MECS and D2D device and determining offloading strategy decision: (

**a**) task blocking rate; (

**b**) task completion rate within delay constraints; (

**c**) total energy consumption (J); (

**d**) throughput.

**Figure 4.**Performance comparison according to cluster types when using different target MECS and D2D device selection methods with the same optimal offloading strategy (the number of devices is 52 and task arrival rate is 0.8): (

**a**) task blocking rate; (

**b**) task completion rate within delay constraints; (

**c**) total energy consumption(J); (

**d**) throughput.

**Figure 5.**Performance comparison according to cluster types when using same-target MECS and D2D device selection methods with the different optimal offloading strategy (the number of devices is 52 and task arrival rate is 0.8): (

**a**) task blocking rate; (

**b**) task completion rate within delay constraints; (

**c**) total energy consumption(J); (

**d**) throughput.

Parameter | Value |
---|---|

coverage of BS | 150 m [18] |

${B}^{cell}\left(t\right)$, ${B}^{d2d}\left(t\right)$ | 10 MHz |

${\sigma}^{2}$ | ${10}^{-10}$ |

${p}_{d,m}^{cell}\left(t\right),{p}_{i,j}^{d2d}\left(t\right)$ | 0.5 W |

${v}_{d}\left(t\right)$ | {600, 800, 1000} Kbits |

${w}_{d}\left(t\right)$ | 1000 cycles/bit |

${f}_{d}$ | 2 GHz |

${f}_{m}$ | 5 GHz |

$\epsilon $ | ${10}^{-27}$ |

$\alpha $ | 0.7 |

time slot duration | 100 ms |

${L}_{req}$ | 80 ms |

$\delta $ | 0.5 |

${q}_{d}^{max}$ | 3 |

${q}_{m}^{max}$ | 5 |

${h}_{d,m}^{cell}\left(t\right)$ | $148.1+40\ast {\mathrm{log}}_{10}\mathrm{distance}\left(\mathrm{km}\right)$ [19] |

${h}_{i,j}^{d2d}(t$) | $128.1+37.6\ast {\mathrm{log}}_{10}\mathrm{distance}\left(\mathrm{km}\right)$ [19] |

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |

© 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Koo, S.; Lim, Y.
A Cluster-Based Optimal Computation Offloading Decision Mechanism Using RL in the IIoT Field. *Appl. Sci.* **2022**, *12*, 384.
https://doi.org/10.3390/app12010384

**AMA Style**

Koo S, Lim Y.
A Cluster-Based Optimal Computation Offloading Decision Mechanism Using RL in the IIoT Field. *Applied Sciences*. 2022; 12(1):384.
https://doi.org/10.3390/app12010384

**Chicago/Turabian Style**

Koo, Seolwon, and Yujin Lim.
2022. "A Cluster-Based Optimal Computation Offloading Decision Mechanism Using RL in the IIoT Field" *Applied Sciences* 12, no. 1: 384.
https://doi.org/10.3390/app12010384