Backpack Client Selection Keeping Swarm Learning in Industrial Digital Twins for Wireless Mapping

Abstract

Digital twin virtual–real mapping and precise modeling require the synchronization of large amounts of data, which leads to high communication overhead in wireless channels in industrial Internet of Things (IoT). To solve this problem, this study proposes an architecture of Digital Twin–Swarm learning (DT-SL) for industrial IoT digital twins. SL is an emerging distributed federated learning (FL) method that eliminates the need for centralized servers completely. However, it faces the problem of wireless channel congestion caused by high concurrent parameter transmission. In view of the above architecture, a novel KSL scheme based on the backpack model is used to construct the DT model. The backpack optimization problem is used to select the client with the largest contribution to participate in Keeping SL twin modeling. In addition, the experimental results evaluated the performance of the proposed method. The absolute value of the client’s updated parameter quantity decreased by 23.6% on average. The convergence rate of the aggregation model increased by 34.1%, and the model aggregation MSE value decreased to 0.03.

1. Introduction

This is regarded as a revolutionary architecture to improve production efficiency in the information age in the industrial IoT, especially in the context of Industry 4.0 [1]. Factory production edge deployment includes DT with wireless mobile edge computing capabilities, which is an edge-side technology [2,3]. DT effectively connects the device layer and the network layer. It collects operational and environmental data and adapts the state control of physical entities, which provides an excellent solution for the intelligent allocation of industrial production line resources [3,4,5].

DT industrial IoT relies heavily on large amounts of multisource heterogeneous data collected from different swarms of devices, thus forming data silos. The secure sharing of data across multiple production lines and logistics channels is another key challenge of DT industrial IoT [6,7]. To solve these problems, SL came into being, which is a fully decentralized FL method that achieves a perfect balance between massive data and privacy in industrial IoT. SL was originally proposed to integrate the privacy of medical data. It is completely independent of central server coordination and satisfies the requirement that data owners do not provide original data for model training [8,9]. The trained model parameters are shared on the wireless swarm network and aggregated into the global model. The aggregated model is sent to the local device to continue participating in the next training stage until the model converges [10].

SL is a special type of FL that uses high-concurrency transmission of model parameters on a recurrent network. The transmission reduces the timeliness of communication in models with high real-time requirements, such as DT, and increases wireless mapping errors [11]. Much scholars’ research on FL is limited to a simplified wireless communication channel model. The gradient parameter transmission is secure but faces the problem of high communication overhead [12,13]. Some scholars have proposed gradient sparsification and quantification methods; however, for the DT industrial IoT architecture, it is difficult to adapt to the needs of real-time and accurate mapping [14,15]. Meanwhile, there is currently little research on the combination of SL and industrial IoT. Moreover, the original SL scheme does not implement the screening of client devices.

The backpack model is a classic combinatorial optimization method that is often used to select a group of objects with the greatest benefits under resource constraints (such as communication bandwidth, energy consumption, etc.). In the industrial IoT scenarios, physical devices are considered items. Their contribution represents value. The communication volume required for parameter upload represents weight, and the communication capacity is set as backpack capacity. By solving the backpack optimization problem, devices with the greatest contribution can be selected to participate in the Keeping SL twin modeling and update parameters without exceeding the communication threshold.

Therefore, in response to the above-mentioned wireless mapping timeliness problem of industrial DT, the contributions of this article include the following:

(1)

We designed a DT-driven industrial IoT SL architecture, in which DT technology realizes dynamic perception in complex industrial IoT environments, and SL enhances the interconnection between heterogeneous devices and production lines.

(2)

Based on DT-SL, we propose a novel KSL scheme based on the backpack model to construct the DT model. Moreover, we select clients with greater contributions to complete wireless mapping and Keeping SL twin modeling to optimize the wireless channel transmission rate.

(3)

Compared with the existing methods, the timeliness and convergence rate of the proposed method are improved.

2. Related Work

2.1. Digital Twins in Industrial IoT

Digital twins are integrated with real-time industrial IoT data to establish comprehensive mappings of equipment and production lines for multisource scenarios, through which industrial production efficiency is enhanced. In [16], a multi-tier industrial DT architecture was constructed, comprising a client layer, an edge layer, and a twin layer. The edge servers continuously update the twin states, including the operational status, historical data, and behavioral models, which improves the execution accuracy of industrial production. An industrial IoT architecture that combines federated learning with DT was proposed in [17], where the twin layer was implemented through lightweight edge servers. The local update coefficients and model compression ratios were adaptively adjusted to achieve real-time precise mapping. Developed on the multi-tier industrial DT architecture in [18], client interaction statuses were analytically modeled, and an efficient communication method for multi-device distributed computing was introduced. Moreover, reference [19] enhanced partial model updating efficiency by incorporating DAG-based block sharding technology into industrial DT architectures, through which edge-enabled industrial twin models were established. Building on equipment operational parameters and interaction states, an asynchronous weighted strategy for global model updating was presented in [20] to optimize resource allocation. Additionally, a geometry–physics behavior-rule multidimensional model fusion methodology was adopted in [2] for developing high-fidelity virtual workshop models, enabling the effective deployment of DT technologies in industrial IoT scenarios.

2.2. Client Selection in Distributed Learning

The computational capabilities of heterogeneous devices in industrial IoT networks are constrained by their resource disparities. Devices with low computing efficiency prolong global model training cycles, thereby degrading convergence rates. To address this issue, a client selection algorithm constrained by time thresholds is proposed in [21], where participating clients are selected through evaluating heterogeneous training durations. In [22], a stratified sampling strategy is introduced to select quasi-static clients within uniform time intervals, minimizing client dropout and communication cycles. Building on this framework, Ref. [23] categorizes clients into hierarchical tiers based on training time windows, ensuring in-tier selections while preventing cross-layer synchronization delays. A dynamic adaptive scheduling method is developed in [24], which determines optimal per-round time cutoffs by analyzing historical convergence patterns to identify the most suitable clients in distributed networks. Similarly, Ref. [25] enhances client selection by prioritizing devices with high-quality data weights within predefined baselines. However, this method creates imbalanced selection criteria, resulting in suboptimal gradient aggregation and coarse-grained model accuracy.

Ref. [26] introduces a client selection strategy that minimizes variance across time scales, prioritizing clients with high availability for training contrasting time-based methods. Ref. [27] employs a chunked incremental learning method to precisely schedule and match client training tasks based on gradient aggregation patterns. Ref. [28] proposes a client selection algorithm using model-parameter-sharing block fading values and local gradient norms, combined with resource allocation, to efficiently schedule low-configuration devices. Ref. [29] develops a balanced scheduling strategy for clients operating in both high- and low-SNR environments, ensuring the usability of the selected devices. Ref. [30] optimizes bandwidth allocation for model parameter transmission through energy-efficient client selection, enabling precise multidimensional client selection.

These methods rely on device availability, computational resources, and energy assessments for client selection. However, they crudely estimate device contributions to model convergence and fail to address redundant data in selected clients, leading to inefficient high-volume parameter transmission without significant accuracy improvements.

3. System Model

There is a typical industrial IoT production environment that includes equipment clusters such as production lines, packaging lines, and logistics lines. A large amount of heterogeneous mechanical equipment and multi-type sensors (temperature, pressure, speed, etc.) in each industrial production line regularly collect operating data perception information, as shown in Figure 1.

A DT-driven industrial IoT SL architecture is designed to realize dynamic perception in complex industrial IoT environments. Based on DT-SL, we propose a novel KSL-scheme-based backpack model to construct the DT model. We select clients with greater contributions to complete wireless mapping and Keeping SL twin modeling to optimize the wireless channel transmission rate.

We provide the meaning and source of the key variables in the article formula, as shown in

Table 1. Key variables description.

Variables	Meaning	Source (Equations)
$L_i({\theta }_i)$	the loss function of device $i$	(1), (3), (12)–(16)
$n_i$	number of local dataset samples of device $i$	(1), (2)
${\theta }_i^{(t)}$	model parameters of device $i$ at iteration $t$	(3), (14)–(18), (22), (23)
$\nabla L_i({\theta }_i^{(t)})$	the gradient of loss function of device $i$ at the $t$th iteration	(3)–(5)
$w_i$	the communication volume (i.e., weight) of device $i$	(11), (19)–(22)
$B_{\max }$	the system’s maximum communication overhead threshold	(7), (9), (10)
$\mu$	strong convexity constant	(13), (15), (17), (18), (22), (23)
${\sigma }^2$	variance of local data distribution	(21)–(23)

. We assume that each device $i$ performs model training on the local dataset $D_i={\{(x_j,y_j)\}}_{j=1}^{n_i}$. The training goal is to minimize the local loss function and optimize the model parameters of a single device. For the regression problem of device clusters, the loss function of a single device is defined as follows:

$$L_i({\theta }_i)={\displaystyle \frac{1}{n_i}}{\displaystyle \sum _{j=1}^{n_i}(f(}{x}_j;{\theta }_i)-y_j{)}^2$$

(1)

where $f(x_j;{\theta }_i)$ is the classification prediction of device $i$ for input sample $x_j$, $y_j$ belongs to the true sample label, and $n_i$ is the number of samples in device $i$.

$$L_i({\theta }_i)=-{\displaystyle \frac{1}{n_i}}{\displaystyle \sum _{j=1}^{n_i}{\displaystyle \sum _{k=1}^C{y}_{jk}}}\log (p_{jk})$$

(2)

where $p_{jk}$ is the predicted probability of a single device $i$ for sample $j$ belonging to category $k$; $C$ is the number of categories in the device production line; and $y_{jk}$ is the label of sample $j$ corresponding to category $k$.

In SL, the gradient descent algorithm is used for optimization. A single device $i$ updates the model parameter ${\theta }_i$, and the formula is expressed as:

$${\theta }_i^{(t+1)}={\theta }_i^{(t)}-\eta \nabla L_i({\theta }_i^{(t)})$$

(3)

where $\eta$ is the learning rate and $\nabla L_i({\theta }_i^{(t)})$ is the gradient of the loss function for device $i$ at iteration $t$. It is a discrete optimization formula that uses the iteration as the update unit. For the mean squared error loss function, the gradient is calculated as:

$$\nabla L_i({\theta }_i)={\displaystyle \frac{2}{n_i}}{\displaystyle \sum _{j=1}^{n_i}(f(}{x}_j;{\theta }_i)-y_j)\nabla f(x_j;{\theta }_i)$$

(4)

where $\nabla f(x_j;{\theta }_i)$ is the gradient of model $f(x_j;{\theta }_i)$ with respect to parameter ${\theta }_i$. The gradient of the cross-entropy loss function is calculated as:

$$\nabla L_i({\theta }_i)=-{\displaystyle \frac{1}{n_i}}{\displaystyle \sum _{j=1}^{n_i}{\displaystyle \sum _{k=1}^C\left(y_{jk}-p_{jk}\right)}}\nabla p_{jk}$$

(5)

where $\nabla p_{jk}$ is the gradient of the model output probability $p_{jk}$, with respect to the model parameter ${\theta }_i$.

4. Client Selection and Backpack Model

In industrial scenarios, limited computing resources select a swarm of devices $S\subseteq \{1,2,\dots ,N\}$, so that the total contribution of the devices is maximized. The total interactive communication parameter of the model is less than or equal to the set maximum threshold $B_{\max }$. The objective function and constraints of the backpack model are expressed as:

$$\mathrm{maximize}\hspace{1em}{\displaystyle \sum _{i=1}^N{c}_i}{x}_i$$

(6)

$${\displaystyle \sum _{i=1}^N{b}_i}{x}_i\le B_{\max }$$

(7)

where $N$ is the total number of devices participating in SL in the industrial production line $S$. The modeling of device $i$ in the swarm network is represented by 1, and the non-modeling is represented by 0, $x_i\in \{0,1\}$. $c_i$ is the contribution of device $i$, and $b_i$ is the parameter quantity of the model uploaded by the device. The model gradually calculates the optimal solution under different communication cost constraints. We define $DP[k]$ as the dynamic programming state, where each entry represents the maximum contribution achievable under a communication budget range $[0,k]$. The state transfer equation is expressed as:

$$DP [k]=\max (DP [k],DP [k-b_i]+c_i) s.t. k\ge b_i$$

(8)

We performed initial calculations on the backpack model. We assume that no equipment in the production line is selected to participate in wireless interaction and build a twin model, where all $DP[k]=0$, as follows:

$$DP [0]=0,\forall k\in [0,B_{\max }]$$

(9)

where, based on the contribution of industrial production equipment $c_i$ and the parameter update transmission volume $b_i$, the optimal solution for calculating the capacity of a single backpack $k$ is:

$$DP [B_{\max }]=\max (DP [B_{\max }], DP [k-b_i] + c_i)$$

(10)

The maximum contribution can be obtained by selecting the optimal solution (backpack capacity) of the client’s device to satisfy $0\le k\le B_{\max }$. When the selected device completes the local model training and updates the client, the device will upload the updated model parameters ${\theta }_i^{(t+1)}$ to the SL network architecture. The weighted average update of the SL network node model is:

$${\theta }^{(t+1)}={\displaystyle \sum _{i=1}^N{w}_i{{\theta }_i}^{(t+1)}}$$

(11)

where ${\theta }^{(t+1)}$ is the global model generated by aggregating the local model updates of all selected clients; $w_i$ is the weight of device $i$; and ${{\theta }_i}^{(t+1)}$ is the model parameter of device $i$ after local training in the $t+1$th round. The threshold is set according to the computing power of the device.

4.1. Convergence Proof

For the local training of each device, the goal is to minimize the loss function $L_i({\theta }_i)$. We assume that $L_i({\theta }_i)$ is a continuous differentiable convex function that satisfies smoothness. Smoothness guarantees stable gradient updates. Convexity ensures the existence of a unique global optimum and allows for the derivation of linear convergence rates under appropriate learning rates, as follows:

$$\Vert \nabla L_i({\theta }_1)-\nabla L_i({\theta }_2)\Vert \le \mathcal{L}\Vert {\theta }_1-{\theta }_2\Vert ,\hspace{1em}\forall {\theta }_1,{\theta }_2$$

(12)

where Formula (12) indicates that the gradient of the loss function is Lipschitz continuous. Constant $\mathcal{L}$ is the Lipschitz constant. The loss function $L_i({\theta }_i)$ has strong convexity. There exists a constant $\mu$, such that:

$$L_i({\theta }_2)\ge L_i({\theta }_1)+\nabla L_i{({\theta }_1)}^{\top }({\theta }_2-{\theta }_1)+{\displaystyle \frac{\mu }{2}}\Vert {\theta }_2-{\theta }_1{\Vert }^2,\forall {\theta }_1,{\theta }_2$$

(13)

where $\mu$ is a strong convexity constant, which limits the curvature of the loss function. We let the local model parameter be ${\theta }_i^{(t)}$. The global optimal solution is ${\theta }_i^{*}$ and the error is ${\theta }_i^{(t)}-{\theta }_i^{*}$. Based on Formulae (12) and (13), we update Formula (3). We expand the square of the error as follows:

$$\Vert {{\theta }_i}^{(t+1)}-{{\theta }_i}^{*}{\Vert }^2=\Vert {{\theta }_i}^{(t)}-{{\theta }_i}^{*}{\Vert }^2-2\eta \nabla L_i{({\theta }_i^{(t)})}^{\top }\Vert {{\theta }_i}^{(t)}-{{\theta }_i}^{*}{\Vert }^2+{\eta }^2\Vert \nabla L_i({\theta }_i^{(t)}){\Vert }^2$$

(14)

The algorithm applies the strong convexity shown in Formula (13) and the Lipschitz continuity in Formula (12), and ${\theta }_1={\theta }_i^{*},\hspace{0.17em}{\theta }_2={\theta }_i^{(t)}$. Because $\nabla L_i({\theta }_i^{*})=0$, we can get the following two inequalities:

$$\nabla L_i{({{\theta }_i}^{(t)}-{{\theta }_i}^{*})}^{\top }({{\theta }_i}^{(t)}-{{\theta }_i}^{*})\ge \mu \Vert {{\theta }_i}^{(t)}-{{\theta }_i}^{*}{\Vert }^2$$

(15)

$$\Vert \nabla L_i({\theta }_i^{(t)}){\Vert }^2\le {\mathcal{L}}^2\Vert {{\theta }_i}^{(t)}-{{\theta }_i}^{*}{\Vert }^2$$

(16)

We substitute the above two inequalities into the square error expression (14) as follows:

$$\Vert {{\theta }_i}^{(t+1)}-{{\theta }_i}^{*}{\Vert }^2\le \left(1-2\eta \mu +{\eta }^2{\mathcal{L}}^2\right)\Vert {{\theta }_i}^{(t)}-{{\theta }_i}^{*}{\Vert }^2$$

(17)

We choose a learning rate $0<\eta \le {\displaystyle \frac{\mu }{{\mathcal{L}}^2}}$. Moreover, because $1-2\eta \mu +{\eta }^2{\mathcal{L}}^2\le 1-\eta \mu$, we know that the gradient descent error satisfies:

$$\Vert {\theta }_i^{(t+1)}-{\theta }_i^{*}{\Vert }^2\le (1-\eta \mu )\Vert {\theta }_i^{(t)}-{\theta }_i^{*}{\Vert }^2$$

(18)

where $1-\eta \mu <1$ ensures that the model error gradually decreases and has linear convergence. The convergence of the SL network node model satisfies:

$${\theta }^{(t+1)}={\displaystyle \sum _{i\in S}{w}_i}{\theta }_i^{(t+1)}$$

(19)

In the device set selected by Formula (10), the model update of each device satisfies the local convergence assumption. We let the global optimal solution be ${\theta }^{*}$ and the global error be ${\theta }^{(t)}-{\theta }^{*}$. We expand the global error and take the square of the norm as follows:

$$\Vert {\theta }^{(t+1)}-{\theta }^{*}{\Vert }^2={\Vert {\displaystyle \sum _{i\in S}{w}_i}\left({\theta }_i^{(t+1)}-{\theta }^{*}\right)\Vert }^2$$

(20)

The algorithm uses Cauchy–Schwarz inequality. Assuming that the data between the devices are independent, the covariance term approaches zero:

$$\left[\begin{array}{l}\Vert {\theta }^{(t+1)}-{\theta }^{*}{\Vert }^2\le {\displaystyle \sum _{i\in S}{w}_i^2}\Vert {\theta }_i^{(t+1)}-{\theta }^{*}{\Vert }^2+{\displaystyle \sum _{i\ne j}{w}_i}{w}_j{({\theta }_i^{(t+1)}-{\theta }^{*})}^{\top }({\theta }_j^{(t+1)}-{\theta }^{*})\hfill \\ \Vert {\theta }^{(t+1)}-{\theta }^{*}{\Vert }^2\le {\displaystyle \sum _{i\in S}{w}_i^2}\Vert {\theta }_i^{(t+1)}-{\theta }^{*}{\Vert }^2+{\sigma }^2\hfill \end{array}\right.$$

(21)

We substitute Formula (14) into the above formula and get:

$$\Vert {\theta }^{(t+1)}-{\theta }^{*}{\Vert }^2\le (1-\eta \mu ){\displaystyle \sum _{i\in S}{w}_i^2}\Vert {\theta }_i^{(t)}-{\theta }^{*}{\Vert }^2+{\sigma }^2$$

(22)

Due to ${\displaystyle \sum _{i\in S}{w}_i}=1$, $w_i\le {\displaystyle \frac{1}{\left|S\right|}}$, and ${\displaystyle \sum _{i\in S}{w}_i^2}\le {\displaystyle \frac{1}{\left|S\right|}}$, we perform variance normalization and variance simplification, as follows:

$${\Vert {\theta }^{(t+1)}-{\theta }^{*}\Vert }^2\le (1-\eta \mu ){\Vert {\theta }^{(t)}-{\theta }^{*}\Vert }^2+{\displaystyle \frac{{\sigma }^2}{\left|S\right|}}$$

(23)

where ${\sigma }^2$ is the local data distribution variance. It extends the local convergence of Equation (14) to the global model, which shows the influence of the variance of the SL network nodes and ${\sigma }^2$ on the convergence speed. The more nodes in the SL network, the faster the convergence rate of the global model.

4.2. Computational Complexity

The communication parameter quantity constraint for the SL network training is $k=0,1,\dots ,B_{\max }$. $N$ devices update $DP[k]$. The time complexity of dynamic programming is calculated as $O(N·B_{\max })$. The space complexity calculation is for the $B_{\max }$ device to store an array $DP:O(B_{\max })$. Meanwhile, the process of dynamically selecting a subset of device clients that meet the constraints to participate in training to improve communication efficiency is shown in Algorithm 1.

Algorithm 1: Backpack-Swarm Learning in DT

1. Input: local dataset $D_i=\{(x_j,y_j){\}}_{j=1}^{n_i}$, number of local rounds $t$, number of global rounds $T$, local learning rate $\eta$, initialization parameters ${\theta }_i$. 2. for $t=\mathrm{0,1},\dots ,T$ do 3. dataset $D_i$ from to $i$ client: 4.    $L_i\left({\theta }_i\right)=-{\displaystyle \frac{1}{n_i}}{\sum }_{j=1}^{n_i}{\sum }_{k=1}^C{y}_{jk}\mathit{log}(p_{jk})$ 5.   for local training ${\theta }_i$ in $i$ do 6    if client $i$ is to satisfy: ${\sum }_{i=1}^N{b}_i{x}_i\le B_{max}$ and $DP\left[k\right]$ 7.     model update ${\theta }_i^{\left(t+1\right)}\leftarrow {\theta }_i$:               ${\theta }_i^{\left(t+1\right)}={\theta }_i^{\left(t\right)}-\eta \nabla L_i\left({\theta }_i^{\left(t\right)}\right)$ 8.       else select client $i$ training: 9.       ${\theta }^{\left(t+1\right)}={\sum }_{i\in S}{w}_i{\theta }_i^{\left(t+1\right)}$ 10.   end for 11.  end for 12. Output: ${\theta }^{\left(t+1\right)}$: updating each node of the swarm

5. Numerical Results

This section contains our experimental setup and analysis. The experiments were conducted on a hardware system equipped with an Intel(R) Core (TM) i7-13700F CPU and NVIDIA RTX4070Ti super 16G GPU (The manufacturers of the CPU and GPU are Intel Corporation and NVIDIA Corporation, respectively. Both manufacturers are located in Santa Clara, CA, USA). The simulation was implemented using PyTorch (CUDA 11.8) to build a distributed training platform. The GPU and CPU communication was facilitated using PCI-E 4.0.

We use two typical datasets (MNIST and CIFAR10), which contain different data types and diverse categories, respectively, as shown in

Table 2. Dataset content.

Data Name	MNIST	CIFAR-10
Data Type	handwritten numbers	actual object image
Image Dimensions	28 × 28, single channel (grayscale)	32 × 32, 3 channels (RGB)
Number Of Categories	10 (numbers 0 to 9)	10 (airplane, bird, and so on)
Training Set Size	70,000	60,000
Data Source	Yann LeCun website	CIFAR website

. The typical baseline methods FedAvg [7], SL [9], DDT-CSL [10], and the KSL-DT are compared and evaluated in three aspects. FedAvg is a classic federated learning method that aggregates the local model parameters through a central server to update the global model. SL is an advanced fully decentralized federated learning method that improves privacy and robustness. DDT-CSL builds twins to perceive the status of physical devices in real time and dynamically optimize resource allocation, which significantly improves the real-time performance and communication efficiency in industrial IoT. KSL-DT selects clients with greater contributions to complete wireless mapping and Keeping SL twin modeling to optimize the wireless channel transmission rate.

5.1. Convergence Rate of the Model

We selected the loss value change trend of four baselines on the MNIST dataset (grayscale images), which was used to evaluate the convergence rate and model fit of the model. The experimental results are shown in Figure 2. As shown in Figure 2a, during initial training, the loss value ranged from 0.5 to 0.6, and all four algorithms showed a relatively stable downward trend. Compared with the other three algorithms, the loss values of KSL-DT in the 20th round were reduced by 0.05, 0.1, and 0.08, respectively. The convergence rate of the KSL-DT model selection in the first 20 rounds of aggregate training was not significantly better than that of the other three baseline algorithms. As shown in Figure 2b, in the first 50 rounds of training, the average loss values of the four algorithms were 0.234, 0.283, 0.343, and 0.409, respectivley. KSL-DT showed the lowest average loss value. As shown in Figure 2c, when gradually increasing the number of training rounds to 100, KSL-DT generally shows the fastest and deepest descent in the experiment, which is significantly better than that of SL and FedAvg. The loss value of the curve approaches 0.04. From the qualitative analysis of the observation results, we found that KSL-DT has the fastest convergence rate, as shown in Figure 2c, compared to DDT-CSL. The curve area is the smallest and the slope of the loss curve is the largest. In addition, the algorithm proposed in this study has the best convergence rate compared with the other three methods and improves by 34.1%. This is because each selected client (the client in the backpack) uploads the local update (gradient parameter) to the SL architecture to achieve wireless mapping $\Phi$:

$$G_{k,i}^{(t)}=\Phi ({{\theta }_{k,i}}^{(t+1)},\mathrm{Parameter} \mathrm{distribution})$$

(24)

where auxiliary information contains the gradient parameter distribution of the local model. In the DT environment (cooperative node), the model parameters in the swarm network are updated and deeply integrated $G_{k,i}^{(t)}$, denoted as $\Omega$:

$${\theta }^{(t+1)}=\Omega (\{G_{k,i}^{(t)}\})$$

(25)

KSL-DT introduces the fast virtual iteration provided by “DT” and the multi-model complementary characteristics brought by “Backpack SL.” It promotes the non-independent closure of the local model training in the swarm network. The global aggregation model is no longer limited to the average weight update. The complex transformation of $\Phi (·)$ and $\Omega (·)$ is used in the DT structure to realize the modeling of the SL network.

5.2. Model MSE Value Verification

On the basis of Experiment A, after we choose CIFAR10 of the RGB image type to verify the backpack model client selection, KSL-DT updates the model’s MSE value. Based on the backpack model client selection method, we, respectively, select the number of clients with the largest contribution value. The number of clients included the following: 10, 20, and 50, as shown in Figure 3. In the initial training phase of the model, the coordinated update of the swarm network nodes was not sufficient, which resulted in a higher initial MSE. (The initial MSE for 10 clients is about 4.02, while the MSE for 20 and 50 clients in rounds 0 to 1 is 4.99 and 6.35, respectively). As the number of selected clients gradually increases, the amount of data for each client decreases relatively. The distribution is more heterogeneous, and the contribution value is more uniform, as shown in

Table 3. MSE under different client selection numbers.

Training Rounds	10 Clients	20 Clients	50 Clients
0	4.02	4.99	6.35
3	2.50	3.05	3.84
5	2.01	2.37	2.96
10	1.56	1.78	1.89
20	0.79	0.95	0.99
50	0.05	0.03	0.03

. Between rounds 0 and 20, the MSE shows an obvious downward trend, with the MSE values for 10, 20, and 50 clients decreasing by 3.23, 4.04, and 5.36, respectively. The MSE with 50 clients has the largest decrease compared to the MSE with 10 and 20 clients. This experimentally verifies the theory of the influence of network node variance on the convergence rate in Formula (23) above. The more nodes (i.e., the number of clients) in the SL network, the faster the global model converges. In the late convergence state of the model, we observed that the result of the three types of clients was around 20 to 30 rounds, and the three curves all dropped below 1 in the Zoom In. Then, they stabilized. Between 30 and 50 rounds, the MSE value continued to slowly approach 0.05 to 0.03, which shows that the model has generalization ability in different client selections. At the same time, as the number of rounds increases, the information on different clients is integrated under the knowledge sharing mechanism $\Omega (·)$ of KSL-DT, and the MSE decreases rapidly. Through the parallel and complementary learning of DT and backpack models, KSL-DT can absorb the characteristics of each client more quickly and fully, which shows fast convergence and a low final error.

We compare and evaluate the different client contributions on the RGB dataset (CIFAR10) to verify the scalability of KSL-DT on the hardware (model: RK3568_HM_costdown/V03). The industrial equipment model is set up, which can be trained and mapped in real time on the mobile terminal, as shown in Figure 4, which supports the application capability of the verification model.

5.3. Parameter Transmission Amount for the Model

Based on Experiments A and B, we validated the proposed method with two types of datasets: grayscale and RGB images. We also compared the amount of model parameter transmission. In MNIST, KSL-DT reduces the amount of parameter transmission by 49.88%, 68.05%, and 78.31% compared with the other three baseline algorithms, respectivley. In CIFAR-10, KSL-DT reduces the amount of parameter transmission by 59.66%, 65.51%, and 84.60% compared with the other three baseline algorithms, respectivley. In both datasets, the amount of parameter transmission of the KSL-DT model is the smallest. This shows that KSL-DT has low communication overhead and reduces network congestion for high concurrent transmission. In terms of the difference in parameter growth, KSL-DT increases from ~40 k to ~69 k, with an absolute value of 72.5%. The other three algorithms increase from ~99 k to ~138 k, ~115 k to ~216 k, and ~258 k to ~319 k, with relative increases of 39.39%, 87.83%, and 23.6%, respectively. Among the four algorithms, the number of model training parameters of CIFAR-10 is significantly higher than that of MNIST. This is because the data structure of CIFAR-10 is complex. Moreover, it requires a deeper or wider network structure to learn the features of color images. In CIFAR-10, KSL-DT still shows the smallest amount of model parameter transmission. This shows that the proposed method has good robustness. At the same time, observing Figure 5, due to the KSL-DT proposed in this study, the representation of some key layers or knowledge distillation is synchronized on the client, which greatly reduces the amount of parameter transmission. Meanwhile, in the wireless mapping channel of the DT, a single device collaboratively calculates and directly updates the local model, which reduces the transmission of high-concurrency parameters.

6. Conclusions

To address the communication overhead caused by data-driven DT and the congestion of high-concurrency wireless channels caused by distributed FL, we propose a new KSL scheme based on the backpack model. It selects the client with the largest contribution to participate in Keeping SL twin modeling, which explicitly solves the defects of high communication overhead and slow convergence speed caused by the concurrent transmission of the model. Through experimental comparison, the robustness and convergence speed of the proposed algorithm are verified. The asynchrony problem of SL will be explored in the future.