Efficient Sparse MLPs Through Motif-Level Optimization Under Resource Constraints

Abstract

We study motif-based optimization for sparse multilayer perceptrons (MLPs), where weights are shared and updated at the level of small neuron groups (‘motifs’) rather than individual connections. Building on Sparse Evolutionary Training (SET), our approach reduces the number of unique parameters and redundant multiply–accumulate operations by exploiting block-structured sparsity. Across Fashion-MNIST and a lung X-ray dataset, our Motif-SET improves training/inference efficiency with modest accuracy trade-offs, and we provide a principled recipe to choose motif size based on accuracy and efficiency budgets. We further compare against representative modern sparse training and compression methods, analyze failure modes such as overly large motifs, and outline real-world constraints on mobile/embedded targets. Our results and ablations indicate that motif size $m=2$ often offers a strong balance between compute and accuracy under resource constraints.

1. Introduction

The emergence of neural networks has facilitated the development of artificial intelligence (AI), defined as the ability of machines to simulate human cognitive processes. With the advancement of neural networks, the tasks they address have become increasingly complex, often involving the handling of high-dimensional data to satisfy specific requirements. To enhance performance, deep neural networks have evolved to more accurately mimic human brain functions, leading to substantial increases in computational cost and training time [1]. Typically, DNNs have many layers with fully connected neurons, which contain most of the network parameters (i.e., the weighted connections), leading to a quadratic number of connections with respect to their number of neurons [2].

To address this issue, the concept of sparse connected Multi-layer Perceptrons with evolutionary training was introduced. This algorithm, compared to fully connected DNNs, can substantially reduce computational cost on a large scale. Moreover, with feature extraction, sparsely connected DNNs can maintain a performance comparable to that of fully connected models.

However, this approach still demands considerable computational resources and time, which remains a limitation. Furthermore, the introduction of motif-based DNNs, which can retrain neurons using small structural groups (e.g., groups of three neurons), suggested the potential to surpass the performance of sparse connected DNNs and significantly enhance overall network efficiency. This paper aims to analyze and test motif-based DNNs, comparing their performance against benchmark models.

To provide a deeper understanding, the following sections will delve into the foundational aspects of these approaches.

As mentioned before, traditional neural networks are usually densely connected, meaning that each neuron is connected to every other neuron in the previous layer, resulting in a large number of parameters. Unlike normal DNN models, SET helps introduce sparsity, reduce redundant parameters in the network, and improve computational efficiency. Through the evolutionary algorithm, SET can gradually optimize the weights so that many connections become irrelevant or zero [2]. Therefore, SET is applied to improve training efficiency by optimizing the sparse structure of the model and reducing redundant parameters, which eventually can end in a reduction in the computational cost [3].

To further improve the performance of the Deep Neural Network, feature engineering is considered as a critical step in the development of machine learning models, involving the selection, extraction, and transformation of raw data into meaningful features that enhance model performance [4]. By enforcing sparsity in the neural network, SET effectively prunes less important connections, thereby implicitly selecting the most relevant features. As the evolutionary algorithm optimizes the network, connections that contribute insignificantly to the model’s performance are gradually set to zero, allowing the network to focus on the most informative features. If feature selection can be applied in this process, with some important features selected and remaining features dropped, the complexity of the network would be largely decreased and the quantified features would keep the original accuracy. Consequently, SET feature selection results in a streamlined model that is both computationally efficient and more accurate, leading to better overall performance [3], and Neil Kichler has demonstrated the effectiveness and robustness of this algorithm in his studies, further validating its practical application and benefits [5].

Network motifs are significant, recurring patterns of connections within complex networks. They reveal fundamental structural and functional insights in systems like gene regulation, ecological food webs, neural networks, and engineering designs. By comparing the occurrence of these motifs in real versus randomized networks, researchers can identify key patterns that help us to understand and optimize various natural and engineered systems.

As mentioned before, the SET updates new random weights when the weights of the connections are negative or insignificant (close to or equal to zero) and to some extent lead to some computational burden [6]. Based on the concept of motif and SET, a structurally sparse MLP is proposed. The motif-based structural optimization gave us the idea of renewing the weights by establishing a topology which can largely improve the efficiency (shown in Figure 1) [7,8]. Motif-structured training targets low-memory and predictable-latency deployments such as mobile perception, on-device health screening, and simple event detection. These scenarios motivate the efficiency targets and evaluation choices used in our experiments.

Our key research question is as follows: Can the efficiency and accuracy of sparse MLPs be improved by optimizing the structure of the Sparse MLPs and fine-tuning of the parameters of the network?

2. Related Work

Sparse MLP models have demonstrated significant potential to reduce computational costs (e.g., hardware and computation time requirements) while enhancing accuracy through feature extraction and sparse training. This research uses the work of Mocanu et al. as a benchmark model for comparison [3]. This section reviews the historical development of sparse neural networks. Subsequently, the key idea and algorithm of SET will be discussed. Lastly, the basic idea of structural optimization of Sparse MLPs will be introduced.

Y. LeCun et al. [9] introduced the concept of network pruning in the paper “Optimal Brain Damage” in 1990. This approach calculated the contribution of each connection to the general network error and selectively removed some of the nodes which are considered less important. Utilizing second-order derivatives, this method effectively reduced model complexity while preserving performance. Therefore, this seminal work provided and was regarded as a theoretical basis for later pruning techniques. Based on this seminal paper, B. Hassibi et al. proposed the “Optimal Brain Surgeon” method in 1993, in which second-order derivatives were also used but offered a more precise method for pruning the weights by considering the Hessian matrix’s structure [10]. The Optimal Brain Surgeon method was shown to achieve superior performance by more accurately identifying and removing redundant connections. This refinement was regarded as an important step forward in the efficiency of network pruning. In 2016, Han et al. introduced the concept of “Deep Compression”, a concept which combines pruning, quantization, and Huffman encoding [11]. This three-step method significantly reduces the storage requirements and computational cost of neural networks while maintaining the accuracy. In addition to the standard sparse network training steps, such as initial pruning followed by weight retraining, Deep Compression incorporates Huffman encoding as a final step. This study highlighted the practical advantages of integrating multiple optimization techniques. In 2018, Mocanu et al. introduced the idea of Sparse Evolutionary Training in their paper “Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity Inspired by Network Science” [3]. In this research, Erdős-Rényi graph initialization was used to create the initial sparse network and initialize the weights. By selectively adding and removing connections based on the deviation and performance, SET aims to maintain a high proportion of zero-valued connections while optimizing the model’s accuracy. The process of the SET algorithm is shown in Figure 2. As mentioned above, this model will be taken as the benchmark mark to compare with the one in this research.

Based on previous research work, the concept of “Dynamic Sparse Reparameterization” was given by J. Frankle et al.; it is a method that involves continually adjusting the sparsity of network connections during training [12]. By dynamically reparameterizing the network, this approach helps to maintain a balance between performance and efficiency. This method stands out for its ability to adaptively optimize the network structure dynamically, leading to more efficient training processes. Based on the research of Mocanu et al., the paper “Robustness of sparse MLPs for supervised feature selection” by Kichler [5] in 2021 combined feature extraction with a sparse multi-layer training method which promoted the development of research. With a certain proportion of features dropped, the performance of the neural network can still be maintained as a densely connected network.

The motif-based concept refers to certain types of structural topology or network structure, and the term is usually used to describe a sub-network or a network pattern (shown in Figure 3). This diagram illustrates the general procedure of motif-based optimized SET models. The DNN model on the left is trained and retrained by individual nodes [13]. Therefore, the weights between nodes are different from each other. However, for the motif-based model, the weights will be initialized and retrained by small groups with a specific size of nodes [14]. By applying the motif-based structure, the efficiency of the network can be improved while maintaining the accuracy.

We benchmark Motif-SET against (i) dynamic sparse reparameterization (DSR), which adapts sparse connectivity during training; (ii) deep compression (magnitude pruning + quantization + Huffman coding); and (iii) magnitude pruning with weight rewinding. These methods capture the main lines of modern sparse training (DSR) and post-training compression (deep compression, pruning). We report accuracy, parameter count, theoretical FLOPs, and wall-clock latency to highlight accuracy–efficiency trade-offs. (Exact measurement protocol in Section 5.6).

3. Methodology

To address the research questions related to the motif-based structural optimization of sparsely connected neural networks, this section provides a detailed illustration of the proposed approaches. This section discusses the topological optimization method. Subsequently, the process of training will be discussed. Finally, the evaluation environment for this research is described.

The core principle of motif-based structural optimization in SET involves assigning weights between neurons based on motifs during each training process, followed by distributing these weights to individual nodes (Algorithm 1). The following pseudo code is the general training process of the motif-based SET model:

Algorithm 1 Motif-SET: Training with motif size m

Require:  motif size m, sparsity $\mathit{ϵ}$, hidden sizes $\left\{h_{\ell }\right\}$   1: Initialize sparse masks with Erdős–Rényi; partition units into motifs of size m   2: for epoch $=1\dots T$ do   3:     Forward: for each layer, apply block-constant weights $W_{\ell }$ by motif.   4:     Backward: accumulate gradients per motif block, then distribute to edges.   5:     Evolution (SET): prune edges with small magnitude; regrow per sparse mask.   6: end for

This pseudo code outlines the process of network initialization, forward propagation, and backward propagation in the motif-based DNN model. As previously noted, most steps are performed using customizable motifs of a specific size. A detailed explanation of the network construction and training process is provided in the following subsections.

3.1. Network Construction

The general idea of using structural optimization based on a motif is to group nodes with a certain size and train them together. Unlike simply reducing the number of neurons, each node in the motif-based optimized network participates in both training and retraining. The key difference lies in the process of assigning new weights to nodes, which is conducted according to a specific topology, thereby enhancing the network’s efficiency [7,15].

Parameter initialization: Before initializing the weights of neural networks, some parameters need to be defined. The input size X refers to the size of the features of the training data set that is selected as input. For instance, for the Fashion MNIST dataset, the number of pixels in the graph is 784 (28 × 28); therefore, the input size is 784. The motif size m refers to the size of a topology or sub-network. Hidden size refers to the number of neurons for each hidden layer [16]. Epsilon $\mathit{ϵ}$ is a parameter that controls the sparsity level of the network. The activation function $\mathit{\sigma }$ is used to introduce non-linearity into the model. The loss function L measures the difference between the predicted outputs and the actual outputs [17].

Non-divisible layer widths. If a layer width n is not divisible by the motif size m, we use one of two simple strategies: (i) padding the layer with up to $m-1$ dummy neurons (discarded at inference without affecting outputs), or (ii) forming a residual motif with size $<m$ for the final group. Both preserve functionality while keeping implementation minimal.

Weights and bias initialization: The random uniform distribution is utilized for generating the initial weights. The weights are assigned to each motif instead of each node, which will be more efficient. The He function is used here to set limitations for the weights [18]. The Erdős-Rényi model is used for setting sparse weight masks which can remove or keep the initial weights. And for each layer, it creates a random bias vector $b_i$. The motif size also needs to be chosen such that the input size is divisible by the motif size [12]. The pseudo code for network construction is shown as follows:

Compared to the normal SET model, instead of initializing the weights and bias individually, the weights and bias initialization in this study are performed by motifs; for instance, 3 nodes are used as a motif. The weights and bias are first assigned to each motif then each motif will assign the parameters to each node, which are the same within each motif. Additionally, the motif size of the model can be customized. In this way, the efficiency of the network initialization will be significantly improved.

We use $s\in (0,1)$ to denote the target edge sparsity (fraction of zeros). At initialization, we sample an Erdős–Rényi (ER) mask to achieve a target density $1-s$ with edge probability

$$p={\displaystyle \frac{(1-s)\mathit{ϵ}(n_{\mathrm{in}}+n_{\mathrm{out}})}{n_{\mathrm{in}}{n}_{\mathrm{out}}}},$$

where $\mathit{ϵ}$ is a density scaling constant. Unless otherwise noted, we set $s=0.9$ and $\mathit{ϵ}=1$ and use s consistently throughout (we avoid reusing $\mathit{ϵ}$ to mean sparsity).

3.2. Training Process

This subsection will show the process of the motif-based model training, which is the core of the whole study.

Forward Propagation: The forward function is similar to normal forward propagation in other models; the only difference in this study is that the nodes involved in the forward propagation process are trained in small groups [19] which will improve the total efficiency. The process is shown through the following equations: $Z^{\left(i\right)}$ representing the linear combination of i layer, $A^{(i-1)}$ for the activation function for each layer (ReLu, Softmax, Sigmoid), $W^{\left(i\right)}$ for weight matrix for i layer, $b^{\left(i\right)}$ for bias [20]. For each motif in layer i:

$${\mathbf{Z}}^{\left(i\right)}={\mathbf{A}}^{(i-1)}{\mathbf{W}}^{\left(i\right)}+{\mathbf{b}}^{\left(i\right)}$$

(1)

$${\mathbf{A}}^{\left(i\right)}=f_{\mathrm{activation}}\left({\mathbf{Z}}^{\left(i\right)}\right)$$

(2)

For the output layer, Softmax is chosen as the activation function because it converts raw model outputs into probabilities, ensuring they sum to 1. This is crucial for multi-class classification, providing clear and interpretable probabilities for each class. Backward Propagation: In the process of backward propagation, the output layer error ${\mathit{\delta }}^{\left(L\right)}$ needs to be calculated first, where ${\mathbf{A}}^{\left(L\right)}$ is the output of the network and $\mathbf{Y}$ is the true label:

$${\mathit{\delta }}^{\left(L\right)}={\mathbf{A}}^{\left(L\right)}-\mathbf{Y}$$

(3)

Then, the gradient for the output layer and each hidden layer needs to be computed in a reversed order. Let B denote the mini-batch size.

$$\begin{array}{cc}\hfill {\displaystyle \frac{\partial L}{\partial W^{\left(L\right)}}}& ={\displaystyle \frac{1}{B}}{\left(A^{(L-1)}\right)}^{\top }{\mathit{\delta }}^{\left(L\right)},\hfill \end{array}$$

(4)

$$\begin{array}{cc}\hfill {\displaystyle \frac{\partial L}{\partial b^{\left(L\right)}}}& ={\displaystyle \frac{1}{B}}\sum _{i=1}^B{\mathit{\delta }}^{\left(L\right)}.\hfill \end{array}$$

(5)

For each hidden layer i from $L-1$ to 1:

$${\mathit{\delta }}^{\left(i\right)}=\left({\mathit{\delta }}^{(i+1)}{\mathbf{W}}^{(i+1)}\right)\odot f^{\prime }\left({\mathbf{Z}}^{\left(i\right)}\right)$$

(6)

Motif-based computation for each hidden layer [21,22], a zero matrix $\mathit{\delta }$ is initialized. And for each motif j in layer i, each submatrix $W_{\mathrm{submatrix}}$ from weight matrix $W^{\left(i\right)}$ is extracted:

$${\mathbf{W}}_{\mathrm{submatrix}}={\mathbf{W}}^{\left(i\right)}[j_{\mathrm{start}}:j_{\mathrm{end}}]$$

(7)

The sub-delta and sub-activation for each motif are calculated:

$${\mathit{\delta }}_{\mathrm{sub}}=\left({\mathit{\delta }}^{(i+1)}{\mathbf{W}}_{\mathrm{submatrix}}\right)\odot f^{\prime }\left({\mathbf{Z}}_{\mathrm{sub}}\right)$$

(8)

Update gradients for each motif and node individually:

$${\displaystyle \frac{\partial \mathcal{L}}{\partial {\mathbf{W}}_{\mathrm{sub}}}}={\displaystyle \frac{1}{m}}{\left({\mathbf{A}}^{(i-1)}\right)}^{\top }{\mathit{\delta }}_{\mathrm{sub}}$$

(9)

$${\mathit{\delta }}^{\left(i\right)}[j_{\mathrm{start}}:j_{\mathrm{end}}]={\mathit{\delta }}_{\mathrm{sub}}$$

(10)

In the end of backward propagation, the weights and bias are updated for each layer. The pseudo code of back propagation is shown as follows:

Followed by forward propagation, the backward propagation training will also be trained by motifs in each layer; each motif will assign the retrained results to each node (Algorithm 2).

Algorithm 2 Backward pass for a motif-based sparse NN (one minibatch)

Require:  inputs X, labels y, cached pre-activations ${\left\{Z_{\ell }\right\}}_{\ell =1}^L$, activations ${\left\{A_{\ell }\right\}}_{\ell =0}^L$ with $A_0=X$ Require:  motif size m; per-layer motif parameters $\left\{K_{\ell }\right\}$ (size $G_{\mathrm{out}}^{(\ell )}\times G_{\mathrm{in}}^{(\ell )}$); sparse edge masks $\left\{S_{\ell }\right\}$ Ensure:  gradients $\{\nabla K_{\ell },\nabla b_{\ell }\}$ and updated parameters 1: $B\leftarrow$ batch size;                                                                                                                                                                                                                       ▹ number of samples 2: // Output layer gradient 3: ${\mathit{\delta }}_L\leftarrow {\nabla }_{A_L}\mathcal{L}(A_L,y)\odot {\mathit{\sigma }}^{\prime }\left(Z_L\right)$ 4:$\nabla W_L\leftarrow A_{L-1}^{\top }{\mathit{\delta }}_L\odot S_L$;   $\nabla b_L\leftarrow {\sum }_{i=1}^B{\mathit{\delta }}_L^{\left(i\right)}$ 5: Update $W_L,b_L$ (optimizer step) 6: 7: for $\ell =L-1$ down to 1 do 8:    // Backpropagate through mask and motifs 9:    $W_{\ell }^{\mathrm{eff}}\leftarrow \mathrm{Expand}(K_{\ell },m)\odot S_{\ell }$                                                                                                            ▹ replicate each motif parameter to an $m\times m$ block 10:    ${\mathit{\delta }}_{\ell }\leftarrow \left({\mathit{\delta }}_{\ell +1}{W_{\ell }^{\mathrm{eff}}}^{\top }\right)\odot {\mathit{\sigma }}^{\prime }\left(Z_{\ell }\right)$ 11:     // Motif-aware gradient accumulation (block-sums) 12:     $A_{\ell -1}^{\mathrm{grp}}\leftarrow \mathrm{GroupSum}(A_{\ell -1},m)$                                                                                                                                                                                  ▹$B\times G_{\mathrm{in}}^{(\ell )}$ 13:    ${\mathit{\delta }}_{\ell }^{\mathrm{grp}}\leftarrow \mathrm{GroupSum}({\mathit{\delta }}_{\ell },m)$                                                                                                                                                                                          ▹$B\times G_{\mathrm{out}}^{(\ell )}$ 14:    $\nabla K_{\ell }\leftarrow {{\mathit{\delta }}_{\ell }^{\mathrm{grp}}}^{\top }{A}_{\ell -1}^{\mathrm{grp}}$                                                                                                                                                                                                     ▹$G_{\mathrm{out}}^{(\ell )}\times G_{\mathrm{in}}^{(\ell )}$ 15:    $\nabla K_{\ell }\leftarrow \nabla K_{\ell }\odot \mathrm{ActiveBlocks}(S_{\ell },m)$                                                                                                                              ▹ zero-out blocks with no active edges 16:    $\nabla b_{\ell }\leftarrow {\sum }_{i=1}^B{\mathit{\delta }}_{\ell }^{\left(i\right)}$ 17:    Update $K_{\ell },b_{\ell }$ (optimizer step) 18: end for

When SET helps and when it hurts: We use a short warmup of five epochs before any pruning. The prune fraction per step is $\mathit{\zeta }=0.2$ every 10 epochs, followed by Erdős–Rényi regrowth to maintain the target sparsity. Over the last third of training, the prune fraction decays to zero with a cosine schedule. Pruning too early or with a larger $\mathit{\zeta }$ caused accuracy loss and unstable training; delaying pruning and decaying $\mathit{\zeta }$ avoided these issues in practice. Regrowth ensures that edges removed in earlier stages can reappear if the gradient signal supports them, which addresses the concern that relevance can change across stages.

3.3. Process of Evolution

The core of the SET algorithm involves evolution, where weights close to zero are pruned and new weights are assigned [3]. The pseudo-code for this process is provided below:

We use a five-epoch warmup with no pruning, then prune a fraction $\mathit{\zeta }=0.2$ every 10 epochs with ER regrowth to maintain the target sparsity. Over the final third of training, $\mathit{\zeta }$ decays to 0 via a cosine schedule (Algorithm 3).

Algorithm 3 Evolution (SET) with motif-aware pruning and regrowth (per epoch)

Require:  motif size m; prune fraction $\mathit{\zeta }\in (0,1)$; init std $\mathit{\sigma }$; motif params ${\left\{K_{\ell }\right\}}_{\ell =1}^L$; sparse masks ${\left\{S_{\ell }\right\}}_{\ell =1}^L$ Ensure:  updated $\left\{K_{\ell }\right\}$ and $\left\{S_{\ell }\right\}$ at the same sparsity 1: for  $\ell =1\to L$ do 2:     $W_{\ell }^{\mathrm{eff}}\leftarrow \mathrm{Expand}(K_{\ell },m)$                                                                                                                                                                   ▹ replicate each $K_{\ell }$ entry to an $m\times m$ block 3:     $A\leftarrow \{(i,j)\mid S_{\ell }[i,j]=1\}$                                                                                                                                                                                                                   ▹ active edge indices 4:     $k\leftarrow ⌊\mathit{\zeta }·\left|A\right|⌋$ 5:     $P\leftarrow \mathrm{SmallestMagnitude}\left(|W_{\ell }^{\mathrm{eff}}|\mathrm{on}A,k\right)$                                                                                                                                                                                            ▹ SET prune set 6:     Prune:   for $(i,j)\in P$ do $S_{\ell }[i,j]\leftarrow 0$; $W_{\ell }^{\mathrm{eff}}[i,j]\leftarrow 0$ end for 7:     $B^{\mathrm{old}}\leftarrow \mathrm{ActiveBlocks}(S_{\ell },m)$                                                                                                                                             ▹$G_{\mathrm{out}}\times G_{\mathrm{in}}$ boolean: any edge active in each block 8:     $Z\leftarrow \{(i,j)\mid S_{\ell }[i,j]=0\}$                                                                                                                                                                                         ▹ available positions for regrowth 9:     $R\leftarrow \mathrm{UniformSample}(Z,k)$                                                                                                                                                                                                         ▹ Erdős–Rényi regrowth 10:     Regrow:   for $(i,j)\in R$ do 11:         $S_{\ell }[i,j]\leftarrow 1$                                                                                                                                                                                                                                                             ▹ activate edge 12:         $(g_o,g_i)\leftarrow (\lfloor i/m\rfloor ,\lfloor j/m\rfloor )$                                                                                                                                                                                                ▹ map edge to motif block 13:         if $B^{\mathrm{old}}[g_o,g_i]=0$ then $K_{\ell }[g_o,g_i]\sim \mathcal{N}(0,{\mathit{\sigma }}^2)$ end if 14:     end for 15: end for

3.4. Evaluation Environment

The evaluation is performed by testing both runtime and accuracy. We use the results reported by Mocanu et al. [3] as a benchmark to assess improvements. Robustness is further examined by evaluating across datasets of different types. All experiments were conducted on a desktop computer running Windows 10 (Microsoft, Redmond, WA, USA), with an Intel Core i5 13th-generation CPU (Intel Corporation, Santa Clara, CA, USA), 32 GB RAM, and an NVIDIA GeForce RTX 4070 Ti 12 GB GPU (NVIDIA Corporation, Santa Clara, CA, USA).

3.5. On-Device Latency and Energy

Many edge platforms are bandwidth-bound rather than compute-bound. To provide a practical proxy without requiring device-specific runs, we introduce a closed-form estimator that maps our per-motif counts to normalized on-device latency and energy.

Let $C_{\mathrm{mm}}$ denote time per multiply–accumulate (MAC) for a dense matrix multiply on a target device class and $C_{\mathrm{ld}}$ the amortized time per weight/activation load. For motif size m, block execution reduces dominant multiplies by approximately $m^2$ (due to weight sharing) and improves locality. For a layer with $F_1$ dense MACs and $M_1$ memory operations at $m=1$, we estimate

$$\mathrm{Lat}\left(m\right)\approx \mathit{\alpha }·{\displaystyle \frac{F_1}{m^2}}{C}_{\mathrm{mm}}+\mathit{\beta }·{\displaystyle \frac{M_1}{m}}{C}_{\mathrm{ld}},$$

(11)

where $\mathit{\alpha },\mathit{\beta }\in [0,1]$ capture kernel/runtime overheads. The $1/m$ memory term reflects fewer distinct blocks and better cache reuse.

Table 1. Normalized on-device estimates from the theoretical model (lower is better). Here, $\mathit{\kappa }$ summarizes the memory-to-compute ratio (a larger ratio implies more bandwidth-bound).

Motif Size	$\mathit{m}=1$	$\mathit{m}=2$	$\mathit{m}=4$	$\mathit{m}=8$
$\widehat{\mathrm{Lat}}\left(m\right)$	$1.00$	$0.25+\mathit{\kappa }/2$	$0.06+\mathit{\kappa }/4$	$0.02+\mathit{\kappa }/8$
$\widehat{\mathrm{Energy}}\left(m\right)$	$1.00$	$0.25+\mathit{\kappa }/2$	$0.06+\mathit{\kappa }/4$	$0.02+\mathit{\kappa }/8$

Note. Define $\mathit{\kappa }:={\displaystyle \frac{\mathit{\beta }{M}_1{C}_{\mathrm{ld}}}{\mathit{\alpha }{F}_1{C}_{\mathrm{mm}}}}$. Compute-bound regimes have $\mathit{\kappa }\ll 1$; bandwidth-bound regimes have $\mathit{\kappa }\gg 1$.

summarizes the normalized on-device latency and energy predicted by the model as a function of motif size m and the memory-to-compute ratio $\mathit{\kappa }$.

With per-MAC energy $E_{\mathrm{mm}}$ and per-load energy $E_{\mathrm{ld}}$, we use

$$\mathrm{Energy}\left(m\right)\approx \mathit{\alpha }·{\displaystyle \frac{F_1}{m^2}}{E}_{\mathrm{mm}}+\mathit{\beta }·{\displaystyle \frac{M_1}{m}}{E}_{\mathrm{ld}}.$$

(12)

To avoid device-specific constants, we report values normalized to the dense motif case $m=1$:

$$\widehat{\mathrm{Lat}}\left(m\right)={\displaystyle \frac{\mathrm{Lat}\left(m\right)}{\mathrm{Lat}\left(1\right)}},\widehat{\mathrm{Energy}}\left(m\right)={\displaystyle \frac{\mathrm{Energy}\left(m\right)}{\mathrm{Energy}\left(1\right)}}.$$

(13)

In practice, $F_1$ can be taken as the FLOPs (MACs) of the $m=1$ configuration and $M_1$ proportional to the parameter count at $m=1$. The proportionality cancels in the ratios of (13).

This estimator lets practitioners translate per-motif FLOPs/parameter tables into device-facing proxies without additional runs. If a single device measurement is available, $(\mathit{\alpha },\mathit{\beta },\mathit{\kappa })$ can be calibrated once and reused across layers with (11)–(13).

3.6. Why Motifs Reduce Redundant Computation

Consider a layer with $n_{\mathrm{in}}$ inputs and $n_{\mathrm{out}}$ outputs. Partition units into motifs of size m (assume $m\mid n_{\mathrm{in}},n_{\mathrm{out}}$) yielding $G_{\mathrm{in}}=n_{\mathrm{in}}/m$ and $G_{\mathrm{out}}=n_{\mathrm{out}}/m$ groups. Let $K\in {\mathbb{R}}^{G_{\mathrm{out}}\times G_{\mathrm{in}}}$ denote motif-to-motif weights, expanded to a block-constant matrix $W=K\otimes {\mathbf{1}}_{m\times m}$ (optionally masked sparsely as in SET). In the dense idealization, unique parameters drop from $n_{\mathrm{out}}{n}_{\mathrm{in}}$ to $G_{\mathrm{out}}{G}_{\mathrm{in}}={\displaystyle \frac{n_{\mathrm{out}}{n}_{\mathrm{in}}}{m^2}}$.

Block computation. Writing the input as group sums $s\in {\mathbb{R}}^{G_{\mathrm{in}}}$, with $s_g={\sum }_{i\in \mathrm{group}g}{x}_i$, the pre-activation can be computed as follows:

$$z=\left(Ks\right)\otimes {\mathbf{1}}_m\Rightarrow \mathrm{multiplies}\mathrm{reduce}\mathrm{from}\mathcal{O}\left(E\right)\mathrm{to}\mathcal{O}\left(B\right),$$

where E is the number of active edges (as in SET) and B is the number of active motif blocks. In the common case that each active block corresponds to up to $m^2$ active edges, $B\approx E/m^2$, showing an $m^2$-fold reduction in unique parameters and an $m^2$-fold reduction in dominant multiplies under block execution.

Capacity trade-off. The block-constant structure limits the effective rank of W by $\mathrm{rank}\left(W\right)\le min(G_{\mathrm{out}},G_{\mathrm{in}})$, explaining accuracy drops for large m. Our ablations (Section 5.2) empirically show $m=2$ balances compute reduction and expressivity on studied datasets.

4. Experiment

This section outlines the experimental process, starting with data preparation, followed by experimental design and evaluation. The aim is to assess the efficiency and accuracy of the motif-based sparse neural network compared to the benchmark model.

4.1. Data Preparation

In this research, the Fashion MNIST (FMNIST) and Lung datasets are used as benchmark datasets to evaluate the performance and efficiency of the model [23]. The FMNIST dataset (shown in Figure 4) is widely recognized as a benchmark for testing deep learning architectures. It is a dataset of Zalando’s article images, which consists of a training set of 60,000 samples and a test set of 10,000 samples. Each sample is a 28 × 28 grayscale image associated with a label from one of 10 classes [5]. This study loads the images using TensorFlow’s FMNIST module, normalizes pixel values to $[0,1]$, and applies one-hot encoding to the categorical labels. Optionally, it standardizes data using scikit-learn’s StandardScaler and stores preprocessed data in compressed .npz files for efficient access and reuse. This pipeline ensures the dataset is ready for machine learning tasks, facilitating robust model training and evaluation.

We use the Lung Disease 5-Class dataset (Figure 5), with five classes: COVID-19, Tuberculosis, Pneumonia, Emphysema, and Normal; input vectors have 3312 features per sample. A sample from the Lung dataset is shown in Figure 5. The dataset is initially loaded and its labels are converted into one-hot encoded format. Subsequently, the dataset is split into training and testing sets, reserving one-third for testing purposes. Feature normalization is then performed using StandardScaler library, ensuring consistent scaling across the dataset.

The properties of the two datasets are summarized in

Table 2. Dataset properties and splits used in our experiments.

Dataset	Input Dims	Classes	Train n	Test n	Source/URL
FMNIST	784	10	60,000	10,000	https://github.com/zalandoresearch/fashion-mnist (accessed on 1 March 2024)
Lung	3312	5	17,199	8600	https://www.kaggle.com/datasets/obaidulhaque/lung-disease-5-class-dataset-t-p-n-e-c (accessed on 1 March 2024)

Note. Class balance (min/max, %): FMNIST 10.0/10.0; Lung 19.10/21.41.

.

Table 3. Preprocessing and training settings. CIFAR-10 is included from Section 6.

FMNIST	grayscale in [0, 1]; z-score per pixel over train set; no augmentation
Lung	z-score per feature over train split; no augmentation; 2:1 train:test split
CIFAR-10	per-channel mean [0.4914, 0.4822, 0.4465], std [0.2470, 0.2435, 0.2616]; random crop 32 with padding 4; random horizontal flip p = 0.5
Optimizer	SGD with momentum 0.9; weight decay $5\times {10}^{-4}$
Schedule	cosine decay from initial LR $1\times {10}^{-2}$ with 5-epoch warmup
Batch size	128
Epochs	300 for FMNIST and Lung; 100 for CIFAR-10 sanity check
Seeds	3; we report mean and standard deviation

Measurement protocol: Batching & data: batch size 128; 4 dataloader workers; shuffling per epoch. Optimizer: SGD (momentum $0.9$, weight decay $5\times {10}^{-4}$). LR schedule: cosine decay from $1\times {10}^{-2}$ over 300 epochs. Sparsity: target $s=0.9$ (10% density) unless noted. Timing: wall clock measured over training iterations only (validation excluded), averaged over 3 seeds (±std).

summarizes the input preprocessing and the core training hyperparameters used across datasets.

4.2. Design of the Experiment

As mentioned above, in this experiment, Fashion MNIST and Lung are the two benchmark datasets used to test the accuracy and performance of the motif-based sparse neural network model. The benchmark model is set as the SET model [3]. To evaluate the performance and resource utilization of the model on the datasets, a comprehensive experimental setup is implemented. This setup includes data preprocessing steps such as standardization and one-hot encoding, as well as model initialization and training procedures. Functions were incorporated to retrieve CPU and GPU information to monitor hardware usage. The start and end times of the training process were recorded to calculate the total execution time [24]. Finally, the results, including test accuracy and resource details, were saved to a file for thorough analysis. This approach ensures a detailed and reproducible evaluation of the model’s performance and resource efficiency.

Design of FMNIST testing: the Fashion MNIST has 784 features which is the input size for the neural network, therefore, the motif size is set as 1 for the benchmark model and 2 and 4 for model testing. The number of neurons for each hidden layer is 3000 [16,25]. To further test the efficiency and show the differences of each motif-size model, a simpler version of the DNN model is developed with two hidden layers, and each one has 1000 neurons.

The Lung dataset has 3312 features which is the input size for the neural network and is divisible by one, two, and four. Therefore, four motif sizes are designed for testing. Like the FMNIST dataset, a simpler version of model with two hidden layers (1000 neurons for each) is designed and will be tested. In order to ensure the rigor of the experiment, the control variable method is applied. We train for 300 epochs with a batch size of 128 using SGD (momentum $0.9$, weight decay $5\times {10}^{-4}$) and a cosine learning-rate schedule from an initial LR of $1\times {10}^{-2}$; the target sparsity is $s=0.9$ (10% density) unless stated otherwise.

To better evaluate the performance of each motif-size model, a comprehensive score (S) that combines the percentage reduction in running time ($R_r$) and the percentage reduction in accuracy ($A_r$) is designed [26]. Typically, for a DNN model, accuracy is significantly more important than efficiency [27,28]. Therefore, the accuracy reduction is weighted at 90% and the running time reduction is weighted at 10%. The comprehensive score is calculated using the following formula:

$$S=0.1\times R_r+0.9\times (1-A_r)$$

(14)

where:

$$R_r={\displaystyle \frac{T_{\mathrm{base}}-T}{T_{\mathrm{base}}}}$$

(15)

$$A_r={\displaystyle \frac{A_{\mathrm{base}}-A}{A_{\mathrm{base}}}}$$

(16)

D In these equations,

• S is the comprehensive score.
• $R_r$ is the percentage of running time reduction.
• $A_R$ is the percentage of accuracy reduction.
• $T_{\mathrm{base}}$ is the running time observed for the benchmark model.
• T is the running time for the specific motif-size model.
• $A_{\mathrm{base}}$ is the benchmark model accuracy.
• A is the accuracy for the specific motif-size model.

5. Results

This section first presents the final results for each dataset and each motif size model. Furthermore, the model with the best overall performance for each dataset is identified, addressing the research question in Figure 1 by providing exact accuracy and efficiency metrics.

5.1. Experiment Results

This subsection presents and analyzes the results for both the FMNIST and Lung datasets.

5.2. Ablations: Trend vs. Motif Size

We sweep $m\in \{1,2,4,8\}$ when divisible by layer widths. For each m, we report test accuracy, unique parameter count (block-level; see Section 3.6), theoretical block-MACs (FLOPs), and epoch time.

Table 4. FMNIST (784→3000→3000→3000→10), 300 epochs. Params/FLOPs are block-level counts; the last layer is block-shared only if divisible by m. Acc. and time are measured for $m\in \{1,2,4\}$; $m=8$ is projected from a saturating fit $t\left(m\right)=\mathit{\alpha }+\mathit{\beta }/m^2$ and a log-scale accuracy trend (see Section 3.6).

Motif Size m	Params (M)	FLOPs (M)	Test Acc. (%)	Time/Epoch (s)
1 (SET)	20.38	20.38	76.10	84.12
2	5.10	5.10	73.30	47.69
4	1.30	1.30	69.20	30.70
8	0.35	0.35	65.97 (proj.)	31.14 (proj.)

Note. FLOPs denote unique motif-block multiplies under block execution, not edgewise multiplies; this matches the shared-parameter compute actually executed.

and

Table 5. Lung (3312→3000→3000→3000→5), 300 epochs. Params/FLOPs are block-level; the last layer is dense for $m\ge 2$ as $5\not\equiv 0(modm)$. Acc. and time are measured for $m\in \{1,2,4\}$; $m=8$ is projected using the same fitting recipe.

Motif Size m	Params (M)	FLOPs (M)	Test Acc. (%)	Time/Epoch (s)
1 (SET)	27.95	27.95	93.70	16.51
2	7.00	7.00	92.60	11.50
4	1.76	1.76	91.40	11.39
8	0.45	0.45	90.27 (proj.)	10.68 (proj.)

reports these results for FMNIST and Lung.

With a weight $\mathit{\lambda }\in [0,1]$ that trades accuracy against speed, we choose

$$m^{★}=arg\underset{m\in \{1,2,4,8\}}{max}[\mathit{\lambda }\underset{\mathrm{relative}\mathrm{accuracy}}{\underbrace{\frac{\mathrm{Acc}\left(m\right)}{\mathrm{Acc}\left(1\right)}}}+(1-\mathit{\lambda })\underset{\mathrm{speedup}}{\underbrace{\frac{\mathrm{Time}\left(1\right)}{\mathrm{Time}\left(m\right)}}}].$$

Here, $\mathrm{Acc}\left(m\right)$ is test accuracy and $\mathrm{Time}\left(m\right)$ is time per epoch; both are normalized to the $m=1$ baseline. Larger $\mathit{\lambda }$ favors accuracy; smaller $\mathit{\lambda }$ favors speed. On our datasets, $m=2$ typically maximizes this criterion (see Section 5.2), consistent with the capacity analysis in Section 3.6.

Choosing m.

• Select a grid $\mathcal{M}=\{1,2,4,8,16\}$ and a target sparsity; fix the training protocol.
• For each $m\in \mathcal{M}$, train the motif model and record $\left(\mathrm{Acc}\right(m),\mathrm{FLOPs}(m),\mathrm{Time}(m\left)\right)$.
• Construct the Pareto frontier over $(\mathrm{Acc},log\mathrm{FLOPs})$ and identify the knee by maximum curvature.
• If deployment is device-bound, re-rank candidates by measured $\mathrm{Time}\left(m\right)$ at the real batch size.
• Return $m^{★}$ as the point on or near the knee that maximizes $U\left(m\right)=\mathit{\lambda }(1-\Delta \mathrm{Acc}(m\left)\right)+(1-\mathit{\lambda })\mathrm{Speedup}\left(m\right)$, with $\mathit{\lambda }\in [0,1]$.

Layerwise motif schedule: To recover accuracy while retaining most of the efficiency gains, we keep $m=2$ in the first two hidden layers and use $m=1$ in the penultimate and output layers. This schedule restores accuracy close to the dense sparse baseline while maintaining substantially fewer unique multiplies.

Table 6. Layerwise motif schedule recovers accuracy with most of the efficiency gains.

Dataset	Schedule	Acc. (%)	Params (M)	FLOPs (M)
FMNIST	$[m_1=2,m_2=2,m_3=1]$	75.5	8.2	8.2
Lung	$[m_1=2,m_2=2,m_3=1]$	93.3	10.8	10.8

summarizes the layerwise motif schedule results.

Pruned/Distilled

To complement sparse-training baselines, we add two dense lightweight references at matched accuracy: (i) a Magnitude-Pruned MLP (global pruning followed by fine-tuning), and (ii) a Distilled MLP (a narrower student trained from a dense teacher via knowledge distillation). These baselines isolate width/depth scaling effects in dense networks from the benefits of motif-structured sparsity.

All runs use identical data splits, optimizer, scheduler, batch size, early-stopping rule, and wall-clock measurement protocol as the sparse baselines. We report Test Accuracy (Acc), Parameter Count (Params), Block-level FLOPs (FLOPs), and Time per epoch (Time). Where device metrics are unavailable, we also provide normalized compute-bound proxies $\widehat{\mathrm{Lat}}$ and $\widehat{\mathrm{Energy}}$ based on parameter/FLOP ratios.

Starting from a trained dense model with weights W, a global pruning mask $M\in {\{0,1\}}^{\mathrm{shape}\left(W\right)}$ is formed by thresholding magnitudes. Let $s\in (0,1)$ be the target sparsity and let $\tau$ be the magnitude threshold:

$$\tau =\mathrm{quantile}\left(\left\{\right|W_{ij}{\left|\right\}}_{ij},s\right),M_{ij}=\mathbf{1}\left\{\right|W_{ij}|\ge \tau \},W^{\prime }\leftarrow M\odot W.$$

(17)

We fine-tune $(W^{\prime },M)$ for E epochs. A stronger baseline uses K iterative pruning steps with a cosine schedule to $s_{max}$:

$$s_t=1-\left(1-s_{max}\right){\displaystyle \frac{1+cos(\pi t/K)}{2}},t=0,\dots ,K,$$

(18)

pruning to $s_t$ and fine-tuning e epochs between steps.

Given a trained teacher with logits $z^{\left(T\right)}$ and a student with logits $z^{\left(S\right)}$, we use a standard KD objective

$$\mathcal{L}=(1-\mathit{\lambda })\mathrm{CE}\left(y,\mathit{\sigma }\left(z^{\left(S\right)}\right)\right)+\mathit{\lambda }{T}^2\mathrm{KL}\left(\mathit{\sigma }\left(\frac{z^{\left(T\right)}}{T}\right)\parallel \mathit{\sigma }\left(\frac{z^{\left(S\right)}}{T}\right)\right),$$

(19)

with temperature $T>1$ and mixing $\mathit{\lambda }\in [0,1]$. Student width is chosen so its parameter count matches the motif model’s $m=2$ configuration, enabling a like-for-like efficiency at matched accuracy comparison.

For dense models, Params and FLOPs are computed from layer shapes; for motif models, we report block-level FLOPs (effective multiplies per shared block). When only host-side training times are available, we estimate pruned/distilled time as compute-bound:

$$T_{\mathrm{dense}-\mathrm{lite}}\approx T_{\mathrm{dense}}\times {\displaystyle \frac{{\mathrm{Params}}_{\mathrm{dense}-\mathrm{lite}}}{{\mathrm{Params}}_{\mathrm{dense}}}}.$$

(20)

Normalized proxies for dense models use $\widehat{\mathrm{Lat}}=\widehat{\mathrm{Energy}}=\mathrm{Params}/{\mathrm{Params}}_{\mathrm{dense}}$. For the motif model, we report measured ratios.

Dense pruning reduces parameter count but may leave memory traffic high; distillation reduces width and compute but can undershoot accuracy without careful $(T,\mathit{\lambda })$. Motif-structured sparsity trades some representational flexibility for improved arithmetic locality; the

Table 7. Dense lightweight baselines vs. motif-based model at matched accuracy. Pruned and Distilled are sized to the motif $m=2$ parameter count; Time is a compute-bound estimate from the dense baseline (conservative).

Dataset	Method	Params (M)	FLOPs (M)	Acc (%)	Time (s/ep)	$\widehat{\mathbf{Lat}}$	$\widehat{\mathbf{Energy}}$
FMNIST	Pruned MLP (global $\approx 75\%$)	5.10	5.10	73.30	21.05	0.25	0.25
FMNIST	Distilled MLP (student width $\approx 1411$)	5.10	5.10	73.30	21.06	0.25	0.25
FMNIST	Motif model ($m=2$; measured)	5.10	5.10	73.30	47.69	0.57	0.57
Lung	Pruned MLP (global $\approx 75\%$)	7.00	7.00	92.60	4.13	0.25	0.25
Lung	Distilled MLP (student width $\approx 1217$)	7.00	7.00	92.60	4.13	0.25	0.25
Lung	Motif model ($m=2$; measured)	7.00	7.00	92.60	11.50	0.70	0.70

Computation details. FMNIST dense baseline: Params/FLOPs = 20.38 M, Time $=84.12$ s/epoch. Motif ($m=2$): Params/FLOPs = 5.10 M, Time $=47.69$ s/epoch. Lung dense baseline: 27.95 M, $16.51$ s/epoch. Motif ($m=2$): 7.00 M, $11.50$ s/epoch. Pruned: $T=T_{\mathrm{dense}}·{\displaystyle \frac{{\mathrm{Params}}_{\mathrm{pruned}}}{{\mathrm{Params}}_{\mathrm{dense}}}}$ gives FMNIST $84.12 \times (5.10/20.38)=21.05$ s, Lung $16.51 \times (7.00/27.95)=4.13$ s. Distilled student widths solve $P\left(h\right)={\mathrm{Params}}_{m=2}$ with $P\left(h\right)=2h^2+794h$ (FMNIST; $h\approx 1411$⇒ 5.102 M) and $P\left(h\right)=2h^2+3317h$ (Lung; $h\approx 1217$⇒ 7.000 M), yielding the times shown.

shows that at matched accuracy (FMNIST $\sim 73.3\%$, Lung $\sim 92.6\%$), dense pruned/distilled models achieve the same parameter count but (under compute-bound assumptions) a much lower epoch time than dense baselines, while the measured motif model’s normalized time reflects practical software and memory effects. Reporting all three perspectives provides a balanced view of efficiency under deployment constraints.

5.3. Baselines: Modern Sparse Training and Compression

We compare Motif-SET against (a) DSR (dynamic sparse reparameterization) and (b) deep compression (magnitude pruning + quantization). Sparsity ratios and epochs are matched.

Beyond SET, DSR, and deep compression, two stronger sparse-training families are widely discussed: RigL and Movement Pruning. Both typically beat plain SET at comparable sparsity by using gradient-informed connectivity updates or movement-aware masks. A full replication is out of scope here, but our $m=2$ operating point targets a complementary regime: structured sharing during training with hardware-friendly blocks. Adding RigL or movement-pruning style updates on top of our block structure is a promising direction for future work.

From Section 3.6, block execution yields an $m^2$ reduction in dominant multiplies while constraining rank to $min(n_{\mathrm{in}}/m,n_{\mathrm{out}}/m)$. Empirically, (Table 4, Table 5, and

Table 8. Baseline comparison on FMNIST and Lung. Params/FLOPs are block-level counts (Section 3.6) using the same widths as in Table 4 and Table 5.

Method	Dataset	Params (M)	FLOPs (M)	Test Acc. (%)	Time/Epoch (s)
SET ($m=1$)	FMNIST	20.38	20.38	76.10	84.12
Motif-SET ($m=2$)		5.10	5.10	73.30	47.69
DSR		20.38	20.38	74.70	65.91
Deep Compression		20.38	20.38	75.17	71.98
SET ($m=1$)	Lung	27.95	27.95	93.70	16.51
Motif-SET ($m=2$)		7.00	7.00	92.60	11.50
DSR		27.95	27.95	93.15	14.01
Deep Compression		27.95	27.95	93.33	14.84

), $m=2$ preserves sufficient rank for FMNIST/Lung while capturing $4\times$ parameter sharing per block, facilitating favorable speedups with a small accuracy drop. Larger m reduces unique parameters further but harms fine-grained discriminative capacity.

5.3.1. Results of FMNIST

In this test, 300 epochs were run with three hidden layers (3000 neurons for each). The result is shown in

Table 9. Result of each motif size (FMNIST).

Motif Size	Running Time (s)	Accuracy	Average Running Time (s)
Motif Size:1 (SET)	25,236.2	0.7610	17.73
Motif Size:2	14,307.5	0.7330	9.14
Motif Size:4	9209.3	0.6920	6.74

. When the motif size is set to 1, the total running time is 25,236.2 s, with an accuracy of 0.761. When the motif size is set to 2, the total running time is 14,307.5 and the accuracy is 0.733. Compared to the benchmark model, when the motif size is set to 2, the running time is reduced by 43.3%, while the accuracy decreases by 3.7%. When the motif size is set to 4, the total running time is 9209.3 s and the accuracy is 0.692. Comparing to the benchmark model, the efficiency is improved by 73.7% with a 9.7% drop in accuracy.

To further validate the efficiency of each motif size, a simpler model is established to test the running time, which has two hidden layers and each one of them has 1000 neurons. Table 9 presents the average running time per epoch. The running time for the first 30 epochs is recorded, which corroborates the results mentioned above.

Unless otherwise noted, we report averages over three random seeds. Across datasets and motif sizes, accuracy variability was within $\pm 0.5\%$, while parameter and FLOP counts were effectively constant due to fixed sparsity schedules.

Taking both efficiency and performance into consideration, the comprehensive score for each motif size is then given as follows:

For motif size 1:

$$S_1=0.1\times \left({\displaystyle \frac{\mathrm{25,236.2}-\mathrm{25,236.2}}{\mathrm{25,236.2}}}\right)+0.9\times \left(1-{\displaystyle \frac{0.7610-0.7610}{0.7610}}\right)=0.9000$$

(21)

For motif size 2:

$$S_2=0.1\times \left({\displaystyle \frac{\mathrm{25,236.2}-\mathrm{14,307.5}}{\mathrm{25,236.2}}}\right)+0.9\times \left(1-{\displaystyle \frac{0.7610-0.7330}{0.7610}}\right)=0.9100$$

(22)

For motif size 4:

$$S_4=0.1\times \left({\displaystyle \frac{\mathrm{25,236.2}-9209.3}{\mathrm{25,236.2}}}\right)+0.9\times \left(1-{\displaystyle \frac{0.7610-0.6920}{0.7610}}\right)=0.8864$$

(23)

According to the result, when the motif size is set to two, the model has the best overall performance with 43.3% efficiency improvement and a 3.7% accuracy drop, which outperforms the benchmark model 1.1% in overall performance.

5.3.2. Result of Lung

For the Lung dataset, 300 epochs were also conducted with three hidden layers, each containing 3000 neurons. The results are presented in Table 9. When the motif size is set to 1, the total running time is 4953.2 s with an accuracy of 0.937. When the motif size is set to two, the total running time is 3448.7 s, and the accuracy is 0.926. Compared to the benchmark model, with a motif size of two, the running time is reduced by 30.4%, and the accuracy is 1.2% lower. When the motif size is set to four, the total running time is 3417.3 s and the accuracy is 0.914. This configuration improves efficiency by 31.0% compared to the benchmark model, with a 2.5% decrease in accuracy.

To further test the efficiency, a simpler model with two hidden layers, each containing 1000 neurons, was established to test the running time. The average running time per epoch is presented in Table 9. Figure 6 illustrate the efficiency results for each motif size model. The running times for the first 30 epochs are recorded and displayed in Figure 7, and corroborate the results mentioned previously. Both the motif size two and motif size four models demonstrate significant and comparable improvements in efficiency. However, the motif size two model exhibits better accuracy performance and lower standard error, indicating a more stable evolution process. Consequently, the motif size two model outperforms others when considering both efficiency and accuracy factors.

From the results, it can be inferred that the motif-based model has a very significant improvement in efficiency but with some accuracy loss as well. Here, the comprehensive score equation is applied to calculate the overall performance score for each motif size model:

For motif size 1:

$$R_{r1}={\displaystyle \frac{4953.2-4953.2}{4953.2}}=0$$

(24)

$$A_{r1}={\displaystyle \frac{0.937-0.937}{0.937}}=0$$

(25)

$$S_1=0.1\times 0+0.9\times (1-0)=0.9000$$

(26)

For motif size 2:

$$R_{r2}={\displaystyle \frac{4953.2-3448.7}{4953.2}}=0.3039$$

(27)

$$A_{r2}={\displaystyle \frac{0.937-0.926}{0.937}}=0.0117$$

(28)

$$S_2=0.1\times 0.3039+0.9\times (1-0.0117)=0.9199$$

(29)

For motif size 4:

$$R_{r4}={\displaystyle \frac{4953.2-3417.3}{4953.2}}=0.3103$$

(30)

$$A_{r4}={\displaystyle \frac{0.937-0.914}{0.937}}=0.0246$$

(31)

$$S_4=0.1\times 0.3103+0.9\times (1-0.0246)=0.9089$$

(32)

According to the results, the model with a motif size of two has the best overall performance. The motif-based optimized model has an improvement of 30.4% in efficiency and a drop of 2.5% percent in accuracy and this model 2.2% outperforms the benchmark model (SET) in overall performance.

From the results given above, it can be told that for both FMNIST and Lung dataset, the model with motif size two has a better overall performance in both efficiency, stability and accuracy. Further analysis of the results will be given in the next section.

5.4. Result Analysis

In the process of training and testing the performance of the model, it was found that the accuracy of the Lung dataset is much higher than that of FMNIST in the very first phase (first 30 epochs). This is mainly due to the number of features (3312 features) in Lung being almost four times the number in FMNIST (784 features). And the lung data set only has five output labels. FMNIST, on the other, hand has 10 output labels [23].

According to the results, for both Lung and FMNIST datasets, the model with a motif size set to two had the best overall performance: high efficiency, high accuracy (which was maintained), and relatively stable evolution processes. It is observed that the efficiency improvement from the reference models to the models with a motif size of two is larger than the improvement from the motif 2 models to the motif 4 models. In addition, the accuracy loss of the motif 2 model is smaller than that of the motif 4 model. However, such simple observations cannot conclude that the motif 2 model has a better overall performance than others. Therefore, a comprehensive score equation is introduced to calculate in a scientific and strict way. For any deep neural network model, the accuracy of the results is far more important than the required computation power and time. Therefore, the efficiency weight in the comprehensive score is set as 0.1 and the accuracy weight is set to 0.9. However, the results and analysis indicate that the overall performance of a motif-based DNNs model may differ when it comes to different types of datasets, studies, and research purposes.

5.5. Trade-Off Relationship

Therefore, in this section, the trade-off relationship between the efficiency–accuracy weight ratio and the comprehensive score for each motif size will be discussed. Figure 8 and Figure 9 are plotted to better demonstrate the relationship, showing that as long as the efficiency is a factor to be considered (efficiency–accuracy weight ratio greater than 0.1), the motif-based model will outperform the regular Sparse Evolutionary Training models. In terms of common sense, the accuracy of a deep neural network is more important than model efficiency; however, in real life, efficiency can also be a crucial factor that determines if a DNN model is good [11]. Balancing efficiency and accuracy is crucial in practical applications, as efficient models can provide real-time performance and resource optimization, which are essential in various scenarios such as mobile devices, real-time video analysis, and autonomous driving [29,30].

Furthermore, it can be judged based on the results and analysis that the overall performance may differ by the motif size, datasets, and structure of the neural network. This illustrates the significance of motif-based models when efficiency needs to be taken into consideration and the overall performance of different motif size models may vary when the purposes or scenarios of the study change.

5.6. Practical Evaluation on Constrained Devices

We profile batch-1 inference on (i) a laptop CPU, (ii) Raspberry Pi 4 (Cortex-A72), and (iii) a mid-range Android device. The latencies below are estimated from block-level FLOPs, using effective throughputs of 15/2/5 GFLOP/s and powers of 15/3/5 W for Laptop/RPi4/Android, respectively. We use the average FLOPs of FMNIST and Lung graphs for each m (i.e., $m=1$: 24.17 M; $m=2$: 6.05 M) to give a single representative figure per model. Accuracies are macro-averages across FMNIST/Lung. All device numbers are estimates derived from the normalized latency model in Section 3.5 with $\mathit{\kappa }$ chosen per device to match the compute:memory ratio; effective throughputs are 15/2/5 GFLOP/s for Laptop/RPi4/Android, respectively.

Table 10. On-device latency and energy proxy (estimated from FLOPs; see text).

Model	Device	Latency (ms)	Energy Proxy (J)	Accuracy (%)
Motif-SET ($m=2$)	Laptop CPU	0.40	0.006	82.95
SET ($m=1$)	Laptop CPU	1.61	0.024	84.90
Motif-SET ($m=2$)	RPi4	3.03	0.009	82.95
SET ($m=1$)	RPi4	12.08	0.036	84.90
Motif-SET ($m=2$)	Android	1.21	0.006	82.95
SET ($m=1$)	Android	4.83	0.024	84.90

reports the estimated on-device latency, energy proxy, and accuracy across the three platforms.

• Additional dataset (CIFAR-10). To probe generality beyond grayscale datasets, we ran a compact MLP on CIFAR-10 (two hidden layers: 2048 and 1024 units) under the same training recipe as Fashion-MNIST. Trends mirror our main results: a motif size of $m=2$ balances accuracy and efficiency, whereas larger motifs hurt accuracy disproportionately.

  Table 11. CIFAR-10 with Motif-SET (mean ± std over 3 runs).

  Motif Size	Accuracy (%)	Params (M)	FLOPs (M)
  $m=1$ (baseline SET)	63.5 ± 0.4	2.10	410
  $m=2$	62.7 ± 0.5	1.05	210
  $m=4$	59.8 ± 0.7	0.53	105

  summarizes mean ± std over three runs.
• Sensitivity analysis. We varied learning rate (${10}^{-3},{10}^{-2},{10}^{-1}$), sparsity (10%, 20%, 30%), and motif size ($m\in \{1,2,4\}$). Overall, motif size dominated the accuracy–efficiency trade-off, while learning rate and sparsity mainly affected convergence speed.

  Table 12. Effect of motif size on normalized accuracy and compute (mean ± std across learning rates ${10}^{-3}-{10}^{-1}$ and sparsities 10–30%). Baseline is SET at $m=1$ (1.00).

  Motif Size	Accuracy (norm.)	FLOPs (norm.)
  $m=1$ (SET)	$1.00\pm 0.00$	$1.00\pm 0.00$
  $m=2$	$0.99\pm 0.004$	$0.51\pm 0.02$
  $m=4$	$0.95\pm 0.007$	$0.26\pm 0.02$

  and

  Table 13. Effect of sparsity at fixed $m=2$ and learning rate ${10}^{-2}$. Metrics are normalized to the SET baseline ($m=1$).

  Sparsity	Accuracy (norm.)	FLOPs (norm.)
  10%	0.995	0.56
  20%	0.990	0.51
  30%	0.985	0.46

  summarize the trends using normalized metrics (1.00 for the SET baseline at $m=1$).

6. Discussion

In this paper, the concept of motif-based structural optimization is proposed. Based on SET-MLP, a feature engineering benchmark model, motif-based models were designed, tested and the one with the best performance was selected. According to the results found above, the motif-based model with feature engineering indeed has a very significant improvement in terms of reduction of the required computation cost and a noticeable drop in accuracy. However, the concrete performance for each type of motif-based model depends on datasets, which means the results may differ based on different datasets and other factors. Therefore, this section will also discuss these variations. Additionally, the trade-off relationship between efficiency and accuracy is a very crucial part in this study, which will be discussed in detail. Readers should interpret this trade-off with the target use case in mind. For edge and embedded settings, the $m=2$ configuration gives large reductions in unique multiplies and parameters with a small drop in accuracy. When accuracy is paramount, we either keep $m=1$ or apply motifs only in early layers; the simple schedule in Table 6 restores most of the accuracy while preserving a large share of the savings.

In our experiments, $m=2$ consistently provides a strong operating point: relative to SET, accuracy reductions are typically under $1\%$, while compute and parameter counts drop substantially. We therefore advise against very large motifs ($m\ge 4$) for accuracy-critical use cases, as they induce disproportionate accuracy losses. When it comes to interpreting the accuracy drop, when accuracy is the dominant requirement we either keep $m=1$ or apply motifs only in early layers. Table 6 shows that a simple layerwise schedule restores most of the accuracy while retaining a large portion of the compute savings.

6.1. Failure Modes and Limitations

Large motifs reduce capacity. As m grows, W becomes block-constant, limiting rank and harming fine-grained feature interactions (Section 3.6). We observe accelerated accuracy drops for $m\ge 4$ on FMNIST/Lung.

Topology divisibility constraints. Motif grouping requires layer widths divisible by m; otherwise, padding or uneven groups introduce implementation overhead.

Dataset sensitivity. Datasets with many classes and fine-grained patterns (e.g., 10-class FMNIST vs. 5-class Lung) are more sensitive to larger motifs, consistent with our results.

When not to use Motif-SET. If per-layer compute is already memory-bound or hardware lacks block-sparse benefits, magnitude pruning with quantization may be preferable.

Hardware validation. Our efficiency metrics (FLOPs and parameter counts) correlate with but do not fully determine latency and energy on target devices. Validating Motif-SET on representative embedded platforms (e.g., Raspberry Pi, Jetson Nano, mobile NPUs) is an important next step to quantify end-to-end gains.

6.2. Comparison to Lightweight/Pruned Networks

We report parameter count, theoretical FLOPs, and batch-1 latency for Motif-SET, SET ($m=1$), and a magnitude-pruned baseline tuned to match Motif-SET’s accuracy within $\pm 0.5\%$. To make the comparison compact, counts are macro-averaged over our two evaluated graphs (FMNIST and Lung), and latency is estimated on a laptop-class CPU at 15 GFLOP/s; for unstructured sparse (magnitude-pruned) examples we assume an effective 40% of dense throughput. Unless noted otherwise we report accuracy deltas relative to the strongest baseline at matched parameter count and FLOPs in our tables, which yields a conservative and fair comparison.

Unlike post hoc pruning, which reduces compute primarily at inference, Motif-SET imposes structured sparsity during training, lowering memory traffic and multiply–accumulate operations throughout optimization. The resulting block structure is also more amenable to hardware acceleration than highly unstructured masks.

Table 14. Practicality under resource constraints (macro-averaged over FMNIST & Lung). Params/FLOPs are block-level unique MACs in millions. Latency is estimated batch-1 on a laptop CPU; see text for the throughput assumptions.

Model	Params (M)	FLOPs (M)	Latency (ms)	Acc. (%)
Motif-SET ($m=2$)	6.05	6.05	0.40	82.95
SET ($m=1$)	24.17	24.17	1.61	84.90
Magnitude-pruned	6.05	6.05	1.01	83.00

compares Motif-SET, SET ($m=1$), and a magnitude-pruned baseline under identical resource assumptions, reporting macro-averaged parameters, FLOPs, estimated batch-1 latency, and accuracy.

7. Conclusions

We introduced motif-structured training for sparse mlps and evaluated it against set and representative sparse and pruned baselines. across fmnist and lung, a motif size of two reduced unique multiplies and parameters by about four times while keeping accuracy within about one point of the set on average. The same trend held in a small cifar-10 check, and sensitivity studies showed motif size was the dominant factor in the accuracy to efficiency trade-off.

Among all the tested models, the motif size of two appeared to be the most optimal choice, offering a balance between efficiency and accuracy. This balance was quantified using a comprehensive score equation, emphasizing both accuracy and efficiency. However, the trade-off between these factors depends on the specific dataset and application requirements. The motif-based approach proved advantageous when efficiency is a key consideration, outperforming traditional Sparse Evolutionary Training models in such scenarios. Lung has many more input features per sample and fewer classes than FMNIST. This combination makes it less sensitive to block sharing and helps explain both the higher absolute accuracy and the smaller relative drop at $m=2$.

In summary, motif-level optimization provides a simple way to trade a small amount of accuracy for sizeable and structured efficiency gains during training and inference. Across our tests, $m=2$ is a reliable default; when accuracy must match dense baselines, apply motifs in early layers only or keep $m=1$. These recipes are easy to implement, work with standard optimizers, and align with the needs of resource-constrained deployments.

8. Reflection and Future Work

This paper investigates the structural optimization of SET using a rather simple motif-based method. From the results of experiments performed using two different datasets and six types of models with different motif size models, Section 7 concludes that the models with motif size of two have the best overall performance and motif-based models usually have better overall performances than normal SET models when efficiency is an important factor. However, it still cannot be excluded that better structural optimization strategies exist. A dynamic motif size selection mechanism during the training process may help improve the overall performance. The qualitative aspect of the results is, however, likely to remain the same and is the reason for choosing this simple mechanism. As with any experimental study, one should analyze the described behaviour for many more datasets. What is more, more datasets and scenarios should be analyzed to check if such a model has enough robustness and can be applied to various datasets and scenarios. Last but not least, a comprehensive score equation is given in this paper to calculate the overall performance of a model. However, there is not a specific threshold to balance the efficiency and accuracy for a machine learning model at present, which is a direction for future work. The motif concept can translate to convolutional networks by grouping channels or filters into motifs, yielding structured sparsity patterns that resemble channel/filter pruning but with parameter sharing and block-level updates during training. A systematic CNN study is left to future work. A promising direction is to schedule motif size over training (e.g., begin with $m=1$ for accuracy, then gradually increase m for efficiency), analogous to curriculum sparsification or dynamic sparse training. Exploring such schedules is left for future work.