A Supernet-Only Framework for Federated Learning in Computationally Heterogeneous Scenarios

Abstract

Federated learning is effective for Internet of Things data privacy and non-independent and identically distributed issues but not device heterogeneity. Neural Architecture Search can alleviate this by constructing multiple model structures to optimize federated learning performance across diverse edge devices. However, existing methods, whether lightweight networks or client grouping, face a tradeoff between scaling to larger federations and utilizing more powerful structures. We decompose residual network blocks, reformulating them as a Neural Architecture Search task. Furthermore, we propose a method for reinterpreting any sequential architecture into a supernet and developed a training pipeline tailored to this reinterpretated architecture, mitigating this frustrating tradeoff. We conduct pretraining on ImageNet1K and federated training on the CIFAR-100, CIFAR-10, and CINIC-10 datasets under both the ring-based federated learning and FedAvg framework. In less constrained environments, our method maintains performance comparable to another top-two method, which varies across experimental settings, while maintaining a margin of at least 1% Top-1 accuracy over the third-best method. Under balanced settings, our method outperforms the second-best approach by more than 1%, and this advantage increases to over 5% as the task difficulty further rises. Under the most challenging setting, our method outperformed AdaptiveFL, a state-of-the-art dynamic network method for federated learning, by 18.3% on CIFAR-100 with 100 clients under a ResNet backbone.

1. Introduction

The Internet of Things (IoT) [1] has become an integral part of daily life and industrial processes, generating vast amounts of distributed data from edge devices. However, leveraging these data presents significant challenges, as follows: privacy concerns arise from sensitive information such as smart home data, medical records, or industrial trade secrets; edge device data often deviate from the Independent and Identically Distributed (IID) assumption due to variations in user behavior or institutional demographics; and edge devices exhibit performance heterogeneity caused by differences in design and technological generations. To address these challenges, Federated Learning (FL) [2] has emerged as a promising approach. FL mitigates privacy risks by performing local model training on edge devices and aggregating updates on a central server, avoiding sensitive data transmission. Additionally, FL enhances usability by adapting to non-Independently and Identically Distributed (non-IID) data distributions through localized training.

Despite the potential of FL, its uniform network structures struggle to accommodate device heterogeneity and integrate lightweight techniques [3] such as pruning, sparsification, and quantization to reduce model complexity and align with varying device capabilities. While these techniques partially alleviate the device heterogeneity problem, their compression capabilities remain limited, leaving this issue only partially addressed. Recent advancements have integrated Neural Architecture Search (NAS) [4] with FL to enhance machine learning services on heterogeneous edge devices, giving rise to Federated NAS (Fed-NAS) [5]. Common approaches include searching for networks of varying sizes or designing multi-scale networks through supernets. However, these methods primarily focus on designing networks for different scales of clients without ensuring shared features among all clients for collective benefits, achieving only partial FL objectives.

The problem is that using the largest model every client can afford and forming a federated cluster with all clients are difficult to achieve simultaneously. In contrast to prior interpretations of residual connections [6], we introduce a new structural perspective that enables reformulating residual networks as NAS-compatible supernets. We observe that the residual networks can be represented as a sum of outputs from all nonlinear transformations, making these weights learnable, which enables us to convert the residual network into a NAS task. By applying transformations to each stage separately, the size of the sub-network of each client could be determined by a truncation parameter. As long as we keep the entire supernet intact, our model can freely switch between different sub-networks. Although our method introduces a small number of additional parameters corresponding to the number of residual units, these are used to dynamically normalize the structural weights during training. This enables consistent output distributions across sub-networks with different truncation lengths and effectively mitigates the distribution shift problem commonly observed in dynamic neural networks. We propose a novel reinterpretation of residual connections, showing that residual networks can be naturally viewed as continuous supernets, where each residual block contributes as a weighted component in a unified architecture space. This perspective transforms the role of residual connections from simple identity shortcuts into explicit architecture-controlling mechanisms, enabling a new class of NAS-compatible designs grounded in standard residual architectures. Unlike conventional NAS supernets where architectural decisions select between competing candidate operations, our reinterpreted supernet only allows truncation from left to right within each stage. By allowing only left-to-right truncation within each stage, every sub-network becomes a deterministic prefix of the full residual sequence, avoiding operation-level conflicts and enabling full parameter sharing—a property that conventional NAS and FedNAS designs inherently lack due to their search-space structure.

To train this supernet, we propose the Single-Path Random Quit (SPRQ) method, in which different exit lengths are randomly selected at each iteration to simulate different network sizes for various clients. SPRQ not only enhances performance but also reduces pretraining time by 24%, significantly improving training efficiency in large-scale settings. Moreover, as the SPRQ method suffers from initialization difficulties, we suggest using the two-stage optimization approach proposed in the DARTS [7] report to perform a warm-up operation. It is evident that our Supernet-Only framework is independent of FL and can be applied to most common FL frameworks.

We perform our pretraining scheme on ImageNet1K [8] and compare the performance of the Supernet-Only framework with that of using lightweight networks and clustering clients to build multiple federations, under identical client communication and computation constraints, on the CIFAR-100, CIFAR-10 [9], and CINIC-10 [10] datasets, as well as on ring-based FL and FedAvg. Our method achieves remarkable results across all datasets and federated learning methods, and its advantages become more pronounced as the generalization requirements of the dataset increase. We also validate the necessity of our proposed SPRQ method and the warm-up method in the pretraining methods. In more challenging scenarios, such as those involving CIFAR-100, large-scale federated settings, or smaller backbone networks, our method consistently achieves more than a 1% improvement in Top-1 accuracy, and it exceeds the second-best method by over 5% in all CIFAR-100 experiments under the FedAvg framework. In less constrained settings, including stronger backbones, smaller-scale federated configurations, and simpler datasets, our method performs on par with the best alternative method in each case, while maintaining a margin of more than 1% over the third-best method.

We summarize our contributions as follows. Firstly, we revisit the role of residual connections and propose a method to reformulate any network composed of residual units into a supernet for NAS tasks, effectively reinterpreting the optimization task of any residual network into a NAS task. Secondly, we propose the Single-Path Random Quit (SPRQ) pretraining method to simulate structural heterogeneity in federated learning, and we subsequently design a complete training pipeline based on it. Finally, we outline several common paths in Fed-NAS methods, validate the superiority of our proposed method across multiple datasets, and further compare it with AdaptiveFL [11], a state-of-the-art dynamic FL approach.

The remainder of this paper is organized as follows: Section 2 reviews the background and related work. Section 3 describes the motivation for our approach and details our methods, including the supernet framework and the training pipeline. Section 4 evaluates the proposed method through experiments, and Section 5 concludes our contributions and discusses future research directions.

2. Background and Related Work

This study focuses on the intersection of FL and NAS, addressing the challenges of lightweight learning and integrating lightweight learning, FL, and NAS.

2.1. Federated Learning

Federated Learning (FL) is a distributed machine learning paradigm that enables multiple devices to collaboratively train a global model without sharing raw data. A central server orchestrates the model, while clients train locally using their data and send updates—typically gradients or trained model parameters—to the server in a privacy-preserving manner. Assume there are K clients (participants), and each client k holds a local dataset $D_k$. The goal is to optimize the global model parameter $\mathcal{P}$ by minimizing the global loss function $\mathcal{L}$ as the weighted sum of K local loss ${\mathcal{L}}_k$:

$$\underset{\mathcal{P}}{min}\mathcal{L}\left(\mathcal{P}\right)=\underset{\mathcal{P}}{min}\sum _{k=1}^K{\displaystyle \frac{|D_k|}{{\sum }_{i=1}^K\left|D_i\right|}}{\mathcal{L}}_k\left(\mathcal{P}\right)$$

(1)

FL encompasses several fundamental approaches [2], such as ring-based aggregation FL [12], gradient-based FedSGD [13] and model-based aggregation FL [14]. FL can mitigate the issues of data non-IID and data privatization. However, it generally requires different clients to use the same network architecture, making it difficult to handle the challenge of highly heterogeneous devices.

Some studies attempt to address the issue of device heterogeneity in federated learning by dynamically adjusting model complexity through pruning strategies. They send networks of different sizes to different clients, where networks of varying sizes share a portion of the channels [15], the early layers [16], or a combination of both [11,17]. Since it is necessary to train directly on networks of different sizes, they face various challenges such as optimal decision and generalization issues [18].

2.2. Neural Architecture Search

Neural Architecture Search (NAS) [4] is an automated approach for optimizing neural network structures, framed as a bi-level optimization problem. The upper-level objective minimizes the validation loss ${\mathcal{L}}_{\mathrm{val}}$ parameterized by model weights $\mathcal{P}$, while the lower-level objective minimizes the training loss ${\mathcal{L}}_{\mathrm{train}}$ given the architecture hyperparameters $\mathcal{W}$:

$$\underset{\mathcal{W}}{min}{\mathcal{L}}_{\mathrm{val}}({\mathcal{P}}^{*}\left(\mathcal{W}\right),\mathcal{P})\mathrm{s}.\mathrm{t}.{\mathcal{P}}^{*}\left(\mathcal{W}\right)=arg\underset{\mathcal{P}}{min}{\mathcal{L}}_{\mathrm{train}}(\mathcal{P},\mathcal{W})$$

(2)

NAS is extensively used to design efficient lightweight neural networks tailored for resource-constrained applications. Advances in NAS have led to the development of numerous lightweight architectures, such as Effnet [19] and MobileNet [20], for resource-limited devices. The gradient-based NAS framework [7,21] is among the most popular NAS frameworks, assigning parameters to candidate modules and alternately optimizing architecture and weights during a single training process. However, ring-based NAS faces challenges such as excessive memory consumption and unreasonable context coupling, which can be effectively addressed through uniform random sampling, named Single-Path One-Shot (SPOS) [22].

2.3. Federated Neural Architecture Search (Fed-NAS)

Fed-NAS [23] research primarily addresses two key challenges in terms of performance. The first focuses on developing high-performance neural networks from diverse client data while preserving privacy. The second emphasizes efficiently using NAS to tailor architectures for varying client capabilities under privacy and computational constraints.

Some studies focus on the first challenge, performing device-aware or offline NAS on high-performance private clients while employing lightweight methods to reduce client burdens during the search process [24]. Some studies explore potential approaches to address the second challenge. However, the resulting sub-networks may become excessively downsized [25], or performance within each federation may be enhanced through client clustering methods [26,27]. Most studies fail to effectively address the issue of computational heterogeneity.

Recent research attempts to resolve these limitations by performing structure searches tailored to resource-constrained clients. For example, we optimize unique sub-networks of each client extracted from a supernet [28] or separate clients used for training or inference [29]. Other approaches partition the model into shared and personalized units, aggregating only the shared units during the federated process [30,31]. However, these methods more or less face a trade-off between federation size and network size.

3. Method

We propose a method to adapt network scales to heterogeneous client devices while maintaining compatibility across varying computational capacities in federated learning. To address this challenge, we introduce the Supernet-Only framework, which comprises the following two components: (i) a reinterpretation of residual sequential networks as a supernet, and (ii) a dedicated training pipeline.

3.1. Reinterpret Approach

In this section, we introduce our motivations and our approach to reinterpreting a residual sequential network into a supernet of a NAS task.

Consider arbitrary residual sequential network $\mathcal{N}$ with parameters $\mathcal{P}$. Each residual unit ${\mathcal{N}}_i$ of $\mathcal{N}$ can be decomposed into two parts as follows: a nonlinear transformation ${\mathcal{N}}_i^{\mathrm{Raw}}$ and an identity mapping $\mathcal{I}$, as follows:

$${\mathcal{N}}_i={\mathcal{N}}_i^{\mathrm{Raw}}+\mathcal{I}.$$

(3)

Let ${\mathcal{N}}_l^{\mathrm{pre}}$ represent the preceding net of the l-th layer which is the combination of the first $l-1$ layers, and adopt ${\mathcal{N}}_1^{\mathrm{Pre}}={\mathcal{N}}_0^{\mathrm{Pre}}={\mathcal{N}}_0^{\mathrm{Raw}}=\mathcal{I}$. The full residual network $\mathcal{N}$, composed of L layers, can then be recursively reformulated as follows:

$$\begin{array}{cc}\hfill \mathcal{N}& =({\mathcal{N}}_L^{\mathrm{Raw}}+\mathcal{I})\circ {\mathcal{N}}_L^{\mathrm{Pre}}\hfill \\ \hfill & ={\mathcal{N}}_L^{\mathrm{Raw}}\circ {\mathcal{N}}_L^{\mathrm{Pre}}+{\mathcal{N}}_L^{\mathrm{Pre}}\hfill \\ \hfill & ={\mathcal{N}}_L^{\mathrm{Raw}}\circ {\mathcal{N}}_L^{\mathrm{Pre}}+({\mathcal{N}}_{L-1}^{\mathrm{Raw}}+\mathcal{I})\circ {\mathcal{N}}_{L-1}^{\mathrm{Pre}}\hfill \\ \hfill & ={\mathcal{N}}_L^{\mathrm{Raw}}\circ {\mathcal{N}}_L^{\mathrm{Pre}}+{\mathcal{N}}_{L-1}^{\mathrm{Raw}}\circ {\mathcal{N}}_{L-1}^{\mathrm{Pre}}+{\mathcal{N}}_{L-1}^{\mathrm{Pre}}\hfill \\ \hfill & \cdots \hfill \\ \hfill & ={\mathcal{N}}_L^{\mathrm{Raw}}\circ {\mathcal{N}}_L^{\mathrm{Pre}}+{\mathcal{N}}_{L-1}^{\mathrm{Raw}}\circ {\mathcal{N}}_{L-1}^{\mathrm{Pre}}+\cdots +{\mathcal{N}}_2^{\mathrm{Raw}}\circ {\mathcal{N}}_2^{\mathrm{Pre}}+{\mathcal{N}}_2^{\mathrm{Pre}}\hfill \\ \hfill & ={\mathcal{N}}_L^{\mathrm{Raw}}\circ {\mathcal{N}}_L^{\mathrm{Pre}}+{\mathcal{N}}_{L-1}^{\mathrm{Raw}}\circ {\mathcal{N}}_{L-1}^{\mathrm{Pre}}+\cdots +{\mathcal{N}}_2^{\mathrm{Raw}}\circ {\mathcal{N}}_2^{\mathrm{Pre}}+{\mathcal{N}}_1\hfill \\ \hfill & ={\mathcal{N}}_L^{\mathrm{Raw}}\circ {\mathcal{N}}_L^{\mathrm{Pre}}+{\mathcal{N}}_{L-1}^{\mathrm{Raw}}\circ {\mathcal{N}}_{L-1}^{\mathrm{Pre}}+\cdots +{\mathcal{N}}_2^{\mathrm{Raw}}\circ {\mathcal{N}}_2^{\mathrm{Pre}}+{\mathcal{N}}_1^{\mathrm{Raw}}+\mathcal{I}\hfill \\ \hfill & =\sum _{l=0}^L{\mathcal{N}}_l^{\mathrm{Raw}}\circ {\mathcal{N}}_l^{\mathrm{Pre}}\hfill \end{array}$$

(4)

Let $x_l$ denote the input to model layer l. The result of the nonlinear transformation is denoted by $y_l^{\mathrm{raw}}={\mathcal{N}}_l^{\mathrm{Raw}}\left(x_l\right)$, and the output of layer l is denoted as $y_l={\mathcal{N}}_l\left(x_l\right)=x_l+y_l^{\mathrm{raw}}$. The input to this layer is both the output of the previous layer, $x_l=y_{l-1}$, and also the output of the preceding net of this layer $x_l={\mathcal{N}}_l^{\mathrm{pre}}\left(x\right)$. Then, the output of each term ${\mathcal{N}}_l^{\mathrm{Raw}}\circ {\mathcal{N}}_l^{\mathrm{pre}}$ in Equation (4) can be represented as follows:

$${\mathcal{N}}_l^{\mathrm{Raw}}\left({\mathcal{N}}_l^{\mathrm{pre}}\left(x\right)\right)=y_l^{\mathrm{Raw}}$$

(5)

Substituting Equation (5) into Equation (4), the output of any residual sequential network can be represented as the sum of the outputs of the nonlinear transformations of each residual unit as follows:

$$\mathcal{N}\left(x\right)=\sum _{l=0}^L{y}_l^{\mathrm{Raw}}$$

(6)

Unlike the conventional view that networks with residual connections are sequential architectures at coarser scales, Equation (6) instead forms a parallel architecture. In discussions surrounding residual networks, some studies [32,33] suggest that low-quality nonlinear transformations can degrade, thereby minimizing their impact on the learning process. Inspired by this, we realized that there exists a hidden weight that determines the contribution ratio of each nonlinear transformation. Defining this parameter as weights, the network output can be expressed as a weighted sum of contributions from all layers $\mathcal{N}\left(x\right)={\sum }_{l=0}^L{\mathcal{W}}_l·y_l^{\mathrm{Raw}}$, where ${\mathcal{W}}_l,(l=0,1,\cdots ,L)$ are the weights satisfying non-negativity and normalization constraints. Furthermore, the model can be represented as follows:

$$\mathcal{N}=\sum _{l=0}^L{\mathcal{W}}_l·{\mathcal{N}}_l^{\mathrm{Raw}}\circ {\mathcal{N}}_l^{\mathrm{Pre}}$$

(7)

Since the weights $\mathcal{W}=\{{\mathcal{W}}_0,{\mathcal{W}}_1,\cdots ,{\mathcal{W}}_L\}$ have been separated from the original training task, the training of this hidden parameter can be considered a meta-learning task ${argmin}_{\mathcal{W}}{\mathcal{L}}_{\mathrm{val}}\left({\mathcal{P}}^{*}\left(\mathcal{W}\right)\right)$ beyond the original network parameter optimization ${argmin}_{\mathcal{P}}{\mathcal{L}}_{\mathrm{train}}\left({\mathcal{W}}^{*}\left(\mathcal{P}\right)\right)$.

To enforce the normalization and non-negativity constraints on the weights, we introduce the canonical link function of the Generalized Linear Model (GLM) with multinomial distribution logits as the bridge between the weights and the meta-learning parameters. Let ${\mathcal{W}}^{\mathrm{Raw}}$ denote the parameters corresponding to the architecture weights. The final weights can then be computed as follows:

$$\mathcal{W}=\mathrm{Softmax}\left({\mathcal{W}}^{\mathrm{Raw}}\right).$$

(8)

Figure 1a illustrates the computational structure of each stage in the reinterpreted supernet.

We observe that the meta-learning component of this two-step optimization process corresponds to neural architecture optimization, while the original task constitutes parameter optimization. This residual network formulation naturally aligns with the bi-level optimization structure commonly adopted in neural architecture search, as represented in Equation (2).

In manually designed networks such as ConvNeXt and Transformers, as well as in searched architectures like EfficientNet, the most common macro network structure typically consists of three parts. The network begins with a preprocessing component, such as an Embedding layer or a Stem layer. This is followed by multiple stages, each composed of a downsampler followed by a feature extraction block, where the feature extraction block is entirely composed of a sequence of residual units. The network ends with one or more detection heads. If each stage is rewritten in the form of Equation (7), while keeping all other components unchanged, the sequential network can be reinterpreted as a NAS supernet.

Although the model may seem to allow the arbitrary selection of intermediate outputs from nonlinear transformations, each output, in fact, depends sequentially on all preceding layers. Once a layer is dropped, all subsequent layers become uncomputable. As a result, sub-networks can only be formed through truncation, and the structure of each truncated stage in the sub-network is shown in Figure 1b.

In the preceding text, we have demonstrated that all architectures with residual connections can be reinterpreted as a NAS problem through a single parameterization step. However, when reinterpreting an arbitrary network with residual connection as a NAS problem, we observe that two issues still need to be addressed. Firstly, modern network architectures are typically composed of multiple stages, where each stage contains one downsampling module and several residual blocks such as ConvNeXt [34] and SWIN Transformer [35]. Secondly, the omission of certain layers alters the distribution of the aggregated representations, which consequently leads to a shift in the distribution of the final softmax output as follows: $\mathrm{Softmax}\left(\mathcal{N}\right(x\left)\right)$.

Although most networks are not entirely composed of residual units, the primary feature extraction within each stage is carried out by a sequence of residual units. Therefore, the transformation described by Equation (7) can be applied to each stage individually, with truncation performed within each stage accordingly. Assume the input to the s-th stage, which contains $L_s$ residual units, is denoted as $x_s$ and the output as $y_s$. The result of a downsampling operation serves as the input $x_{s,1}$ to the first residual unit. The input to the l-th residual unit ${\mathcal{N}}_{s,l}$ within this stage is represented by $x_{s,l}={\mathcal{N}}_{s,l}^{\mathrm{Pre}}\left(x_{s,1}\right)$, where ${\mathcal{N}}_{s,l}^{\mathrm{Pre}}$ denotes the preceding net of the l-th layer within stage s. Therefore, the featuring of this stage can be represented as follows:

$$y_s=\sum _{l=1}^{L_s}{\mathcal{W}}_{s,l}·{\mathcal{N}}_{s,l}^{\mathrm{Raw}}(x_{s,l};{\mathcal{P}}_{s,l})+{\mathcal{W}}_{s,0}·x_{s,1},$$

(9)

where ${\mathcal{W}}_{s,l}$ and ${\mathcal{P}}_{s,l}$ denote the weight and parameters of the l-th residual unit ${\mathcal{N}}_{s,l}$ in stage s, respectively.

In manually designed networks such as ConvNeXt and Transformers, as well as in searched architectures like EfficientNet, the most common macro network structure typically consists of a preprocessing component, such as an Embedding layer or a Stem layer, followed by multiple stages. Each stage is composed of a downsampler followed by a feature extraction block, and the network ends with one or more detection heads. If each stage is rewritten in the form of Equation (9), while keeping all other components unchanged, the sequential network can be reinterpreted as a supernet for a NAS task.

For the second issue, we chose to truncate the parameters of weights along with the network itself in order to reduce the magnitude of distribution shifts after dropping certain layers. Although the subsequent weights are discarded along with the truncated part of the network, the truncated weights still satisfy normalization and non-negativity, since only the weight parameters involved in subsequent computations are mapped to the probability simplex by the softmax function. Suppose in a certain training or inference scenario, stage s is truncated, retaining $l_s$ residual units. Even if the weight parameters are not updated, the $l_s+1$ weights that are used are recomputed as follows:

$${\mathcal{W}}_{s,l}={\displaystyle \frac{e^{{\mathcal{W}}_{s,l}}}{{\sum }_{j=0}^{l_s}{e}^{{\mathcal{W}}_{s,j}}}},\mathrm{where}0\le l\le l_s.$$

(10)

The complete calculation of the truncated s-th stage is as follows:

$$\begin{array}{cc}\hfill & y_s=\sum _{l=1}^{l_s}{\mathcal{W}}_{s,l}·{\mathcal{N}}_{s,l}^{\mathrm{Raw}}(x_{s,l};{\mathcal{P}}_{s,l})+{\mathcal{W}}_{s,0}·x_{s,1}\hfill \\ \hfill \mathrm{where}& \left\{\begin{array}{c}{x}_{s,1}={\mathrm{Downsampler}}_s\left(x_s\right)\hfill \\ {\mathcal{W}}_{s,l}={\displaystyle \frac{e^{{\mathcal{W}}_{s,l}}}{{\sum }_{j=0}^{l_s}{e}^{{\mathcal{W}}_{s,j}}}},0\le l\le l_s\hfill \end{array}\right.\hfill \end{array}$$

(11)

This process is illustrated in Figure 2, where each residual unit can be selectively retained up to $l_s$ units per stage.

By reinterpreting each stage in this way, while keeping other modules (e.g., Stem, Head) unchanged, any modern multi-stage residual network can be reformulated as a supernet suitable for NAS.

As shown in Algorithm 1, the forward process of a truncated supernet can be represented as combination of several of the truncated stages described above.

Algorithm 1 Supernet forward algorithm

Require: Input: x, Truncation lengthes: L, Network: $\mathcal{N}=\{\mathrm{Stem},\left\{{\mathcal{N}}_s\right\},\mathrm{Head}\}$       Each stage ${\mathcal{N}}_s$ contains:       - ${\mathrm{Downsampler}}_s$       - Weight parameters ${\mathcal{W}}_s^{\mathrm{Raw}}=\left\{{\mathcal{W}}_{s,l}^{\mathrm{Raw}}\right\}l=1,\cdots ,L_s$       - Nonlinear transformations ${\mathcal{N}}_s^{\mathrm{Raw}}=\left\{{\mathcal{N}}_{s,l}^{\mathrm{Raw}}\right\}l=1,\cdots ,L_s$ Ensure: y   1: $x_1\leftarrow \mathrm{Stem}\left(x\right)$;   2: for s = 1 to len$\left(L\right)$ do   3:      $l_s\leftarrow L\left[s\right]$;   4:      ${\mathcal{W}}_s[0:l_s]\leftarrow \mathrm{Softmax}\left({\mathcal{W}}_s^{\mathrm{Raw}}[0:l_s]\right)$;   5:      $y_{s,0}\leftarrow {\mathrm{Downsampler}}_s\left(x_s\right)$;   6:      $x_{s,1}\leftarrow y_{s,0}$;   7:      for l = 1 to $l_s$ do   8:          $y_{s,l}\leftarrow {\mathcal{N}}_{s,l}^{\mathrm{Raw}}\left(x_{s,l}\right)$;   9:          $x_{s,l+1}\leftarrow y_{s,l}+x_{s,l}$; 10:      end for 11:      $x_{s+1}\leftarrow {\sum }_{l=0}^{l_s}{\mathcal{W}}_{s,l}·y_{s,l}$; 12: end for 13: $y\leftarrow \mathrm{Head}\left(x_{S+1}\right)$; 14: return y;

The sub-network construction strategy can follow standard NAS- or pruning-based methods using architecture weights $\mathcal{W}$. A simple instantiation used in our experiments is described in Section 4.1.3.

As a supplementary illustration to Algorithm 1, we present a concrete forward pass of a two-stage supernet under client-specific truncation.

Suppose Stage 1 contains two residual units, and Stage 2 contains ten. For a given client, the truncation policy selects $L=[2,1]$, meaning that both residual units are active in Stage 1, and only one is active in Stage 2.

The input x is first processed by the Stem module to obtain $x_1$.

Stage 1. The input $x_1$ is passed through the downsampler as follows:

$$y_{1,0}={\mathrm{Downsampler}}_1\left(x_1\right),x_{1,1}=y_{1,0}.$$

(12)

The raw weights ${\mathcal{W}}_1^{\mathrm{Raw}}=[{\mathcal{W}}_{1,0}^{\mathrm{Raw}},{\mathcal{W}}_{1,1}^{\mathrm{Raw}},{\mathcal{W}}_{1,2}^{\mathrm{Raw}}]$ are normalized via softmax as follows:

$${\mathcal{W}}_{1,l}={\displaystyle \frac{e^{{\mathcal{W}}_{1,l}^{\mathrm{Raw}}}}{{\sum }_{j=0}^2{e}^{{\mathcal{W}}_{1,j}^{\mathrm{Raw}}}}},l=0,1,2.$$

(13)

Then the residual path is computed as follows:

$$y_{1,1}={\mathcal{N}}_{1,1}^{\mathrm{Raw}}\left(x_{1,1}\right),x_{1,2}=x_{1,1}+y_{1,1},$$

(14)

$$y_{1,2}={\mathcal{N}}_{1,2}^{\mathrm{Raw}}\left(x_{1,2}\right),x_{1,3}=x_{1,2}+y_{1,2}.$$

(15)

The output of Stage 1 is the weighted sum, as follows:

$$x_2={\mathcal{W}}_{1,0}·y_{1,0}+{\mathcal{W}}_{1,1}·y_{1,1}+{\mathcal{W}}_{1,2}·y_{1,2}.$$

(16)

Stage 2. The input $x_2$ is processed similarly. Since $l_2=1$, only one residual unit is used, as follows:

$$y_{2,0}={\mathrm{Downsampler}}_2\left(x_2\right),x_{2,1}=y_{2,0},$$

(17)

$${\mathcal{W}}_{2,l}={\displaystyle \frac{e^{{\mathcal{W}}_{2,l}^{\mathrm{Raw}}}}{{\sum }_{j=0}^1{e}^{{\mathcal{W}}_{2,j}^{\mathrm{Raw}}}}},l=0,1,$$

(18)

$$y_{2,1}={\mathcal{N}}_{2,1}^{\mathrm{Raw}}\left(x_{2,1}\right),x_{2,2}=x_{2,1}+y_{2,1}.$$

(19)

The output of Stage 2 is as follows:

$$x_3={\mathcal{W}}_{2,0}·y_{2,0}+{\mathcal{W}}_{2,1}·y_{2,1}.$$

(20)

Finally, the Head module computes the network output as follows:

$$y=\mathrm{Head}\left(x_3\right).$$

(21)

This example demonstrates how a supernet dynamically executes operations with different truncation lengths per stage, computing outputs through residual updates and stage-wise weighted aggregation, as defined in Algorithm 1.

3.2. Training Pipeline

We recommend a two-phase pretraining process prior to the FL training phase. Then, we introduce a federated sub-network generation method tailored to our specific supernet. The proposed training pipeline is shown in Figure 3.

Our supernet is constructed by reinterpreting residual sequential networks, derived either from manually designed architectures or those obtained via NAS. As a result, various training methods originally developed for manually crafted networks, such as using Adam [36] as the optimizer, can be directly applied to our reinterpreted supernet. In standard training procedures, we optimize block weights $\mathcal{W}$ and feature weights $\mathcal{P}$ jointly, without distinguishing between them.

Although our supernet retains a residual sequential structure, we decouple the training of structural weights and formulate it as a separate GLM-based learning task for structural optimization. Therefore, most training strategies designed for gradient-based NAS, including the bi-level optimization method proposed in DARTS, are also applicable to our approach.

However, standard training methods do not explicitly optimize parameterized network structures. Traditional NAS approaches typically rely on a fixed set of candidate components, such as predefined connections or blocks, which do not align well with the sub-networks produced by our truncation-based method. To address this, we propose a new training strategy tailored for sequential supernets. In each iteration, a truncation length is randomly sampled for each stage to determine the exit point of the sub-network. This mechanism enables random sampling of sub-architectures and simulates the federated learning scenario where different clients operate with different model depths. We refer to this strategy as Single-Path Random Quit (SPRQ), as it randomly selects an exit point in each stage.

From a theoretical perspective, we establish that SPRQ does not alter the optimization target in expectation. We introduce binary mask variables $m_{t,(s,l)}$ to indicate whether parameter ${\mathcal{P}}_{s,l}^{\prime }=\{{\mathcal{P}}_{s,l},{\mathcal{W}}_{s,l}\}$ is active under a randomly truncated path, and we rescale the corresponding gradients as $g_{t,(s,l)}={\displaystyle \frac{m_{t,(s,l)}}{p_{s,l}}}·g_{(s,l)}^{\mathrm{full}}$. This ensures that the gradient estimator remains unbiased, that is,

$$\mathbb{E}\left[g_{t,(s,l)}\right]=g_{(s,l)}^{\mathrm{full}}.$$

(22)

Thus, the stochastic sampling process introduces no bias in gradient estimation.

The per-iteration parameter update of SPRQ is expressed as follows:

$${\mathcal{P}}_{t+1}^{\prime }={\mathcal{P}}_t^{\prime }-\mathrm{diag}\left({\tilde{\eta }}_t\right){\nabla }_{{\mathcal{P}}^{\prime }}\mathcal{L},$$

(23)

where ${\tilde{\eta }}_{t,(s,l)}={\eta }_t·{\displaystyle \frac{m_{t,(s,l)}}{p_{s,l}}}$ denotes the effective learning rate used to update each parameter in response to the loss $\mathcal{L}$. Since $\mathbb{E}\left[{\tilde{\eta }}_t\right]={\eta }_t·1$, we have

$$\mathbb{E}\left[{\mathcal{P}}_{t+1}^{\prime }\right]={\mathcal{P}}_t^{\prime }-\mathbb{E}\left[\mathrm{diag}\left({\tilde{\eta }}_t\right){\nabla }_{{\mathcal{P}}^{\prime }}\mathcal{L}\right]={\mathcal{P}}_t^{\prime }-{\eta }_t·{\nabla }_{{\mathcal{P}}^{\prime }}\mathcal{L},$$

(24)

which, in expectation, matches the behavior of SGD as follows:

$${\mathcal{P}}_{t+1}^{\prime \mathrm{SGD}}={\mathcal{P}}_t^{\prime }-{\eta }_t·{\nabla }_{{\mathcal{P}}^{\prime }}\mathcal{L}.$$

(25)

Under the standard Robbins–Monro conditions ${\sum }_{t=0}^{\infty }{\eta }_t=\infty$ and ${\sum }_{t=0}^{\infty }{\eta }_t^2<\infty$, and with the masks $\left\{m_{t,(s,l)}\right\}$ being independent of the mini-batch noise, the Robbins–Monro theorem [37] implies that ${\mathcal{P}}_t^{\prime }$ almost surely converges to a stationary point of $\mathcal{L}$.

In fact, Equation (23) matches the SGD with per-parameter learning rates. The convergence of such issues has already been extensively addressed [38,39,40,41].

Therefore, SPRQ maintains the convergence characteristics of unbiased stochastic gradient methods while offering practical scalability.

Although this method can train excellent pretraining parameters, suitable for device-heterogeneous federated learning, the initial training phase may face more difficulties in early convergence due to the dual randomness of parameters and structures. Therefore, we suggest replacing SPRQ with specialized NAS optimization methods during the warm-up stage.

In the federated learning stage, based on the characteristics of our network architecture, the sub-networks provided to different clients only differ in truncation lengths, while the model parameters, including structural and weight parameters, are completely consistent. This allows us to directly adopt most common federated learning methods without modifying the parameter aggregation approach, such as ring-based federated learning or FedAvg architectures based on parameter transmission. Even if the model sizes of different clients have quantitative differences, the above methods still apply.

In practical applications, we need to replace the detection Head and Stem parts of the reinterpreted models to adapt to the distribution differences and category number differences between pretraining datasets and federated datasets. After completing this adjustment, we only need to select the maximum network that each client can affordor send the complete supernet to clients and let them decide the length of each stage. In this way, each client can use the strongest model they can support while helping improve the overall model performance. When aggregating client model parameters, based on the early exit characteristics of our network, there is no need to distinguish between different sizes; simply synchronizing the iterated parameters or accumulated gradients in sub-networks to the global model is sufficient.

In centralized federated topologies, communication efficiency can be maintained by aggregating only the subset of parameters relevant to each local model during each round of training, resulting in a communication cost comparable to that of standard FL approaches. In contrast, decentralized federated topologies inherently require more extensive parameter exchange, often involving the entire supernet. Nevertheless, owing to the compact design of our supernet relative to conventional supernet architectures, the associated communication overhead is significantly reduced.

4. Experiments

In this section, we design two sets of experiments to evaluate the impact of different components of the approach. Our method primarily consists of two parts, the Supernet-Only FL framework and a complementary pretraining approach.

4.1. Evaluation of Supernet-Only Framework

We use ConvNeXt Small [34] as the backbone model and design our supernet based on it. ConvNeXt Small consists of four stages; the first, second, and fourth stages have three layers each, while the third stage contains 27 layers. All stages are modified according to the proposed method. However, due to the large size of Stage 3, sub-network pruning is only applied to this stage, while other stages remain unpruned during the FL phase.

To ensure fairness in comparison, a unified pretraining process is conducted for all federated methods in this part of the experiment.

The pretraining lasts for 5 epochs with a learning rate of 1 × ${10}^{-5}$ as warm-up, followed by 20 epochs with a learning rate of 1 × ${10}^{-4}$. The batch size is set to 96, and the Nadam optimizer is used with default settings.

During the pretraining phase, all supernet parameters are fully optimized. In the federated training phase, the first and the last convolutional layer are replaced before federated transfer learning. After training, the performance of all client models is evaluated on the validation set to verify the effectiveness of the method. Although ConvNeXt Small is used as the example model, our method can be generalized to any deep neural networks (DNNs) [42] with skip connections like ResNet [43], ConvNeXt [34], or Transformer [44].

We categorize existing NAS and Fed-NAS strategies under resource-constrained conditions as follows:

• Max Trainable: The largest sub-network that all clients can train is used as the federated model, equivalent to traditional FL.
• Max Runnable: The smallest sub-network that all clients can use for inference is selected as the federated model. Clients unable to train only perform inference, while capable clients participate in training.
• No Federated: Each client uses the strongest sub-network it can support without FL.
• Grouped: Clients are divided into three different scale categories, with each group using the largest sub-network it can support for federated learning.
• Peaches: The cells of sub-network are separated into private cells and public ones. Only public cells are shared among the federal model.

In addition, we conduct comparative experiments under the distribution strategy of AdaptiveFL to validate the effectiveness of the Supernet-Only framework against dynamic network strategies for FL. Max Trainable and Max Runnable represent cases of lightweight global FL, No Federated represents device-aware NAS for personalized search, and Grouped represents clustering methods for Fed-NAS problems. These control groups aim to show the trade-off between using stronger models and sharing parameters across a larger range without our novel method.

Our method has high adaptability, so we test it under the following two standard federated learning structures: ring-based federated learning and the FedAvg framework.

We assume that the computational capabilities of clients are uniformly distributed between the smallest and largest sizes of residual units in the backbone network. The weakest clients can support sub-networks with all variable-sized units set to 1, while the strongest clients can handle the full backbone as it appears before transformation in Section 3.1. For simplicity, we also assume the training cost is twice the inference cost.

4.1.1. Evaluation of the Supernet-Only Framework Under Ring-Based Federated Learning

We first evaluate the Supernet-Only framework using ring-based FL [12] under simulated resource-constrained conditions. The supernet is pretrained on ImageNet1K [8] using the method recommended in Section 3.2. Federated datasets are divided into fourteen parts to simulate private datasets owned by fourteen clients with different performance levels. The variable module counts allowed for each client simulate the computational constraints. In the start of the federated learning phase, the first client of each federated cluster obtain the model from a provider. Then, each client trains the model using private data and passes the updated model to the next client following training until the last client passes the model to the first one. Thus, the cycle goes on, simulating ring-based FL.

Federated training is conducted for 20 epochs with a batch size of 64, using a learning rate of 1 × ${10}^{-4}$ and the Nadam optimizer with default settings. Following the related work, we select the public datasets CIFAR-100, CIFAR-10 [9], and CINIC-10 [10] for evaluation. CIFAR-10 and CIFAR-100 each provide 50,000 training images and 10,000 validation images, while CINIC-10 provides 90,000 images for training and 90,000 for validation. To simulate online learning, the random generator is not turned off during inference, leading to some performance fluctuations in the Max Trainable and Max Runnable groups. After the completion of the federated learning process, the mean and variance of the accuracy for all clients on the validation set are as shown in

Table 1. Accuracy statistic under ring-based FL: Every value illustrates the mean validation accuracy and the corresponding standard deviation across different clients under federated learning settings and dataset correspondents (the best and second-best results are in bold and underlined, respectively).

Dataset	CIFAR-100	CIFAR-10	CINIC-10
Supernet-Only	$\mathbf{78}.\mathbf{57}\pm \mathbf{1}.\mathbf{78}$	$\mathbf{95}.\mathbf{12}\pm \mathbf{0}.\mathbf{84}$	$\mathbf{88}.\mathbf{09}\pm \mathbf{0}.\mathbf{81}$
Max Trainable	$\underline{77.28\pm 0.23}$	$\underline{94.93\pm 0.19}$	$\underline{88.07\pm 0.15}$
Max Runnable	$76.61\pm 0.19$	$93.89\pm 0.20$	$86.82\pm 0.25$
No Federated	$56.01\pm 1.66$	$88.95\pm 0.63$	$79.60\pm 0.71$
Grouped	$67.54\pm 1.67$	$91.49\pm 0.70$	$83.46\pm 0.45$
Peaches	$63.64\pm 16.68$	$85.17\pm 19.67$	$78.70\pm 17.60$

. The mean reflects the federated training performance of the model on heterogeneous clients under various data conditions. The variance of accuracy among all clients indicates the variability in performance within the group.

As shown in Table 1, our method achieved the best performance across all datasets, benefiting from its enhanced federated training capabilities. In the experiments on CIFAR-100, which has only 600 samples per class and presents greater challenges for model generalization, our method demonstrated the most significant improvement. However, in the federated training on CIFAR-10 and CINIC-10, the gaps between our method and the Max Trainable group are narrow, whereas the performance gaps between our method and the Max Runnable are significant. Across all datasets, Peaches, along with both the No Federated and Grouped settings, exhibited inferior overall performance due to its limited ability to share desensitized information. Furthermore, the No Federated group, which had the least sharing, performed the worst, despite each client being able to use the strongest model it could afford.

4.1.2. Evaluation of the Supernet-Only Framework Under Federated Averaging (FedAvg)

We then evaluate the Supernet-Only framework using FedAvg [14] under similar evaluation settings with experiments on the ring-based federated model. FedAvg is the most commonly used FL algorithm to train a global model across distributed clients without centralizing data. The supernet is pretrained and we distribute pruned sub-networks to clients as needed. At the end of each federated training round, the server aggregates the model by averaging the parameter updates from clients. In experiments with smaller models, the parameter update process is relatively simple. However, in the supernet-only framework, client models of different sizes share all parameters. To address this, only the parts used by each client are aggregated. For weights used by c clients in a certain layer, server-side parameter updates are averaged across these c clients. After the completion of the federated learning process, the mean and variance of the accuracy for all clients on the validation set are as shown in

Table 2. Accuracy statistic under FedAvg: Every value illustrates the mean validation accuracies and their corresponding standard deviations across different clients (the best and second-best results are in bold and underlined, respectively).

Dataset	CIFAR-100	CIFAR-10	CINIC-10
Supernet-Only	$\mathbf{67}.\mathbf{67}\pm \mathbf{1}.\mathbf{43}$	$\underline{91.07\pm 0.61}$	$\underline{83.22\pm 0.62}$
Max Trainable	$58.69\pm 0.43$	$88.87\pm 0.37$	$81.12\pm 0.27$
Max Runnable	$\underline{62.31\pm 0.45}$	$\mathbf{91}.\mathbf{43}\pm \mathbf{0}.\mathbf{37}$	$\mathbf{83}.\mathbf{30}\pm \mathbf{0}.\mathbf{30}$
No Federated	$54.35\pm 0.90$	$88.37\pm 1.14$	$79.89\pm 0.60$
Grouped	$59.22\pm 1.89$	$90.09\pm 0.76$	$81.93\pm 0.96$
Peaches	$43.37\pm 1.37$	$72.28\pm 2.58$	$72.37\pm 0.94$

.

As shown in Table 2, our method achieved promising results. In the experiments on CIFAR-100, our methods outperform the secondary best Max Runnable group by more than $5\%$. In the FedAvg experiments on other datasets, the results differed significantly from those observed in ring-based federated learning. Specifically, for CIFAR-10 and CINIC-10, our method showed a narrow disadvantage compared to the Max Runnable group, while Max Trainable struggled due to overfitting and constraints imposed by the limited model size. In these two datasets, where data are relatively abundant, the advantage of shared knowledge of Supernet-Only may not be fully realized.

To validate that our method maintains competitive performance and does not degrade relative to other methods when the federated scale varies, we performed FedAvg experiments on federated clusters consisting of 128 clients. In order to simulate a larger federate of 128 clients, we employed devices with higher GPU memory capacity, which consequently introduced some changes to the computational environment. The other experiments’ settings are the same as those described in Section 4.1.2. After the completion of the federated learning process, the mean and variance of the accuracy for all clients on the validation set are shown in

Table 3. Accuracy statistic under FedAvg for the large-scale federal model: Every value illustrates the mean validation accuracies and their corresponding standard deviations across 128 clients. This experiment was trained for 50 epochs using an H20-96G GPU, whereas other experiments under the ConvNeXt configuration were conducted for 20 epochs on a 4090-24G GPU, due to the increased memory requirements from large-scale client simulation and the slower convergence rate in this setting (the best and second-best results are in bold and underlined, respectively).

Dataset	CIFAR-100	CIFAR-10	CINIC-10
Supernet-Only	$\mathbf{77}.\mathbf{84}\pm \mathbf{0}.\mathbf{91}$	$\mathbf{94}.\mathbf{70}\pm \mathbf{0}.\mathbf{62}$	$\mathbf{87}.\mathbf{12}\pm \mathbf{0}.\mathbf{88}$
Max Trainable	$70.88\pm 0.89$	$92.72\pm 0.59$	$84.13\pm 0.54$
Max Runnable	$\underline{72.42\pm 0.86}$	$\underline{93.41\pm 0.52}$	$\underline{85.75\pm 0.67}$
No Federated	$13.78\pm 8.08$	$69.49\pm 12.00$	$63.44\pm 6.64$
Grouped	$71.45\pm 1.00$	$93.38\pm 0.53$	$83.95\pm 1.01$
Peaches	$57.43\pm 10.18$	$85.79\pm 15.58$	$76.89\pm 3.31$

.

As the scale of the federation increases, the convergence of federated learning approaches that of SGD, and the federated learning performance of the model surpasses those of models of a moderate scale. At the same time, the No Federated method, which does not involve federation, performs poorly.

To further evaluate the reliability and significance of the observed performance differences, we applied the Wilcoxon signed-rank test to compare Supernet-Only with two competitive baselines, Max Runnable and Max Trainable, using client-wise paired results. This non-parametric test is suitable for comparing matched samples without assuming a specific distribution. In the experiment, the test accuracy obtained under Supernet-Only was paired with the corresponding value from each baseline for the same client. Since only 79 out of 128 clients completed training in the Max Runnable setting due to resource limitations, this comparison was limited to the 79 matched clients. In contrast, the comparison involving Max Trainable used results from all 128 clients.

For each dataset and baseline, we calculated the Wilcoxon test statistic W, which is the sum of the ranks of the signed differences in performance. We also reported the standardized score Z, which indicates the magnitude and direction of the difference, the raw p-value, and the Bonferroni-corrected p-value for controlling the family-wise error rate across multiple tests. The raw p-value reflects the probability of observing the given result under the null hypothesis in a single test, while the corrected p-value adjusts for multiple comparisons to reduce the likelihood of Type I errors. In addition, we computed the effect size $r=Z/\sqrt{N}$, where N represents the number of clients included in the comparison. The effect size describes the practical relevance of the result; values of r greater than $0.3$ typically indicate a large effect. These indicators together provide a comprehensive assessment of both the statistical and practical significance of the observed differences, as shown in

Table 4. Wilcoxon signed-rank test results for client-level accuracy comparisons: This table reports statistical test results comparing the performance of our proposed method (Supernet-Only) against two baseline strategies (Max Runnable and Max Trainable) across three datasets. Reported metrics include the Wilcoxon test statistic W, standardized statistic Z, raw p-value, Bonferroni-corrected p-value, and effect size r. Significant differences ($p<0.01$) and medium-to-large effect sizes ($r\ge 0.45$) indicate meaningful performance differences at the client level.

Dataset	Compared to	W	Z	Raw p	Corr. p	r
CIFAR-10	Max Runnable	30	−7.57	1.08 × ${10}^{-13}$	2.17 × ${10}^{-13}$	0.85
	Max Trainable	1986	−5.09	9.43 × ${10}^{-4}$	1.89 × ${10}^{-3}$	0.45
CIFAR-100	Max Runnable	159	−6.94	1.41 × ${10}^{-11}$	2.82 × ${10}^{-11}$	0.78
	Max Trainable	959	−7.54	9.35 × ${10}^{-13}$	1.87 × ${10}^{-12}$	0.67
CINIC-10	Max Runnable	8	−7.68	3.30 × ${10}^{-14}$	6.61 × ${10}^{-14}$	0.86
	Max Trainable	920	−7.63	2.54 × ${10}^{-11}$	5.08 × ${10}^{-11}$	0.67

.

The results indicate that Supernet-Only consistently and significantly outperforms Max Runnable on all of the following three datasets: CIFAR-10, CIFAR-100, and CINIC-10. The Bonferroni-corrected p-values in these comparisons are all below $0.001$, with a value of $2.17\times {10}^{-13}$ on CIFAR-10, which reflects strong statistical evidence. The corresponding effect sizes are large, with r values ranging from $0.78$ to $0.86$. These results show that the observed improvements are not only statistically significant but also of substantial practical importance at the client level.

When compared with Max Trainable, Supernet-Only also demonstrates statistically significant advantages across all datasets. Although the differences in this case are smaller than those observed against Max Runnable, they remain meaningful. All corrected p-values are below $0.01$, such as $1.89\times {10}^{-3}$ on CIFAR-10. The effect sizes range from $0.45$ to $0.67$, which corresponds to medium-to-large practical effects. These results confirm that Supernet-Only offers both wider applicability across clients and consistently higher predictive performance.

Reviewing Table 1, Table 2 and Table 3, across all different sizes of federations and federate frameworks, our method consistently outperforms the suboptimal method by more than $1\%$ on the CIFAR-100 dataset, and this superiority exceeds 5% when using FedAvg, as shown in Table 2 and Table 3. Experimental results on the CIFAR-10 and CINIC-10 datasets demonstrate that our approach consistently ranks among the top performers. Compared to other state-of-the-art methods, our approach either maintains a certain advantage or only shows a minimal disadvantage, regardless of the performance changes in these methods. This indicates that our method is highly competitive and robust.

4.1.3. Evaluation of the Supernet-Only Framework Under FedAvg with Small Backbone

To validate the performance differences between our method and the use of dynamic networks, we implemented the Supernet-Only framework following the latest AdaptiveFL approach. In the reports on AdaptiveFL [11], two smaller networks are utilized, VGG without residual connections and ResNet18 with residual connections. Dynamic neural networks are developed based on these models to meet the requirements of AdaptiveFL. Considering that the VGG network represents an early architecture lacking residual connections, modifications can only be applied to ResNet18 to satisfy the criteria for network mutation as defined in Section 3.1.

To enhance the flexibility of sub-network generation, we decompose the original residual blocks in ResNet18 such that each individual Conv-BN-ReLU layer forms a separate residual unit, resulting in 16 residual units in total. Based on the architecture weights of each stage in the Supernet, we apply softmax normalization and rank the four candidate operations within each stage. We then generate a series of sub-networks by progressively reducing the number of retained operations per stage from four to one, following the ranked priorities. We adopt the federated training setting from AdaptiveFL while performing our experiments based on the client distribution described in Section 4.1. In our setting, the average client size is significantly smaller, and the size difference among clients is substantially greater compared to the report of AdaptiveFL. After training through the federated phase, the accuracy achieved by different methods across various datasets is summarized in

Table 5. Accuracy statistic under the AdaptiveFL and Supernet-Only frameworks with or without pretraining. Every value illustrates the mean validation accuracies and their corresponding standard deviations across 13 model sizes and 100 clients under federated setting of AdaptiveFL (the best and second-best results are in bold and underlined, respectively).

Dataset	CIFAR-100	CIFAR-10	CINIC-10
Supernet-Only (pretrained)	$\mathbf{57}.\mathbf{76}\pm \mathbf{7}.\mathbf{11}$	$\mathbf{87}.\mathbf{78}\pm \mathbf{5}.\mathbf{76}$	$\mathbf{79}.\mathbf{22}\pm \mathbf{6}.\mathbf{14}$
Supernet-Only (not pretrained)	$\underline{49.56\pm 12.31}$	$\underline{77.55\pm 18.16}$	$\underline{73.89\pm 7.28}$
AdaptiveFL	$31.26\pm 19.39$	$67.77\pm 25.36$	$57.92\pm 21.68$

.

Table 6. Comparison of performance gaps between the largest and best structures. Comparisons of the full-size and best-performing models in AdaptiveFL with max and best sub-networks in the Supernet-Only framework without pretraining. Full and max refer to the largest network configurations (both matching ResNet-18), while best indicates the sub-network or pruned network achieving the highest test accuracy.

Dataset	AdaptiveFL		Supernet Only
	Best	Full	Best	Max
CIFAR-100	$53.75$	$14.03$	$56.91$	$53.34$
CIFAR-10	$87.36$	$76.64$	$88.81$	$88.64$
CINIC-10	$74.72$	$59.15$	$78.05$	$75.97$

provides a detailed comparison between the largest-sized network and the strongest network from AdaptiveFL and the Supernet-Only framework without pretraining.

In AdaptiveFL, networks of different sizes are separately created, which leads to convergence issues if they are pretrained. In contrast, our sub-networks are generated by a supernet, enabling our method to benefit from pretraining. Even without the unique advantage of pretraining, our method still demonstrates superior performance, benefiting from module selection and optimization guided by structure weights $\mathcal{W}$.

In AdaptiveFL, as the difference in model size is significant, the performance degradation of the full-sized network compared to the best-performing pruned network exceeds $10\%$. In comparison, our method effectively minimizes the discrepancy between the optimal method and the largest network size, maintaining a stable performance. Once the pretraining is activated, the sub-network with the largest size always outperforms the smaller ones within the Supernet-Only framework.

Overall, we observe that as the generalization difficulty of the training setting increases, the advantage of our method becomes more pronounced. Additionally, clustering federated devices into multiple federated subgroups does not appear to yield superior results compared to lightweight model selection. However, it is important to note that our backbone model, ConvNeXt, is inherently strong. In our experiments, model representational capacity may not be significantly constrained, which worked to the advantage of both the Max Runnable and Max Trainable groups.

4.2. Evaluation of the Training Pipeline

In Section 3.2, we propose a recommended pretraining routine for FL. In this section, we validate the effectiveness of our proposed pretraining method. We evaluate three different training methods during the warm-up and pretraining phases, including the standard training method used in conventional neural network training, the two-step optimization method proposed in DARTS, and our proposed SPRQ method that simulates federated client group behavior.

• Normal: In this set, we pretrain the supernet using conventional network training methods and common optimizers, treating architecture parameters as regular network parameters.
• DARTS: In this set, we use the two-step optimization method from DARTS [7], alternately training weights and architecture parameters on training and validation datasets.
• SPRQ: In this set, we generate sub-networks by randomly skipping layers during pretraining to simulate client diversity and computational constraints (detailed in Section 3.2).

Next, we evaluate their respective performance in the pretraining and warm-up phases to verify the necessity of using the DARTS warm-up followed by the SPRQ pretraining process.

4.2.1. Evaluation of the Main Pretraining Phase

In Section 3.2, we recommend the SPRQ method for the main pretraining phase. Fixing the warm-up training method as DARTS, we used SPRQ, DARTS, and standard optimization methods during the pretraining phase. Pretraining and warm-up were conducted on the ImageNet1k dataset, with the supernet’s loss and accuracy on the validation set during pretraining shown in Figure 4a,b. Subsequently, FL was performed on the CIFAR-100, CIFAR-10, and CINIC-10 datasets, and the accuracy of different clients was evaluated using the respective validation sets. The mean and standard deviation of client accuracies are shown as error bars in Figure 4c,d.

As shown in Figure 4a,b, during the pretraining phase, SPRQ did not outperform DARTS or Normal in terms of loss and accuracy. The Normal group showed the best convergence ability and final precision. However, despite the additional randomness introduced, SPRQ demonstrated convergence comparable to DARTS and Normal and achieved the highest average accuracy across all clients with the smallest standard deviation in FL, as shown in Figure 4c,d. Conversely, the Normal training method performed poorly during FL. The Normal group had the lowest average client performance and the largest performance standard deviation.

Furthermore, as shown in

Table 7. Pretraining time (20 epochs, hours). Each value represents the time cost statistics during the 20 epochs of the pretraining phase on an RTX 4090 GPU.

	DARTS	Standard	SPRQ
Time Spent (h)	25.40	25.66	19.34

, SPRQ consumed significantly less time during pretraining due to frequent early exits.

4.2.2. Evaluation of Warm-Up Phase

In Section 3.2, we suggest using DARTS for warm-up training to overcome the challenges of initial SPRQ training. To validate its effectiveness, we pretrained the network on the ImageNet1K dataset with different warm-up methods and then performed pretraining using the SPRQ method on the same dataset. Loss and accuracy on the ImageNet1K validation set during the pretraining phase are shown in Figure 5a,b. Subsequently, FL was conducted with the same setup as in the evaluation of the main pretraining phase, and the federated average accuracy and standard deviation under different datasets and methods are shown in Figure 5c,d.

As shown in Figure 5a,b, the DARTS warm-up method outperformed Normal and SPRQ in terms of convergence speed and final performance. The error bars in Figure 5c,d demonstrate that DARTS provided the best average performance across all clients, with the lowest standard deviation in validation accuracy. This validates our recommendation to use a DARTS-based training method during the warm-up phase.

5. Discussion

Existing Fed-NAS research faces the challenge of balancing comprehensive data anonymization and sharing with the full utilization of client performance.

We observe that the output of sequential networks composed of residual units with skip connections can be represented as a weighted sum of the outputs from each nonlinear transformation. This allows us to reinterpret any network composed of residual units into a NAS task by making these learnable weights independent as a metal-earning task. Based on this insight, we introduce a new methodology for converting any residual-based sequential neural network into a NAS task, enabling flexible and efficient architecture search. Even though standard NAS and non-NAS methods can be employed to train the transformed supernet, its particular structure led us to discover a more fitting training strategy, referred to as Single-Path Random Quit (SPRQ). Our method facilitates the extensive sharing of anonymized data while maximizing the performance of each client, considering their varying computational capabilities. This effectively eliminates the previously frustrating trade-off, as shown in

Table 8. Comparison of Fed-NAS methods. This table compares representative Fed-NAS methods in terms of their ability to maximize federation, computation utilization, and model sharing. Maximizing Federation refers to all clients forming a single federation. Maximizing Computation means each client can select the best-performing sub-network according to its own computational capacity. Full Sharing indicates that all parameters used by clients are included in the federated synchronization. A ✔ indicates that the method supports the corresponding feature.

Method	Maximize Federation	Maximize Computation	Full Sharing
Supernet-Only	✔	✔	✔
Max Trainable	✔		✔
Max Runnable			✔
No Federated		✔
Grouped			✔
Peaches	✔	✔

.

Extensive experiments on CIFAR-10, CIFAR-100, and CINIC-10 demonstrate the effectiveness of the proposed Supernet-Only framework for federated learning. Under both ring-based and FedAvg training schemes, the framework generally achieves superior or at least comparable performance to strong baselines across diverse and resource-constrained client settings. On CIFAR-100, the Supernet-Only framework improves average accuracy by more than 5% over the Max Trainable baseline in the FedAvg setup. Statistical analysis using the Wilcoxon signed-rank test confirms that these gains are significant at the client level, with consistently low p-values and large effect sizes. In large-scale federated learning with 128 clients, the framework maintains strong performance, reaching an average accuracy of 77.84% on CIFAR-100 and outperforming the next-best method by over 5%. On CIFAR-10 and CINIC-10, while performance is sometimes comparable to Max Runnable, Supernet-Only remains among the top-performing methods in all cases. When applied to smaller backbones such as ResNet-18 under the AdaptiveFL configuration, it yields accuracy improvements of up to 18.3%. Additionally, the proposed training pipeline, which combines DARTS-based warm-up with SPRQ-based pretraining, achieves the best overall federated performance with reduced variance and cuts pretraining time by approximately 24% compared to other methods. These findings indicate that the Supernet-Only framework offers a practical, scalable, and reliable solution for federated learning in computational heterogeneous environments.

Despite the promising results, our study has several limitations. First, the experiments are conducted on commonly-used public benchmarks such as CIFAR-10, CIFAR-100, and CINIC-10. While these datasets facilitate comparability with prior work, they do not fully capture the complexity, heterogeneity, and privacy challenges of real-world federated learning applications, such as those in healthcare, mobile devices, or industrial systems. Second, our current framework assumes a stable communication environment with synchronous client participation. It does not consider real-world factors such as limited bandwidth, intermittent connectivity, or client dropouts, which can significantly impact performance in practical federated learning scenarios. Third, our reinterpretation of residual networks as supernets under the neural architecture search paradigm is, to the best of our knowledge, the first of its kind. However, this exploration remains preliminary and limited in scope. Further investigation is needed to fully understand the expressiveness, generalization behavior, and potential drawbacks of this formulation. Finally, all evaluations are conducted in simulated environments. The absence of deployment on actual mobile or edge devices means that important system-level aspects such as latency, memory usage, and energy consumption have not yet been evaluated.

Future work will focus on addressing these limitations to improve the real-world applicability of our approach. Specifically, we plan to evaluate our method using more realistic federated datasets that better reflect cross-device and cross-silo environments, such as MedMNIST [45], OpenFL benchmarks [46], and other LEAF benchmarks [47], which capture non-IID characteristics, device heterogeneity, and practical deployment constraints. We also intend to extend the framework to handle communication-limited and unreliable scenarios, including asynchronous client updates, adaptive participation, and the mitigation of straggler clients. Additionally, we aim to further explore the reinterpretation of residual networks as supernets by exploring more expressive weighting mechanisms, analyzing their convergence behavior in heterogeneous conditions, and reintegrating them with advanced neural architecture search strategies. Finally, we will investigate the deployment of our method on actual hardware platforms such as smartphones, edge devices, or IoT systems in order to assess system-level performance, resource consumption, and feasibility in real-world applications.

6. Conclusions

In this paper, we propose the Supernet-Only framework, which reinterprets residual sequential networks as supernets in a NAS task by expressing their outputs as weighted sums of nonlinear transformations, where the weights are optimized through a meta-learning process. By introducing stage-wise truncation and softmax-normalized architectural weights, our framework enables dynamic sub-network generation based on client-specific constraints, allowing modern multi-stage residual architectures to be efficiently adapted to federated learning environments. We design a training pipeline consisting of a DARTS-based warm-up phase, SPRQ-based pretraining, and federated fine-tuning with sub-network truncation, enabling efficient supernet training across heterogeneous clients while preserving compatibility with standard federated learning protocols.

The experimental results confirm that the proposed Supernet-Only framework achieves consistently strong performance across diverse federated learning scenarios and datasets, outperforming existing baselines in both large-scale and small-backbone settings while benefiting from an effective DARTS-SPRQ pretraining pipeline.

However, limitations remain, such as reliance on standard benchmark datasets, assumptions of stable communication, and the lack of deployment on real-world devices. Future work will focus on addressing these limitations by introducing realistic federated datasets, supporting asynchronous and unreliable environments, enhancing the generalization of the supernet formulation, and evaluating the system-level performance on real hardware.