Supervised Feature Selection Method Using Stackable Attention Networks

Abstract

Mainstream DNN-based feature selection methods share a similar design strategy: employing one specially designed feature selection module to learn the importance of features along with the model-training process. While these works achieve great success in feature selection, their shallow structures, which evaluate feature importance from one perspective, are easily disturbed by noisy samples, especially in datasets with high-dimensional features and complex structures. To alleviate this limitation, this paper innovatively introduces a Stackable Attention architecture for Feature Selection (SAFS), which can calculate stable and accurate feature weights through a set of Stackable Attention Blocks (SABlocks) rather than from a single module. To avoid information loss from stacking, a feature jump concatenation structure is designed. Furthermore, an inertia-based weight update method is proposed to generate a more robust feature weight distribution. Experiments on twelve real-world datasets, including multiple domains, demonstrate that SAFS produced the best results with significant performance edges compared to thirteen baselines.

1. Introduction

With the rapid advancement of the Internet of Things (IoT) and industrial automation systems, both the number and dimensionality of data samples are increasing at an unprecedented rate [1]. Feature selection (FS) aims to identify a subset of relevant features that contribute most to the supervisory objective, thereby improving model interpretability and robustness [2]. Although numerous methods have been proposed to address this classical problem [3,4], many traditional approaches rely on the computation of global metrics or functions, making them increasingly challenging to apply in the era of high-dimensional and large-volume data [5].

In recent years, Deep Neural Networks (DNNs) have emerged as a prominent research direction in feature selection, owing to their ability to capture complex feature interactions and alleviate the “curse of dimensionality and volume.” Significant attention has been directed toward designing effective feature weighting mechanisms. For instance, ref. [6] introduced Deep Feature Selection (DFS), which employs a sparse one-to-one linear layer whose weights are directly used as feature weights—though this design remains susceptible to noise. To mitigate this issue, ref. [7] utilized activation potentials from individual input dimensions as a selection criterion. Ref. [8] pioneered the integration of attention mechanisms into feature selection (referred to as AFS), where feature weights are generated via a soft attention network. Subsequently, ref. [9] enhanced the attention network by incorporating multi-head self-attention for more stable performance. In a different vein, ref. [10] adopted stochastic gates—continuous relaxations of the Bernoulli distribution optimized via gradient descent—to identify relevant features. These methods generally rely on specific modules, such as neural weights [6,11], activation potentials [7], attention weights [8], or stochastic gates [10], to determine salient features. However, as our experimental results indicate, such voting-based architectures often struggle to capture the intricate feature–label relationships necessary for effective feature selection.

Recent FS research has primarily focused on two directions: (1) developing robust feature weighting mechanisms—e.g., ref. [12] applied batch-normalized attention to improve weight stability; (2) leveraging self-supervised learning to model data distribution—e.g., refs. [13,14] employed multi-task autoencoders to denoise input data and facilitate feature selection, while ref. [15] adopted similarity-based graph neural networks to extract structural information from unlabeled data. Notably, however, these approaches still fundamentally depend on conventional feature-wise weight generation layers. We argue that such a constrained architectural design significantly limits feature selection potential, particularly when dealing with high-dimensional data and diverse label spaces.

• Motivation. The design of SAFS is grounded in the principles of ensemble learning and multi-perspective learning. Ensemble methods demonstrate that combining multiple weak learners yields a stronger, more accurate model. In feature selection, however, a single feature selection module (a “weak selector”) can be misled by noisy samples or complex feature interactions. This raises a key question: instead of relying on a single module, can multiple selectors work collaboratively to produce more robust and discriminative feature weights?

To address this, SAFS employs a stacking architecture where multiple selection modules are cascaded, forming a strong selector. Each successive layer refines the feature weights from its predecessor, effectively performing multi-round, collaborative voting on feature importance. The approach is theoretically supported by the bias-variance trade-off; stacking reduces variance in feature weight estimation, leading to more stable and reliable feature subsets. Furthermore, inspired by deep residual networks [16], which show that stacking non-linear transformations with skip connections helps model complex relationships while easing optimization, we stack SABlocks with feature jump connections. Additional mechanisms are incorporated to mitigate information loss during stacking and to counteract the impact of noisy data, thereby supporting more stable weight updates.

The main contributions of this paper are summarized as follows:

• Beyond one-layer structures for feature selection: Rather than using one layer structure for feature weight generation, SAFS allows for the feature weights are generated collaboratively from a set of stacked SABlocks. This structure can enlarge the differentiation between essential and less important features and further reduce the weight of features that are accidentally relevant.
• Key Designs of SAFS: This research proposes a set of crucial designs to support the stackable feature weight generation: (D1) SABlock is a new type of feature weight generation module which can be easily stacked for collaborative feature weight generation. This design can capture complex and intricate feature-label interactions while generating robust feature weights. (D2) “Feature Jump Concatenation” is designed to allow stacked SABlock to access the original feature values rather than processed data to avoid error accumulation. (D3) “Inertia-based weight updation” is proposed. Similarly to the update strategy in reinforcement learning, feature weights are updated by leveraging both current and historical weights to achieve stable weight generations.
• Excellent experimental results: Our extensive experiments have been performed on twelve real-world datasets to validate our design. The results show that SAFS identifies the most relevant features that provide the best performance compared to thirteen state-of-the-art baselines.

2. Related Work

Feature selection, as a fundamental problem in machine learning, aims to identify the most relevant feature subset for a given learning task. Existing methods can be broadly categorized into three paradigms: filter methods, wrapper methods, and embedded methods [17]. Among these, embedded methods have gained increasing attention due to their ability to integrate feature selection within model training, particularly with the advancement of deep learning techniques. In this section, we systematically review the evolution of feature selection methods and highlight how our proposed SAFS framework addresses key limitations in existing approaches.

2.1. Traditional Feature Selection Methods

Traditional feature selection methods typically rely on handcrafted criteria to evaluate feature relevance. These include

• Similarity-based methods, such as Fisher Score [18], which select features that maximize inter-class separation and minimize intra-class variance.
• Information-theoretic approaches, like mRmR [19], which consider both feature relevance and redundancy using mutual information.
• Regularization-based techniques, exemplified by LASSO [20], that induce sparsity through L1-norm regularization.

While effective in low-dimensional settings, these methods face significant challenges with high-dimensional and large-volume datasets due to their reliance on global metric computations [5]. Recent traditional methods have attempted to address these limitations by incorporating data structural information, such as neighborhood rough sets [21], multi-center and local structure learning [22], and feature dependency analysis [23]. However, their ability to capture complex, non-linear feature interactions remains limited, which motivates the need for more expressive deep learning-based approaches.

2.2. DNN-Based Feature Selection Methods

The emergence of deep neural networks has revolutionized feature selection by enabling the learning of complex feature interactions directly from data. DNN-based methods typically incorporate a custom-designed feature selection module within the network architecture:

• Weight-based methods: DFS [6] introduced a sparse one-to-one linear layer where network weights directly represent feature importance. This approach, however, is susceptible to noise interference.
• Activation-based methods: Roy et al. [7] used activation potentials contributed by individual input dimensions as selection metrics.
• Attention mechanisms: AFS [8] pioneered the use of soft attention networks for feature weighting, while SANs [9] extended this with multi-head self-attention for improved stability.
• Stochastic approaches: STG [10] employed continuous relaxations of Bernoulli distributions (stochastic gates) trained via gradient descent.
• Architectural innovations: FIR [4] proposed a dual-network architecture with separate selector and operator networks, while FM [12] incorporated batch-wise attention to attenuate noise.

More recent developments include external attention-based feature rankers [24], sequential attention networks [25], and neurodynamics-based approaches [26]. FsNet [27] introduced a specialized selection layer for high-dimensional biological data.

Despite these advancements, a fundamental limitation persists across most DNN-based methods: their reliance on a single feature selection module. This monolithic architecture restricts their capacity to capture the complex, multi-faceted nature of feature relevance in high-dimensional spaces. When faced with noisy samples or accidental feature correlations, these single-perspective approaches are prone to generating suboptimal feature weights. It is precisely this limitation that motivates our stackable architecture in SAFS, which enables collaborative feature evaluation from multiple perspectives.

2.3. Self-Supervised and Semi-Supervised Methods

Recent research has recognized the importance of leveraging unlabeled data to improve feature selection performance. These methods employ various techniques to extract structural information from tabular data:

• Autoencoder-based approaches: SEFS [13] and A-SFS [14] utilize multi-task autoencoders to learn latent relationships in unlabeled data, reducing noise in the original features.
• Graph-based methods: Tan et al. [15] employed similarity-based graph neural networks to capture structural information in unlabeled data.
• Other techniques: These include clustering assumptions [28], fuzzy relevance and redundancy analysis [29,30], heuristic functions [31], and adaptive structure learning [32,33].

While these methods demonstrate the value of leveraging unlabeled data and have shown promise in capturing data structures, they largely inherit the fundamental architectural limitations of their supervised counterparts. Most self-supervised approaches still rely on single-module feature weighting mechanisms and focus primarily on data preprocessing or auxiliary tasks, without addressing the core challenge of robust feature weight generation in the presence of noise and complex feature interactions.

This gap in the literature highlights the need for a more fundamental rethinking of feature selection architectures. Our SAFS framework addresses this by proposing a novel stackable architecture that can work in conjunction with both supervised and self-supervised paradigms. The feature jump concatenation and inertia-based weight updating mechanisms in SAFS provide a more robust foundation for feature weighting that can potentially enhance existing self-supervised methods when combined with our architecture.

In summary, while existing methods have made significant progress in feature selection, they largely overlook the potential of multi-perspective, collaborative feature evaluation. SAFS represents a paradigm shift from single-module architectures to a stackable framework that can generate more robust and discriminative feature weights through layered processing and historical weight integration.

3. Our Proposed Model

3.1. Notation and Problem Formulation

Generally, this paper presents the matrix as an uppercase bold character (e.g., $\mathbf{A}$), vector as lowercase bold (e.g., $\mathbf{a}$) and value as lowercase (e.g., a). Some important notations are described in

Table 1. Notation description.

Notation	Description
n	The number of samples
m	The number of features
g	Inertia parameter
s	Stack parameter
${\mathbf{x}}_i$	The i-th sample
$y_i$	The i-th label
$c_i$	The bias vector
$x_{ij}$	The j-th feature of the i-th sample
K	The number of selected features
$P(·)$	Distribution of dataset
$\mathbf{X}$	Original data matrix
$\mathbf{y}$	The label set corresponding to $\mathbf{X}$
$\mathbf{h}$	Hidden units in a neural network
$\mathbf{w}$	The weight vector of features
${\mathbf{X}}^{\mathbf{v}}$	The selected features
${\mathbf{X}}_i$	The i-th feature
${\mathbf{X}}_B$	The batch-wise inputs
$\mathbf{H}$	Low dimension embeddings
$\mathsf{\Theta }$	Trainable weight matrix
$\mathcal{L}(·)$	Loss function

. The tabular dataset is defined as $\mathbf{X}\in {\mathbb{R}}^{n\times m}$ with n samples and m dimensions (features), the i-th sample is denoted as ${\mathbf{x}}_i$, the i-th feature is ${\mathbf{X}}_i$ and the j-th feature of the i-th sample is denoted as $x_{ij}$. All samples in $\mathbf{X}$ are labeled, corresponding to $\mathbf{y}=\left\{y_1,y_2,\dots ,y_n\right\}$.

Given the realizations of unknown data distribution $P_{(X,y)}$, the goal of feature selection is to select a subset with k indices $v\subset \left[m\right]$, $\left|v\right|=k$, and ${\mathbf{X}}^{\mathbf{v}}$ that can best predict the target $\mathbf{y}$. Thus, the objective function of embedded feature selection can be achieved by solving the following optimization problem:

$$\underset{v\subset \left[m\right]}{min}{\mathbb{E}}_{P_{(X,y)}}\left[\mathcal{L}(f\left({\mathbf{X}}^{\mathbf{v}}\right),\mathbf{y})\right]$$

(1)

where $\mathcal{L}(·)$ is the loss function (e.g., cross-entropy for classification), $f(·)$ is a predictive model that maps the selected features ${\mathbf{X}}^{\mathbf{v}}$ to the target $\mathbf{y}$. The expectation ${\mathbb{E}}_{P_{(X,y)}}$ is taken over the underlying data distribution, emphasizing that the goal is to find a feature subset that generalizes well. Solving this optimization directly is intractable due to the combinatorial nature of the subset selection problem $v\subset \left[m\right]$. Embedded methods like SAFS circumvent this by jointly learning the feature weights (which implicitly define v) and the model parameters through continuous optimization, making the process computationally feasible.

3.2. Design Principle

In order to accurately calculate feature weights from the collaboration of multiple FS modules, it is important to design an appropriate architecture to allow for collaborative weight generation. One natural solution involves sequentially stacking several feature selection modules, using their multiplied weights as the final feature results. Another approach is to ensemble several modules in parallel and use their averaged weights. However, both designs have their respective trade-offs. For the sequential structure, each feature selection module can work as a filter with weights for each feature. Due to its multiplicative properties, this structure can filter out irrelevant features hierarchically and expand the difference in feature weights. Because feature selection usually only focuses on a small number of the most relevant features, often only 1 to 3% of the total features are selected, the difference in feature weights can obviously lead to better feature selection. However, this structure is vulnerable to small amounts of errors. If any of the modules accidentally generate wrong feature weights, those errors would propagate to its downstream modules. However, in the parallel structure, different modules can directly access the inputs and individually would not be influenced by others, thus its feature selection capabilities could be impaired.

3.3. SAFS Architecture

Thus, rather than using one of the two, SAFS adopts a hybrid structure with both. Figure 1 illustrates the global structure of SAFS. The basic feature selection block is SABlock, which evaluates the importance of the feature. From a feature weight generation perspective, SABlocks are placed sequentially. However, except for the SABlock outputs from the previous layer, each block can directly access the original inputs to avoid possible information loss. For collaborative weight generation, an ‘inertia-based collaborative weight updating’ mechanism is also designed, adopted from deep reinforcement learning.

• The Stackable Attention Block. This research introduces the basic building block of SAFS, named SABlock, which can assign individual weights to each feature. Its structure is shown in the left part of Figure 2. The SABlock consists of two fully connected (FC) layers and one batch normalization layer. For each iteration, SABlocks receives batch-wise inputs ${\mathbf{X}}_{\mathbf{B}}={[{\mathbf{x}}_{\mathbf{1}},{\mathbf{x}}_{\mathbf{2}},\dots ,{\mathbf{x}}_{\mathbf{B}}]}^T$, sampled from the training dataset $\mathbf{X}$, B is the batch size. The FC layer 1 is defined as Equation (2).

$$\mathbf{H}={\mathsf{\Theta }}_1·{\mathbf{X}}_{\mathbf{B}}+{\mathbf{c}}_1$$

(2)

where ${\mathsf{\Theta }}_{\mathbf{1}}$ is a trainable weight matrix, ${\mathbf{c}}_{\mathbf{1}}$ is bias vector. This layer condenses original data with $\mathit{m}$ dimensions into embeddings $\mathbf{H}$ with a smaller space $\left|\mathbf{h}\right|<\mathit{m}$ ($\left|\mathbf{h}\right|$ means the number of hidden units) to reduce possible noise and duplication. The activation function is removed to avoid overly complex non-linear transformations. To mitigate the bias caused by the internal covariate shift and prevent overfitting, a batch normalization layer is incorporated to re-center and re-scale the input $\mathbf{H}$ with

$$\widehat{\mathbf{H}}=\mathbf{\gamma }·\mathit{N}\left(\mathbf{H}\right)+\mathit{ϵ}$$

(3)

where $\mathit{N}(·)$ is the normalization function, bias $\mathit{ϵ}$ and gain $\mathit{\gamma }$ are learnable parameters. Then, the FC layer 2 maps $\widehat{\mathbf{H}}$ back to a batch-wise weight matrix of the same size as the input. The $\mathit{tanh}(·)$ activation function is used to capture non-linear relationships among features. Then, the feature weight $\mathbf{w}$ is calculated by column-wise averaging of the weight matrix to mitigate the impact of noise introduced by certain noisy samples and/or accident correlations. This process is depicted by Equation (4), as follows:

$$\mathbf{w}={\displaystyle \frac{\mathbf{1}}{\mathit{B}}}{\displaystyle \sum _{\mathit{k}=\mathbf{1}}^{\mathit{B}}}\mathit{tanh}\left({\mathsf{\Theta }}_{\mathbf{2}}·\widehat{\mathbf{H}}+{\mathbf{c}}_{\mathbf{2}}\right)$$

(4)

where ${\mathsf{\Theta }}_{\mathbf{2}}$ is a trainable weight matrix, ${\mathbf{c}}_{\mathbf{2}}$ is bias vector.

• Design for Collaborative Weight Generation. This section elaborates on the key designs that enable multiple SABlocks to work in concert, thereby generating more robust and discriminative feature weights. To precisely delineate the data flow and weight generation across different layers, we employ superscripts to denote the layer of origin. For example, ${\mathbf{X}}_{\mathbf{B}}^{\left(\mathbf{0}\right)}$ signifies the original input batch, whereas ${\mathbf{w}}^{\left(\mathbf{s}\right)}$ represents the feature weight vector generated by the $\mathit{s}$-th SABlock.
• Feature Jump Concatenation: Mitigating Information Attenuation. A critical challenge in deep sequential feature weighting is the potential loss of information through layers, where features attenuated early might be irrecoverable later even if they are relevant. To address this, we design the feature jump concatenation mechanism, allowing each SABlock to receive inputs from two parallel paths, as illustrated in the right panel of Figure 2.

1.

Path A: Refined Input from Predecessor: This path carries the output of the preceding SABlock, given by the element-wise product ${\mathbf{w}}^{\left(\mathbf{s}\right)}\odot {\mathbf{X}}_{\mathbf{B}}^{\left(\mathbf{s}\right)}$. This represents a refined view of the features, where the influence of each feature has been selectively amplified or dampened based on the consensus of the previous $\mathit{s}$ layers.

2.

Path B: Persistent Original Input: This path provides direct, unaltered access to the original input batch ${\mathbf{X}}_{\mathbf{B}}^{\left(\mathbf{0}\right)}$. It serves as an information-rich anchor or reference, ensuring that no feature value is entirely lost due to potentially premature weighting in earlier blocks.

The input to the next SABlock is formed by concatenating these two paths:

$${\mathbf{X}}_{\mathbf{B}}^{(\mathbf{s}+\mathbf{1})}=\mathrm{Concat}\left({\mathbf{w}}^{\left(\mathbf{s}\right)}\odot {\mathbf{X}}_{\mathbf{B}}^{\left(\mathbf{s}\right)},{\mathbf{X}}_{\mathbf{B}}^{\left(\mathbf{0}\right)}\right)$$

(5)

This design ensures that every SABlock can perform its importance assessment with full context—it can either reinforce the weighting decisions of its predecessors by further amplifying features that are consistently important (via Path A), or it can re-evaluate and potentially rescue features that were previously under-weighted by accessing their original values (via Path B). This effectively combats progressive information loss and error accumulation in the stack.

• Inertia-based Collaborative Weight Updating: Ensuring Stability. To foster coherent collaboration across the stack and prevent erratic weight fluctuations, we introduce an inertia-based weight update mechanism. This strategy incorporates a form of memory into the weight evolution process, ensuring that the feature weights change smoothly across layers.

The update rule, inspired by momentum optimization in gradient descent but applied directly to the feature weights, is defined as follows:

$${\mathbf{w}}^{(\mathbf{s}+\mathbf{1})}\leftarrow (\mathbf{1}-\mathit{g})·{\mathbf{w}}^{(\mathbf{s}+\mathbf{1})}+\mathit{g}·{\mathbf{w}}^{\left(\mathbf{s}\right)}$$

(6)

Here, ${\mathbf{w}}^{(\mathbf{s}+\mathbf{1})}$ on the right-hand side is the new weight vector freshly computed by the $(\mathit{s}+\mathbf{1})$-th SABlock based on its input. ${\mathbf{w}}^{\left(\mathbf{s}\right)}$ is the weight vector from the previous layer. The inertia coefficient $\mathit{g}$ (where $\mathbf{0}\le \mathit{g}\le \mathbf{1}$) is a crucial hyperparameter that controls the blend.

1.

A high value of $\mathit{g}$ (e.g., close to 1) implies strong inertia, meaning the system heavily relies on the historical trajectory of weights, leading to very stable but potentially slower-to-adapt updates.

2.

A low value of $\mathit{g}$ (e.g., close to 0) gives more weight to the current layer’s instant assessment, making the system more responsive but also more susceptible to noise.

This mechanism provides two key benefits: (1) Stability: It smooths the weight updates, making the training process less sensitive to noisy batches or outliers that might cause large swings in a single layer’s estimation. (2) Consensus-driven weighting: It forces a feature to be consistently deemed important across successive layers to achieve a high final weight, as a single layer’s strong opinion can be tempered by the historical consensus. The process is initialized with ${\mathbf{w}}^{\left(\mathbf{0}\right)}=\mathbf{0}$.

Finally, the output weight vector from the last SABlock (layer $\mathit{S}$) is normalized via the softmax function to produce the final feature importance scores $\mathit{\alpha }=\mathrm{softmax}\left({\mathbf{w}}^{\left(\mathbf{S}\right)}\right)$, which have a sum of 1 and are used for feature selection.

• Learning module. Learning module is used to model the model between the twisted inputs, generated from the last SABlock and outputs. It can be any form of DNN. Here, a multilayer perceptron with two FC layers is generally adopted. All SABlocks and weights are adjusted during the backpropagation until convergence. The loss function $\mathcal{L}$ is as follows:

$$\mathcal{L}=\mathit{CrossEntropy}(\widehat{\mathbf{y}},\mathbf{y})$$

(7)

For multi-category classification, the softmax cross-entropy is used; for binary-category classification, the binary cross-entropy is used.

3.4. Theoretical Analysis of Stacking for Stability

The sequential stacking of SABlocks can be viewed as an ensemble method applied to feature weighting. We can provide a simplified theoretical intuition for why stacking enhances stability.

Consider the final feature weight vector $\mathit{\alpha }$ as a function of the input batch ${\mathbf{X}}_{\mathbf{B}}$. A single-layer feature selector (e.g., a lone SABlock or attention layer) computes $\mathit{\alpha }=\mathit{f}\left({\mathbf{X}}_{\mathbf{B}}\right)$. The variance of this estimator can be high if $\mathit{f}$ is susceptible to noise in ${\mathbf{X}}_{\mathbf{B}}$.

In SAFS, with $\mathit{S}$ stacked blocks and inertia $\mathit{g}$, the weight update can be unfolded as follows:

$${\mathbf{w}}^{\left(\mathbf{S}\right)}\approx {(\mathbf{1}-\mathit{g})}^{\mathit{S}}{\mathit{f}}_{\mathit{S}}\left({\mathbf{X}}_{\mathbf{B}}\right)+\mathit{g}{(\mathbf{1}-\mathit{g})}^{\mathit{S}-\mathbf{1}}{\mathit{f}}_{\mathit{S}-\mathbf{1}}\left({\mathbf{X}}_{\mathbf{B}}\right)+\cdots +{\mathit{g}}^{\mathit{S}}{\mathit{f}}_{\mathbf{0}}\left({\mathbf{X}}_{\mathbf{B}}\right)$$

(8)

where ${\mathit{f}}_{\mathit{s}}$ represents the transformation of the $\mathit{s}$-th SABlock. This is a form of exponential smoothing across layers. This smoothing has two effects:

1.

Variance Reduction: The final weight ${\mathbf{w}}^{\left(\mathbf{S}\right)}$ is a weighted average of the opinions of all $\mathit{S}$ layers. Assuming the noise in each layer’s estimation is somewhat independent, the variance of the average is reduced compared to the variance of a single layer. This makes the feature ranking less volatile across different training batches.

2.

Error Robustness: An erroneous weight assignment in one specific layer (${\mathit{f}}_{\mathit{s}}$) is dampened by the contributions of other layers. The inertia parameter $\mathit{g}$ controls the strength of this smoothing; a higher $\mathit{g}$ places more trust in the historical consensus, further increasing stability at the cost of potentially slower adaptation.

This analysis aligns with the empirical observations in our sensitivity analysis (Section 4.6), where increasing the inertia $\mathit{g}$ generally led to more stable performance. Furthermore, the ablation study (Section 4.5) confirms that removing the stacking structure (SAFS-s) or the inertia mechanism (SAFS-i) consistently leads to a performance drop or increased standard deviation, validating the importance of this design for robust feature selection.

3.5. Pseudocode of Proposed SAFS

The pseudocode of SAFS is shown in Algorithm 1. In each iteration, a batch of B samples is randomly selected from the dataset, which is then processed by the SABlock to obtain a weight vector $\mathit{w}$. After passing the last SABlock, a softmax function is used to transform the weights into feature important scores $\mathit{\alpha }$; finally, $\mathit{\alpha }$ is evaluated by a multi-layer perceptron (MLP).

Algorithm 1 Stackable Attention Network for Feature Selection

Input: Dataset $\mathbf{X}$, Batchsize B, Bias $\mathit{ϵ}$, Gain $\mathit{\gamma }$, inertia parameter $\mathit{g}$, Number of stacked layers S, Iteration K

Output: feature important scores $\mathit{\alpha }$

$\mathit{t},\mathit{s},{\mathbf{w}}^{\left(\mathbf{0}\right)}\leftarrow \mathbf{0}$

while $\mathit{k}<\mathit{K}$ do

${\mathbf{X}}_{\mathbf{B}}^{\left(\mathbf{0}\right)}\leftarrow \mathit{Sampling}\left(\mathbf{X}\right)$ # Sampling a batch as input

while $\mathit{s}<\mathit{S}$ do

$\mathbf{H}\leftarrow \mathit{FC}\mathit{Layer}\mathbf{1}\left({\mathbf{X}}_{\mathbf{B}}^{\left(\mathbf{0}\right)}\right)$

$\widehat{\mathbf{H}}\leftarrow \mathit{\gamma }·\mathit{N}\left(\mathbf{H}\right)+\mathit{ϵ}$

${\mathbf{w}}^{\left(\mathbf{s}\right)}\leftarrow {\displaystyle \frac{\mathbf{1}}{\mathit{B}}}{\sum }_{\mathit{k}=\mathbf{1}}^{\mathit{B}}\mathit{FC}\mathit{Layer}\mathbf{2}\left(\widehat{\mathbf{H}}\right)$

${\mathbf{X}}_{\mathbf{B}}^{(\mathbf{s}+\mathbf{1})}=\mathit{Concat}(({\mathbf{w}}^{\left(\mathbf{s}\right)}\odot {\mathbf{X}}_{\mathbf{B}}^{\left(\mathbf{s}\right)}),{\mathbf{X}}_{\mathbf{B}}^{\left(\mathbf{0}\right)})$ # Feature jump concatenation

if $\mathit{s}>\mathbf{2}$ then

${\mathbf{w}}^{\left(\mathbf{s}\right)}\leftarrow (\mathbf{1}-\mathit{g})·{\mathbf{w}}^{\left(\mathbf{s}\right)}+\mathit{g}·{\mathbf{w}}^{\left(\mathbf{0}\right)}$ # Weight updating

end if

$\mathit{s}\leftarrow \mathit{s}+\mathbf{1}$

end while

$\mathit{\alpha }\leftarrow \mathit{softmax}\left({\mathbf{w}}^{\left(\mathbf{s}\right)}\right)$

$\widehat{\mathit{y}}\leftarrow \mathit{Learning}\_\mathit{Module}({\mathit{X}}_{\mathit{B}}^{\left(\mathbf{0}\right)}\odot \mathit{\alpha })$ # MLP

$\mathcal{L}=\mathit{CrossEntropy}(\widehat{\mathit{y}},\mathit{y})$

$\mathit{k}\leftarrow \mathit{k}+\mathbf{1}$

end while

The trainable parameters of SAFS, including the weight matrices ${\mathsf{\Theta }}_{\mathbf{1}},{\mathsf{\Theta }}_{\mathbf{2}}$ and bias vectors ${\mathbf{c}}_{\mathbf{1}},{\mathbf{c}}_{\mathbf{2}}$ in each SABlock, are initialized using the Xavier uniform initializer. This initialization method is chosen to maintain stable gradient flow through the network at the start of training. The initial feature weight vector ${\mathbf{w}}^{\left(\mathbf{0}\right)}$ is explicitly set to $\mathbf{0}$, as indicated in Algorithm 1. This zero-initialization ensures that the first SABlock initially passes the input features forward uniformly, allowing the learning process to gradually and stably differentiate feature importance from an unbiased starting point. All parameters, including the feature weights which are generated dynamically per batch, are updated via backpropagation to minimize the loss function $\mathcal{L}$ (Equation (7)).

3.6. Parallel Stacking Model (SAFS-Pa)

In theory, such a stack structure can have infinite combinations. This research proposes a parallel stacking model for analysis, named SAFS-Pa. It is noteworthy to point out that the parallel model still belongs to the one-layer feature selection architecture albeit it can enhance the stability of the feature selection process. As shown in the left part of Figure 3, several SABlocks are used, and each SABlock generates an attention weight vector $\mathbf{w}$. Then, weight vectors from multiple SABlocks are averaged as overall weights:

$$\mathbf{w}=\mathit{Mean}({\mathbf{w}}_{\mathbf{1}},{\mathbf{w}}_{\mathbf{2}},\dots ,{\mathbf{w}}_{\mathbf{n}})$$

(9)

Finally, a following softmax function is used to regulate the weights into (0, 1) range with most of them close to 0:

$$\mathit{\alpha }={\displaystyle \frac{{\mathit{e}}^{\mathbf{w}}}{{\sum }_{\mathit{j}=\mathbf{1}}^{\mathit{m}}{\mathit{e}}^{{\mathbf{w}}_{\mathit{j}}}}}$$

(10)

4. Experiment

This section compares SAFS with several state-of-the-art baselines on twelve real-world datasets and six synthetic datasets. The source code is public available at https://github.com/Icannotnamemyselff/SAFS (accessed on 4 November 2025).

4.1. Experiment Settings

• Real-world datasets. Twelve real-world tabular datasets from public repositories (OpenML and Datamicroarray) are selected for evaluation to ensure diversity and real-world applicability. Our selection criteria are designed to cover a wide spectrum of challenges in feature selection:

• Dimensionality: We include low-dimensional (e.g., SEGMENT, 19 features), medium-dimensional (e.g., HAR, 561 features), and high-dimensional datasets (e.g., Chiaretti, 12,625 features) to test scalability.
• Sample Size. The number of samples ranges from a few (e.g., Alon, 62 samples) to many (e.g., SVHN/CIFAR-10, 10,000 samples), assessing performance in both data-scarce and data-rich regimes.
• Number of Classes: We include binary classification (e.g., Gravier), multi-class classification with a moderate number of classes (e.g., DNA, 3 classes), and many-class problems (e.g., ISOLET, 26 classes).
• Domain Diversity: Datasets are drawn from various domains, including medicine, image processing (represented as tabular data), speech recognition, physics, and biology, to ensure generalizability.

This comprehensive selection mitigates the risk of overfitting to a specific data characteristic and provides a robust evaluation of the proposed method. Detailed characteristics of these datasets are summarized in

Table 2. Datasets description.

Datasets	Features	TopK	Classes	Samples	Domains
Chiar.	12,625	%3(390)	4	127	Medical
SVHN(Tabular)	3072	%3(92)	10	10,000	Picture
CIFAR10(Tabular)	3072	%3(92)	10	10,000	Picture
Gravier	2905	%3(90)	2	168	Medical
Alon	2000	%3(60)	2	62	Medical
MNIST(Tabular)	784	%3(24)	10	8000	Picture
ISOLET	618	%3(18)	26	2600	Speech
HAR	561	%3(16)	6	3000	Physics
DNA	180	%3(5)	3	450	Biology
GAS	128	%3(5)	6	6000	Chemistry
SATIMAGE(SAT.)	37	5	6	600	Physics
SEGMENT(SEG.)	19	5	7	1400	Picture

.

• Case Study Dataset. To further validate SAFS in a real-world industrial scenario, we conducted a case study on short-term wind power forecasting using a publicly available dataset from Kaggle. This time-series dataset originates from a real wind farm in Germany, containing sensor readings recorded every 15 min. It presents a challenging regression task with high feature dimensionality (76 sensors over time) and strong temporal dependencies, moving beyond curated academic benchmarks to a practical application.

• Synthetic datasets. First, following the widely used data models $\mathbf{E}\mathbf{1}\sim \mathbf{E}\mathbf{6}$ which were also used for evaluation in [34,35,36,37,38]. The input features are independently generated from a 20-dimensional Gaussian distribution with no correlations across the features ($\mathbf{X}\sim \mathcal{N}(\mathbf{0},\mathbf{I})$), where $\mathit{I}$ is the $\mathbf{20}\times \mathbf{20}$ identity matrix. The target is $\mathit{y}={\mathbf{1}}_{\mathit{A}}\left({\displaystyle \frac{\mathbf{1}}{\mathbf{1}+\mathit{Logit}\left(\mathbf{X}\right)}}\right)>\mathbf{0}.\mathbf{5}$, where ${\mathbf{1}}_{\mathit{A}}(·)$ is an indicator function. The $\mathit{Logit}\left(\mathbf{X}\right)$ for each sample is calculated based on different features, depending on the sign of the 11-th feature ${\mathbf{X}}_{\mathbf{11}}$, and is defined as follows:

$\mathbf{E}\mathbf{1}:\mathit{Logit}=\mathit{exp}({\mathbf{X}}_{\mathbf{1}}\times {\mathbf{X}}_{\mathbf{2}})$;

$\mathbf{E}\mathbf{2}:\mathit{Logit}=\mathit{exp}\left({\mathsf{\sum }}_{\mathit{i}=\mathbf{3}}^{\mathbf{6}}({\left({\mathbf{X}}_{\mathit{i}}\right)}^{\mathbf{2}}-\mathbf{4})\right)$;

$\mathbf{E}\mathbf{3}:\mathit{Logit}=\mathit{exp}(-\mathbf{10}\times \left(\mathit{sin}\left(\mathbf{2}{\mathbf{X}}_{\mathbf{7}}\right)\right))+\mathbf{2}\left|{\mathbf{X}}_{\mathbf{8}}\right|+{\mathbf{X}}_{\mathbf{9}}+\mathit{exp}(-{\mathbf{X}}_{\mathbf{10}})$;

$\mathbf{E}\mathbf{4}:\mathit{Logit}\mathit{follows}\mathbf{E}\mathbf{1}\mathit{if}{\mathbf{X}}_{\mathbf{11}}<\mathbf{0}\mathit{else}\mathbf{E}\mathbf{2}$;

$\mathbf{E}\mathbf{5}:\mathit{Logit}\mathit{follows}\mathbf{E}\mathbf{1}\mathit{if}{\mathbf{X}}_{\mathbf{11}}<\mathbf{0}\mathit{else}\mathbf{E}\mathbf{3}$;

$\mathbf{E}\mathbf{6}:\mathit{Logit}\mathit{follows}\mathbf{E}\mathbf{2}\mathit{if}{\mathbf{X}}_{\mathbf{11}}<\mathbf{0}\mathit{else}\mathbf{E}\mathbf{3}$;

• Baselines. The experiments compare SAFS with 13 classical and/or novel feature selection methods from three main streams.

• ML-Based:

LASSO [20]: a classical regularized linear/logistic model; RF [39]: a classical tree-based model; XGB [40]: Gradient boosted decision trees; CCM [41]: a kernel-based model which measures of independence to find a feature subset.

• DNN-Based:

AFS [8] and SANs [9]: attention mechanism-based model; FIR [4]: a dual-net architecture, the selector generates a feature subset, while the operator makes predictions; FM [12]: batch-wise attenuation model with almost no hyperparameters; STG [10]: continuous-relaxation-based model; NeuroFS [11]: By gradually pruning the uninformative features from the input layer of a sparse neural network.

• Self-supervised:

A-SFS [14] and SEFS [13]: Using multi-task autoencoder to learn latent relationship in unlabeled data.

Evaluation Protocols. In practice, the ground-truth about actual feature relevance is generally unavailable. Hence, following common practice in feature selection, the performance of each baseline is evaluated based on the prediction accuracy achieved using its TopK selected features. This is an indirect way to assess the relevance of the discovered features on the target tasks.

Considering the need for both effectiveness and overfitting prevention in deep neural networks, two powerful and popular classifiers are employed: LightGBM [3] and CatBoost [42]. For the classification task, the Micro-F1 score (in %) is used for evaluation. All datasets are split into training and test sets at an 8:2 ratio.

In the evaluation, the number of selected TopK features is fixed at 3% of the total feature dimension, with a minimum value of K set to 5. All experiments are repeated 10 times using different random seeds, and the averaged Micro-F1 score on the test sets is reported.

Parameter Details. All baselines selected for comparison use the default/recommended settings specified in their paper. For SAFS, the number of stacked layers is set to 3, and the number of hidden units is set to 64. This configuration was determined based on a balance between model capacity and computational cost through preliminary experiments on a held-out validation set. Deeper stacks (e.g., 5, 10) showed diminishing returns or potential overfitting on smaller datasets, while shallower stacks (1, 2) underperformed on complex datasets. The hidden unit size of 64 provided a good compromise between representation power and efficiency for the datasets in our study.

The gain $\mathit{\gamma }$ and bias $\mathit{ϵ}$ of the BN-layer are set to 0.9 and 10⁻⁵, respectively. The value for $\mathit{\gamma }$ is slightly lower than the default of 1.0 to initially allow for milder re-scaling of the normalized activations, which we found to contribute to training stability. The value for $\mathit{ϵ}$ is a common default used to ensure numerical stability during normalization.

The inertia parameter $\mathit{g}$ is set to 0.8, favoring a strong reliance on historical weights for stable updates. The batch size constitutes 10% of the total samples, a common practice that provides a reasonable estimate of the gradient. The Adam optimization method [43] is utilized, with a learning rate of 0.002.

4.2. Main Results: Real-World Data

Table 3. Average performance (Micro-F1↑) with LightGBM classifier with ten runs, ‘OT’ means overtime (more than 24 h of calculation in EPYC 7552*2 192 cores), ‘−’ means no result because of an internal error occurred. The best- and second-best results are highlighted in bold and with underline, respectively.

Algorithm	SEG.	SAT.	GAS	DNA	HAR	ISOLET	MNIST	Alon	Gravier	SVHN	CIFAR10	Chiar.
LASSO	86.95	74.77	90.21	68.89	80.57	66.82	54.31	72.45	73.72	22.55	27.23	82.56
RFE	93.09	72.88	94.85	31.25	83.24	55.86	40.75	66.84	68.43	OT	OT	OT
RF	96.50	83.44	89.37	87.48	93.78	76.73	61.34	81.05	82.15	51.96	36.50	82.82
XGB	90.52	83.88	95.37	82.74	93.35	68.38	61.78	84.73	87.25	57.24	41.43	85.64
CCM	95.12	82.56	93.63	64.59	82.77	59.12	42.71	80.05	75.68	45.52	41.34	−
FIR	91.73	73.88	93.61	43.33	76.72	63.28	41.18	80.00	68.23	47.40	41.24	73.84
AFS	95.98	82.09	96.21	72.67	91.56	75.20	62.80	80.00	80.88	55.89	38.16	86.92
SANs	94.04	79.11	94.37	61.03	88.77	68.79	36.86	84.31	71.37	45.22	39.84	71.28
FM	95.24	80.45	97.01	83.91	89.94	75.88	63.67	78.02	77.57	57.92	42.87	78.85
STG	96.33	81.77	95.57	79.55	94.06	77.94	62.44	83.15	80.78	56.30	40.61	83.68
NeuroFS	94.34	82.54	86.11	70.37	94.95	60.59	55.61	78.59	80.26	45.90	42.42	87.91
A-SFS	96.17	80.04	94.35	80.59	92.44	69.25	64.16	75.78	72.55	42.14	38.94	74.36
SEFS	95.59	83.42	96.94	76.13	−	68.07	61.77	78.64	75.61	53.76	39.99	−
SAFS-Pa	96.42	84.91	97.14	87.64	93.86	74.00	64.71	85.38	85.58	54.40	41.66	88.23
SAFS	96.75	84.33	97.99	89.33	96.17	80.84	62.64	86.92	87.55	60.61	43.86	88.46

and

Table 4. Average performance (Micro-F1↑) with Catboost classifier with ten runs, ‘OT’ means overtime (more than 24 h of calculation in EPYC 7552*2 192 cores), ‘−’ means no result because of an internal error occurred. The best- and second-best results are highlighted in bold and with underline, respectively.

Algorithm	SEG.	SAT.	GAS	DNA	HAR	ISOLET	MNIST	Alon	Gravier	SVHN	CIFAR10	Chiar.
LASSO	87.88	75.50	90.68	69.85	80.51	68.20	56.85	78.42	73.92	25.68	29.29	78.68
RFE	93.83	77.89	95.87	46.67	85.58	63.72	43.68	OT	OT	OT	OT	OT
RF	97.42	84.38	89.50	87.11	93.81	77.29	63.22	83.15	82.54	58.01	39.27	84.47
XGB	90.69	83.72	95.40	82.44	92.67	69.23	64.25	85.79	89.21	63.38	43.88	89.73
CCM	96.12	82.67	94.05	68.51	89.83	67.01	56.10	84.21	76.07	52.04	43.97	−
FIR	94.04	82.22	92.24	40.74	76.83	65.33	38.23	83.15	74.51	55.82	43.76	69.74
AFS	95.02	78.67	96.38	77.89	91.80	75.96	65.10	83.07	80.51	58.39	42.56	85.38
SANs	93.66	79.55	94.30	59.40	89.11	70.71	30.32	85.36	76.47	54.26	41.27	72.30
FM	95.33	80.76	97.12	83.47	90.45	76.33	64.02	81.14	77.73	59.26	44.56	76.30
STG	97.09	83.77	96.44	86.22	91.20	75.35	65.11	86.88	80.78	59.75	43.22	81.57
NeuroFS	95.28	83.03	88.49	82.05	95.22	58.47	57.64	82.87	84.26	46.62	43.58	86.67
A-SFS	95.57	83.55	94.35	80.51	87.73	76.64	65.99	76.38	74.07	56.29	42.23	76.92
SEFS	96.08	82.79	96.67	83.33	−	77.02	65.42	80.29	77.89	54.78	43.01	−
SAFS-Pa	96.96	84.50	97.12	86.22	94.43	75.44	66.63	86.15	84.61	60.74	45.29	85.76
SAFS	97.17	82.62	98.18	89.11	95.95	81.82	65.33	87.38	88.11	66.86	46.89	89.69

present the experimental results with 13 baselines across two robust classifiers. Notably, SAFS achieves good results on all datasets and shows significant improvements over the compared baselines, including several state-of-the-art methods. When utilizing with the CatBoost classifier, most baseline results demonstrate enhancement, particularly for RF and XGB, but SAFS maintains leading compared to baselines, particularly on large-scale datasets such as ISOLET, SVHN, and CIFAR-10.

For the main results on Table 3, compared to the average performance of other baselines, SAFS can improve ranging from 3.08% to 26.01%. The improvements on datasets with high dimensions, e.g., HAR, ISOLET, SVHN and Chiar. are especially significant. This fact highlights the effectiveness of SAFS on high-dimensional data.

Focus on other baselines, LASSO, due to its linear nature, achieves the worst performance in almost all datasets. RFE, as a wrapper algorithm, demands a vast amount of calculations, often leading to overtime error (over 24 h of computation on EPYC 7552 with 48 × 2 core) on high-dimensional datasets. XGB and RF, known for their strong performance and robustness in feature selection, generally rank second or third, outperform most DNN-based solutions. However, when the dimension increases, their performance generally lags behind recent baselines in those high-dimensional data. The DNN-based methods, AFS, SANs, and FIR, are susceptible to inter-sample noise, worse than tree-based methods. Two recent baselines, STG and NeuroFS, also achieve unstable performance in these datasets because they are susceptible to accidental correlation. Self-supervised solutions, A-SFS and SEFS, display no significant performance improvement since they typically depend on additional unlabeled data for self-supervision and offer limited enhancements to the feature weight generation module designed for the high-dimensional nature of the dataset.

These results suggest that the stacked architecture can significantly improve the ability to capture complex feature interactions compared to existing one-layer-based approaches. SAFS also has a relatively low standard deviation, please refer to detailed experiment results which clearly shows the robustness of the SAFS weight generation towards different datasets.

Subsequently, the feature selection performance is compared across varying numbers of TopK features. Figure 4a–d illustrates the classification accuracy on four different datasets. Notably, SAFS demonstrates significant performance advantages over other baselines across nearly all evaluated TopK ranges, particularly with respect to the initial features. This is because as the stacking deepens, features that are more relevant to the label will be given greater weights, while similar features are given more discriminative weights. This merit is rather important in real-world FS applications, as it shows that SAFS generates high-quality feature weights for different features: both for highly related features and those less related but still valuable features.

While SAFS achieves the best or second-best performance on most datasets, we observe on the SEGMENT dataset (Table 3) that STG achieves a highly competitive accuracy of 96.33%, closely followed by SAFS at 96.75%. This minor difference does not undermine the value of SAFS but rather highlights a key characteristic: for some well-structured datasets with less complex feature interactions, a single strong baseline like STG can perform exceptionally well. The advantage of SAFS’s stacked architecture becomes more pronounced in more challenging scenarios, as evidenced by its substantial performance gains on high-dimensional (e.g., SVHN and ISOLET) and noisy datasets (as shown in Section 4.5). The stackable design aims for robust superiority across a wider range of data conditions rather than necessarily outperforming every baseline on every single dataset by a large margin.

• Why Stacking works? This part aims to answer an important question: Why does stacking help the FS process? As previously analyzed, later SABlocks can provide a more precise view of the pertinent features as accidental associated features can be gradually filtered out by this layered structure.

To experimentally verify this, the MNIST (Tabular) dataset is utilized to visualize the weights generated by SABlocks with different depths. The weights collected from different layers undergo a min–max normalization operation for better presentation. The kernel density estimation (KDE) of feature weights is illustrated in Figure 5. Please note that the Gaussian kernel used in the visualization also applies smoothing at the boundaries, which can lead to some density estimates to overflow outside the range [0, 1].

In fact, the FS task usually focuses on a few features that are most relevant to the target; therefore, enhancing the identifiability of the TopK features is logically beneficial. Compared to Layer-1 and Layer-2, the weights generated by Layer-3 exhibit superior discriminative capabilities. When the weight exceeds 0.8, Layer-3 shows a lower feature density. At the same time, the peak of weight density in Layer-3 shifts to the left, which indicates that more features are considered unimportant.

4.3. Main Results: Synthetic Data

This section uses synthetic datasets where the target value only depends on a subset of features, which varies across samples. All datasets in this study consist of 2000 generated samples, split into training and testing sets at an 8:2 ratio. This experimental regime presents a more challenging scenario than that explored in [34]. The experiments evaluate all baselines by measuring the $\mathit{TPR}$ of informative features and accuracy score of the LightGBM classifier ($\mathit{TPR}={\displaystyle \frac{\mathit{TP}}{\mathit{TP}+\mathit{FN}}}$, where TP is the number of informative features that are selected by FS method, and FN is the number of informative features unrecovered by the model).

The results are shown in

Table 5. Relevant feature discovery results for synthetic datasets with 20 features.The best and second-best results are highlighted in bold and underline, respectively.

Dataset	E1	E2	E3	E4	E5	E6
Metrics (%)	TPR/F1	TPR/F1	TPR/F1	TPR/F1	TPR/F1	TPR/F1
XGB	100/66.7	100/66.7	96.7/64.4	85.7/63.2	85.7/63.2	88.9/64.0
RF	100/66.7	100/66.7	100/66.7	71.4/58.8	74.3/59.6	76.5/60.4
AFS	100/66.7	100/66.7	96.7/64.4	46.7/48.0	57.2/53.3	46.7/48.0
STG	100/66.7	100/66.7	100/66.7	71.4/58.8	61.9/55.2	85.2/62.9
NeuroFS	100/66.7	100/66.7	96.7/64.4	45.6/47.6	65.6/57.2	52.5/51.6
SAFS	100/66.7	100/66.7	100/66.7	98.6/65.6	86.7/63.5	85.2/62.9

, where SAFS performs the best. For the relatively easy $\mathbf{E}\mathbf{1}\sim \mathbf{E}\mathbf{3}$ datasets, all methods perform well; on the complex $\mathbf{E}\mathbf{4}\sim \mathbf{E}\mathbf{6}$ datasets, DNN-based methods are weaker than tree-based methods, while SAFS still performs well on these datasets, which shows that SAFS has still good feature discrimination capabilities for capturing complex non-linear feature-label relations.

4.4. Feature Selection Under Different Regimes

This section evaluates the performance of SAFS under different regimes, including scenarios where (1) the number of features exceeds the number of samples; (2) the complexity of feature selection tasks evolves; and (3) feature selection is conducted under various types of noise.

• Feature selection in the $\mathit{m}>\mathit{n}$ regime. The performance of SAFS is next evaluated under a more challenging regime where the number of features exceeds the number of samples ($\mathit{m}>\mathit{n}$), using only 10% of the randomly sampled data. The results are reported across varying numbers of TopK features to examine whether the proposed method can consistently identify the optimal feature subset under different TopK levels.

Figure 6a–c show the performance of different baselines with different TopK. Across almost all TopK ranges, SAFS achieves superior performance with few samples per class and keeps a steady increment of performance with increasing K. This experiment shows that SAFS can effectively weight high-dimensional features with limited labels.

• Feature selection under noise disturbance. Building robust models in noisy environments (e.g., industrial data, genetic engineering) is essential. To analyze model robustness under noise conditions, this study designs two noisy scenarios for evaluating feature selection methods.

• Feature Perturbation: Three types of noise are injected into all datasets to assess baseline robustness on contaminated data:

  (a) 

  Gaussian noise with mean 0 and variance 0.3;

  (b) 

  Salt-and-pepper (S&P) noise with a noise ratio of 0.3;

  (c) 

  Mask noise where 30% of features are randomly set to zero.

The results in Figure 7a show that SAFS provides clear advantages over all baselines. The linear embedding method LASSO performs poorly in high-dimensional noisy settings. DNN-based methods AFS and STG exhibit sensitivity to noise, while XGB, as a strong tree-ensemble baseline, demonstrates robustness comparable to SAFS. The hybrid architecture of SAFS functions similarly to a multi-round feature voting mechanism, effectively mitigating noise impact.

Further experiments evaluate noise resistance with 30% feature masking on three real-world datasets of different scales. As shown in Figure 6d–f, stacking provides significant performance gains on the large-scale SVHN dataset. For medium- and small-scale datasets (ISOLET and DNA), the stacking depth should be limited to 5–10 layers to prevent overfitting.

• Label Perturbation: Random label noise is introduced by replacing original labels with random ones at varying ratios. To examine the impact on critical features, the Top-1% features selected by each method are used for comparison.

Figure 7b demonstrates that SAFS consistently identifies robust feature subsets under label perturbation, while all methods degrade as noise increases, SAFS maintains stable performance across noise levels, confirming its hybrid architecture’s effectiveness.

An interesting observation is that most FS methods remain relatively stable under 5% label perturbation. As indicated in recent work [44], this may be because limited out-of-distribution (OOD) data prevents over-reliance on category-specific features.

• Feature selection with class variation. Here, two special tasks are prepared to evaluate SAFS’s generalization ability, the ISOLET dataset is used, which has 26 categories. Classes are selected according to Label orders.

• Changing feature selection complexity: In this setting, the number of classes gradually increases based on the index order from 0 to 25. As the number of classes grows, the dataset and the feature selection task become progressively more complex and challenging.

Results in Figure 8a indicate that as the number of categories increases, the task becomes more challenging, leading to a gradual decline in accuracy on the feature subsets. Compared to other baselines, SAFS achieves the best overall performance and demonstrates greater effectiveness with increasing category counts.

• Feature Selection for Transfer Learning: This setting evaluates feature selection using only a subset of classes (e.g., only classes 0–3), while testing classification performance across all 26 categories. This setup assesses whether the selected features generalize to unseen classes, with fewer available classes making the task more challenging.

Figure 8b shows that with very few classes (e.g., 3 classes), most feature selection methods tend to learn features specific to the minority classes rather than generalizable representations. As the number of classes increases, the performance advantage of stronger methods becomes more pronounced. The proposed method achieves the best performance when the number of classes exceeds 3. It should be noted that while more classes provide more information, they also increase the difficulty of feature selection. Some methods (e.g., XGB and CCM) exhibit performance degradation as the number of classes grows. In contrast, SAFS maintains consistently improving performance across all evaluated ranges, clearly validating its effectiveness in this transfer learning scenario.

4.5. Ablation and Component Analysis

To better understand the impact of individual components in SAFS, ablation studies were conducted by sequentially removing each key component to observe corresponding performance changes.

• Overall results. 

  Table 6. Ablation studies. SAFS-s: Remove the stack architecture, do FS only use one SABlock; SAFS-bn: Remove the BN-layer of SABlock; SAFS-i: Remove the inertia-based weight update strategy; SAFS-c: Remove the feature skip connection. The best and second-best results are highlighted in bold and underline, respectively.

  Datasets	SAFS	SAFS-s	SAFS-bn	SAFS-i	SAFS-c
  Chiar.	88.46 ± 6.70	86.92 ± 7.33	70.38 ± 14.49	87.27 ± 7.52	86.00 ± 7.13
  SVHN	60.61 ± 0.74	57.03 ± 3.18	57.82 ± 3.23	59.40 ± 1.06	59.47 ± 1.43
  ISOLET	80.84 ± 2.16	73.01 ± 3.68	65.73 ± 0.66	78.20 ± 2.68	79.09 ± 2.29
  GAS	97.99 ± 0.39	96.83 ± 0.72	93.95 ± 2.07	97.08 ± 0.81	97.65 ± 0.59

  clearly shows that all designs are important for improving performance. Our key design, the stack architecture, plays a crucial role, as removing it results in a significant performance drop on high-dimensional datasets. Additionally, the BN-layer in each SABlock is essential for reducing the standard deviation. From the results, the feature skip connection design and the inertia-based update strategy also increase the stability of the SAFS.

• SABlock v.s. MLP. The multi-layer perceptron (MLP), as a basic feature selection DNN baseline, can be easily stacked into multiple layers. If all the designed components are removed, SABLock will degenerate into MLP. In this comparison, the SABlock architecture is evaluated against a standard MLP baseline without additional design modifications. All parameters for SABlock and MLP are set identically: The hidden layer units are set to 64, the learning rate is 0.002, the batch size is 10% of the total sample size, and the number of training epochs is 10,000.

The results in Figure 9 show that the SABlock architecture achieves more stable performance on both datasets, particularly with 2∼10 layers. Although MLP exhibits some performance improvement with increasing layers, it shows unstable performance on the two datasets (low accuracy with high standard deviation), while SABlock shows stable changes.

4.6. Hyperparameter Analysis

This section presents a sensitivity analysis of key hyper-parameters: stacking depth, batch size, and inertia rate.

• Varying stacking layers. Here, performance variations are evaluated across different numbers of stacking blocks in the proposed architecture.

  Table 7. Performance comparison with different variants. The best and second-best results are highlighted in bold and underline, respectively.

  # of Layer	HAR	ISOLET	SVHN	Chiar.
  1	94.85 ± 0.81	73.01 ± 3.68	54.68 ± 1.61	86.92 ± 7.33
  2	95.92 ± 0.99	81.94 ± 1.71	55.62 ± 1.78	88.23 ± 8.03
  3(Ours)	96.17 ± 0.83	80.84 ± 2.16	60.61 ± 0.74	88.46 ± 6.70
  5	96.08 ± 0.83	80.34 ± 1.13	59.64 ± 0.63	90.00 ± 7.73
  10	95.71 ± 0.95	80.63 ± 2.05	59.54 ± 0.79	88.84 ± 7.38
  20	95.63 ± 0.64	77.36 ± 3.03	59.69 ± 0.86	87.30 ± 8.25

  presents results for six architectural variants using 1, 2, 3, 5, 10, and 20 blocks, respectively. It can be observed that SAFS generally achieves better and more stable performance by stacking more SABlocks, particularly in the range 2∼10. When there are too many stacks (e.g., layer = 20), the generated weights may focus only on the most important features and ignore the less important features, resulting in certain performance degradation.

• Batch size and inertia parameter. Figure 10a demonstrates that when batch size B is too small, it negatively impacts accuracy because the model more likely to be influenced by some noisy samples. An appropriate batch size contributes to the accuracy of gradient estimation and the stability of the optimization process. A comparably large batch size should be used to update feature weights. This is especially evident for the SVHN and Gravier datasets. Generally, a batch size greater than 10% of the training set is preferable.

Figure 10b shows the impacts of different inertia. The results of these datasets indicate a similar upward trend in accuracy as $\mathit{g}$ increases. When the inertia $\mathit{g}$ = 0, SABlocks would not collaborate and the performance is worst. The increase of $\mathit{g}$ would enhance the overall stability of the training process, but being too high might impair the learning of an SABlock. $\mathit{g}$ = 0.8 generatlly achive good performance.

4.7. Discussion of Computational Complexity

During the feature selection process, the main computational complexity lies in the stacking phase (in particular, SABlocks). The computational complexity per batch for SABlock is $\mathit{O}\left({\mathit{LD}}^{\mathbf{2}}\right)$, where $\mathit{L}$ is the number of SABlock layer, and $\mathit{D}$ is the dimension of input data. Although this architecture does introduce some additional complexity than one-layer solutions, e.g., AFS, STG, and SANs, but this architecture can be efficiently executed via GPU calculation.

In

Table 8. Computational complexity per-iteration (in seconds).

Dataset	(Sam./Dim.)	FIR	AFS	SANs	STG	SAFS
SVHN	(10,000, 3072)	0.2128	0.0474	3.2687	0.1534	0.0748
MNIST	(8000, 784)	0.0710	0.0233	0.8126	0.0474	0.0316
GAS	(6000, 128)	0.0121	0.0073	0.1778	0.0111	0.0118

, the computational overheads of different DNN-based feature selection methods are illustrated. It can be observed that when the dataset is large (e.g., SVHN), the computational overhead of DNN-based methods increases significantly. The results show that although SAFS is deeper, its computational time is less than those of STG, SANs, and FIR and is a bit slower than AFS.

4.8. More Detailed Experiment Results

4.8.1. Overall Performance with Standard Deviation

Table 9. Average performance (Micro-F1↑) with LightGBM classifier with ten runs, ‘OT’ means overtime (more than 24 h of calculation in EPYC 7552*2 192 cores), ‘−’ means no result because of an internal error occurred. The best- and second-best results are highlighted in bold and with underline, respectively.

Algorithm	SEGMENT	SATIMAGE	GAS	DNA	HAR	ISOLET
LASSO	86.95 $\pm \mathbf{0.78}$	74.77 $\pm \mathbf{1.22}$	90.21 $\pm \mathbf{0.62}$	68.89 $\pm \mathbf{3.11}$	80.57 $\pm \mathbf{1.02}$	66.82 $\pm \mathbf{1.60}$
RFE	93.09 $\pm \mathbf{1.17}$	72.88 $\pm \mathbf{1.97}$	94.85 $\pm \mathbf{0.42}$	31.25 $\pm \mathbf{2.99}$	83.24 $\pm \mathbf{1.36}$	55.86 $\pm \mathbf{0.51}$
RF	96.50 $\pm \mathbf{0.84}$	83.44 $\pm \mathbf{1.83}$	89.37 $\pm \mathbf{3.13}$	87.48 $\pm \mathbf{2.48}$	93.78 $\pm \mathbf{0.67}$	76.73 $\pm \mathbf{2.25}$
XGB	90.52 $\pm \mathbf{0.88}$	83.88 $\pm \mathbf{2.23}$	95.37 $\pm \mathbf{0.26}$	82.74 $\pm \mathbf{2.77}$	93.35 $\pm \mathbf{0.53}$	68.38 $\pm \mathbf{1.45}$
CCM	95.12 $\pm \mathbf{0.58}$	82.56 $\pm \mathbf{2.47}$	93.63 $\pm \mathbf{0.82}$	64.59 $\pm \mathbf{5.76}$	82.77 $\pm \mathbf{7.83}$	59.12 $\pm \mathbf{2.66}$
FIR	91.73 $\pm \mathbf{4.45}$	73.88 $\pm \mathbf{5.49}$	93.61 $\pm \mathbf{2.04}$	43.33 $\pm \mathbf{7.02}$	76.72 $\pm \mathbf{6.23}$	63.28 $\pm \mathbf{5.77}$
AFS	95.98 $\pm \mathbf{1.05}$	82.09 $\pm \mathbf{2.56}$	96.21 $\pm \mathbf{0.94}$	72.67 $\pm \mathbf{8.47}$	91.56 $\pm \mathbf{2.96}$	75.20 $\pm \mathbf{2.21}$
SANs	94.04 $\pm \mathbf{1.84}$	79.11 $\pm \mathbf{2.69}$	94.37 $\pm \mathbf{2.29}$	61.03 $\pm \mathbf{5.74}$	88.77 $\pm \mathbf{1.96}$	68.79 $\pm \mathbf{5.11}$
FM	95.24 $\pm \mathbf{0.59}$	80.45 $\pm \mathbf{1.87}$	97.01 $\pm \mathbf{0.98}$	83.91 $\pm \mathbf{3.36}$	89.94 $\pm \mathbf{3.51}$	75.88 $\pm \mathbf{3.44}$
STG	96.33 $\pm \mathbf{0.43}$	81.77 $\pm \mathbf{3.46}$	95.57 $\pm \mathbf{1.79}$	79.55 $\pm \mathbf{5.88}$	94.06 $\pm \mathbf{0.72}$	77.94 $\pm \mathbf{1.72}$
NeuroFS	94.34 $\pm \mathbf{1.55}$	82.54 $\pm \mathbf{1.91}$	86.11 $\pm \mathbf{5.98}$	70.37 $\pm \mathbf{6.08}$	94.95 $\pm \mathbf{1.51}$	60.59 $\pm \mathbf{4.45}$
A-SFS	96.17 $\pm \mathbf{0.48}$	80.04 $\pm \mathbf{1.63}$	94.35 $\pm \mathbf{1.33}$	80.59 $\pm \mathbf{2.56}$	92.44 $\pm \mathbf{1.76}$	69.25 $\pm \mathbf{3.33}$
SEFS	95.59 $\pm \mathbf{0.47}$	83.42 $\pm \mathbf{0.97}$	96.94 $\pm \mathbf{0.83}$	76.13 $\pm \mathbf{4.29}$	−	68.07 $\pm \mathbf{4.24}$
SAFS-Pa	96.42 $\pm \mathbf{0.33}$	84.91 $\pm \mathbf{3.03}$	97.14 $\pm \mathbf{0.42}$	87.64 $\pm \mathbf{3.14}$	93.86 $\pm \mathbf{0.80}$	74.00 $\pm \mathbf{4.04}$
SAFS	96.75 $\pm \mathbf{1.40}$	84.33 $\pm \mathbf{3.06}$	97.99 $\pm \mathbf{0.39}$	89.33 $\pm \mathbf{3.41}$	96.17 $\pm \mathbf{0.83}$	80.84 $\pm \mathbf{2.16}$
	MNIST	Alon	Gravier	SVHN	CIFAR10	Chiaretti
LASSO	54.31 $\pm \mathbf{0.88}$	72.45 $\pm \mathbf{10.05}$	73.72 $\pm \mathbf{6.45}$	22.55 $\pm \mathbf{0.66}$	27.23 $\pm \mathbf{0.91}$	82.56 $\pm \mathbf{5.10}$
RFE	40.75 $\pm \mathbf{0.69}$	66.84 $\pm \mathbf{10.53}$	68.43 $\pm \mathbf{7.86}$	OT	OT	OT
RF	61.34 $\pm \mathbf{1.18}$	81.05 $\pm \mathbf{8.91}$	82.15 $\pm \mathbf{3.53}$	51.96 $\pm \mathbf{1.39}$	36.50 $\pm \mathbf{1.96}$	82.82 $\pm \mathbf{6.98}$
XGB	61.78 $\pm \mathbf{0.81}$	84.73 $\pm \mathbf{8.28}$	87.25 $\pm \mathbf{3.24}$	57.24 $\pm \mathbf{0.80}$	41.43 $\pm \mathbf{0.87}$	85.64 $\pm \mathbf{7.21}$
CCM	42.71 $\pm \mathbf{4.79}$	80.05 $\pm \mathbf{7.14}$	75.68 $\pm \mathbf{5.34}$	45.52 $\pm \mathbf{2.54}$	41.34 $\pm \mathbf{1.11}$	−
FIR	41.18 $\pm \mathbf{3.85}$	80.00 $\pm \mathbf{3.93}$	68.23 $\pm \mathbf{4.36}$	47.40 $\pm \mathbf{1.09}$	41.24 $\pm \mathbf{0.76}$	73.84 $\pm \mathbf{6.76}$
AFS	62.80 $\pm \mathbf{2.01}$	80.00 $\pm \mathbf{9.85}$	80.88 $\pm \mathbf{6.20}$	55.89 $\pm \mathbf{1.27}$	38.16 $\pm \mathbf{3.28}$	86.92 $\pm \mathbf{7.33}$
SANs	36.86 $\pm \mathbf{2.39}$	84.31 $\pm \mathbf{4.21}$	71.37 $\pm \mathbf{6.63}$	45.22 $\pm \mathbf{4.84}$	39.84 $\pm \mathbf{1.16}$	71.28 $\pm \mathbf{7.50}$
FM	63.67 $\pm \mathbf{2.35}$	78.02 $\pm \mathbf{8.41}$	77.57 $\pm \mathbf{7.19}$	57.92 $\pm \mathbf{0.89}$	42.87 $\pm \mathbf{2.25}$	78.85 $\pm \mathbf{9.42}$
STG	62.44 $\pm \mathbf{2.67}$	83.15 $\pm \mathbf{6.13}$	80.78 $\pm \mathbf{4.80}$	56.30 $\pm \mathbf{1.04}$	40.61 $\pm \mathbf{0.82}$	83.68 $\pm \mathbf{8.71}$
NeuroFS	55.61 $\pm \mathbf{3.97}$	78.59 $\pm \mathbf{9.41}$	80.26 $\pm \mathbf{4.55}$	45.90 $\pm \mathbf{1.46}$	42.42 $\pm \mathbf{0.73}$	87.91 $\pm \mathbf{6.67}$
A-SFS	64.16 $\pm \mathbf{2.02}$	75.78 $\pm \mathbf{7.13}$	72.55 $\pm \mathbf{8.36}$	42.14 $\pm \mathbf{1.70}$	38.94 $\pm \mathbf{1.95}$	74.36 $\pm \mathbf{7.72}$
SEFS	61.77 $\pm \mathbf{2.87}$	78.64 $\pm \mathbf{5.71}$	75.61 $\pm \mathbf{6.13}$	53.76 $\pm \mathbf{5.68}$	39.99 $\pm \mathbf{1.50}$	−
SAFS-Pa	64.71 $\pm \mathbf{2.08}$	85.38 $\pm \mathbf{7.25}$	85.58 $\pm \mathbf{3.83}$	54.40 $\pm \mathbf{2.08}$	41.66 $\pm \mathbf{1.22}$	88.23 $\pm \mathbf{7.84}$
SAFS	62.64 $\pm \mathbf{2.37}$	86.92 $\pm \mathbf{6.92}$	87.55 $\pm \mathbf{4.23}$	60.61 $\pm \mathbf{0.74}$	43.86 $\pm \mathbf{1.31}$	88.46 $\pm \mathbf{6.70}$

and

Table 10. Average performance (Micro-F1↑) with Catboost classifier with ten runs, ‘OT’ means overtime (more than 24 h of calculation in EPYC 7552*2 192 cores), ‘−’ means no result because of an internal error occurred. The best- and second-best results are highlighted in bold and with underline, respectively.

Algorithm	SEGMENT	SATIMAGE	GAS	DNA	HAR	ISOLET
LASSO	87.88 $\pm \mathbf{0.99}$	75.50 $\pm \mathbf{2.02}$	90.68 $\pm \mathbf{0.57}$	69.85 $\pm \mathbf{3.44}$	80.51 $\pm \mathbf{1.39}$	68.20 $\pm \mathbf{1.49}$
RFE	93.83 $\pm \mathbf{0.90}$	77.89 $\pm \mathbf{5.94}$	95.87 $\pm \mathbf{1.52}$	46.67 $\pm \mathbf{10.72}$	85.58 $\pm \mathbf{1.48}$	63.72 $\pm \mathbf{2.69}$
RF	97.42 $\pm \mathbf{0.62}$	84.38 $\pm \mathbf{1.81}$	89.50 $\pm \mathbf{3.14}$	87.11 $\pm \mathbf{2.63}$	93.81 $\pm \mathbf{0.86}$	77.29 $\pm \mathbf{1.97}$
XGB	90.69 $\pm \mathbf{0.84}$	83.72 $\pm \mathbf{1.53}$	95.40 $\pm \mathbf{0.35}$	82.44 $\pm \mathbf{2.86}$	92.67 $\pm \mathbf{0.41}$	69.23 $\pm \mathbf{0.86}$
CCM	96.12 $\pm \mathbf{0.76}$	82.67 $\pm \mathbf{2.31}$	94.05 $\pm \mathbf{1.92}$	68.51 $\pm \mathbf{3.07}$	89.83 $\pm \mathbf{3.11}$	67.01 $\pm \mathbf{6.09}$
FIR	94.04 $\pm \mathbf{5.73}$	82.22 $\pm \mathbf{3.89}$	92.24 $\pm \mathbf{2.81}$	40.74 $\pm \mathbf{3.12}$	76.83 $\pm \mathbf{5.20}$	65.33 $\pm \mathbf{3.34}$
AFS	95.02 $\pm \mathbf{2.16}$	78.67 $\pm \mathbf{3.01}$	96.38 $\pm \mathbf{1.01}$	77.89 $\pm \mathbf{4.11}$	91.80 $\pm \mathbf{2.21}$	75.96 $\pm \mathbf{2.15}$
SANs	93.66 $\pm \mathbf{2.87}$	79.55 $\pm \mathbf{3.34}$	94.30 $\pm \mathbf{2.60}$	59.40 $\pm \mathbf{7.53}$	89.11 $\pm \mathbf{2.05}$	70.71 $\pm \mathbf{5.03}$
STG	97.09 $\pm \mathbf{0.45}$	83.77 $\pm \mathbf{1.70}$	96.44 $\pm \mathbf{1.10}$	86.22 $\pm \mathbf{5.49}$	91.20 $\pm \mathbf{3.92}$	75.35 $\pm \mathbf{4.50}$
FM	95.33 $\pm \mathbf{0.57}$	80.76 $\pm \mathbf{1.79}$	97.12 $\pm \mathbf{0.96}$	83.47 $\pm \mathbf{3.56}$	90.45 $\pm \mathbf{3.17}$	76.33 $\pm \mathbf{3.03}$
NeuroFS	95.28 $\pm \mathbf{1.56}$	83.03 $\pm \mathbf{1.24}$	88.49 $\pm \mathbf{4.82}$	82.05 $\pm \mathbf{5.87}$	95.22 $\pm \mathbf{2.38}$	58.47 $\pm \mathbf{5.27}$
A-SFS	95.57 $\pm \mathbf{3.55}$	83.55 $\pm \mathbf{1.08}$	94.35 $\pm \mathbf{1.96}$	80.51 $\pm \mathbf{2.77}$	87.73 $\pm \mathbf{4.20}$	76.64 $\pm \mathbf{4.79}$
SEFS	96.08 $\pm \mathbf{0.57}$	82.79 $\pm \mathbf{2.46}$	96.67 $\pm \mathbf{0.74}$	83.33 $\pm \mathbf{2.49}$	−	77.02 $\pm \mathbf{2.91}$
SAFS-Pa	96.96 $\pm \mathbf{0.97}$	84.50 $\pm \mathbf{2.53}$	97.12 $\pm \mathbf{0.57}$	86.22 $\pm \mathbf{6.72}$	94.43 $\pm \mathbf{0.80}$	75.44 $\pm \mathbf{3.40}$
SAFS	97.17 $\pm \mathbf{1.15}$	82.62 $\pm \mathbf{1.95}$	98.18 $\pm \mathbf{0.25}$	89.11 $\pm \mathbf{3.36}$	95.95 $\pm \mathbf{0.72}$	81.82 $\pm \mathbf{4.20}$
LASSO	56.85 $\pm \mathbf{0.74}$	78.42 $\pm \mathbf{7.96}$	73.92 $\pm \mathbf{6.38}$	25.68 $\pm \mathbf{0.55}$	29.29 $\pm \mathbf{0.71}$	78.68 $\pm \mathbf{7.20}$
RFE	43.68 $\pm \mathbf{0.52}$	OT	OT	OT	OT	OT
RF	63.22 $\pm \mathbf{1.28}$	83.15 $\pm \mathbf{7.36}$	82.54 $\pm \mathbf{5.21}$	58.01 $\pm \mathbf{1.82}$	39.27 $\pm \mathbf{1.39}$	84.47 $\pm \mathbf{9.07}$
XGB	64.25 $\pm \mathbf{0.61}$	85.79 $\pm \mathbf{8.17}$	89.21 $\pm \mathbf{5.13}$	63.38 $\pm \mathbf{0.60}$	43.88 $\pm \mathbf{1.02}$	89.73 $\pm \mathbf{7.57}$
CCM	56.10 $\pm \mathbf{4.25}$	84.21 $\pm \mathbf{8.15}$	76.07 $\pm \mathbf{7.27}$	52.04 $\pm \mathbf{1.54}$	43.97 $\pm \mathbf{0.62}$	−
FIR	38.23 $\pm \mathbf{2.77}$	83.15 $\pm \mathbf{5.15}$	74.51 $\pm \mathbf{5.26}$	55.82 $\pm \mathbf{1.21}$	43.76 $\pm \mathbf{0.40}$	69.74 $\pm \mathbf{6.95}$
AFS	65.10 $\pm \mathbf{1.82}$	83.07 $\pm \mathbf{5.75}$	80.51 $\pm \mathbf{6.19}$	58.39 $\pm \mathbf{2.10}$	42.56 $\pm \mathbf{0.98}$	85.38 $\pm \mathbf{11.76}$
SANs	30.32 $\pm \mathbf{6.67}$	85.36 $\pm \mathbf{5.36}$	76.47 $\pm \mathbf{7.12}$	54.26 $\pm \mathbf{2.47}$	41.27 $\pm \mathbf{0.78}$	72.30 $\pm \mathbf{7.84}$
STG	65.11 $\pm \mathbf{2.65}$	86.88 $\pm \mathbf{5.36}$	80.78 $\pm \mathbf{5.13}$	59.75 $\pm \mathbf{1.46}$	43.22 $\pm \mathbf{0.99}$	81.57 $\pm \mathbf{8.88}$
FM	64.02 $\pm \mathbf{2.26}$	81.14 $\pm \mathbf{7.56}$	77.73 $\pm \mathbf{6.97}$	59.26 $\pm \mathbf{0.47}$	44.56 $\pm \mathbf{0.63}$	76.30 $\pm \mathbf{8.24}$
NeuroFS	57.64 $\pm \mathbf{2.31}$	82.87 $\pm \mathbf{8.52}$	84.26 $\pm \mathbf{4.74}$	46.62 $\pm \mathbf{1.57}$	43.58 $\pm \mathbf{0.69}$	86.67 $\pm \mathbf{7.10}$
A-SFS	65.99 $\pm \mathbf{2.40}$	76.38 $\pm \mathbf{4.70}$	74.07 $\pm \mathbf{8.54}$	56.29 $\pm \mathbf{4.14}$	42.23 $\pm \mathbf{2.67}$	76.92 $\pm \mathbf{6.45}$
SEFS	65.42 $\pm \mathbf{2.14}$	80.29 $\pm \mathbf{6.67}$	77.89 $\pm \mathbf{7.52}$	54.78 $\pm \mathbf{3.62}$	43.01 $\pm \mathbf{2.78}$	−
SAFS-Pa	66.63 $\pm \mathbf{2.77}$	86.15 $\pm \mathbf{4.61}$	84.61 $\pm \mathbf{4.56}$	60.74 $\pm \mathbf{2.21}$	45.29 $\pm \mathbf{1.38}$	85.76 $\pm \mathbf{10.88}$
SAFS	65.33 $\pm \mathbf{1.79}$	87.38 $\pm \mathbf{3.52}$	88.11 $\pm \mathbf{5.12}$	66.86 $\pm \mathbf{0.88}$	46.89 $\pm \mathbf{1.27}$	89.69 $\pm \mathbf{8.13}$

present the experimental results with 13 baselines across two robust classifiers. Notably, SAFS achieves good results on all datasets and shows significant improvements over the compared baselines, including several state-of-the-art methods. When utilizing with the CatBoost classifier, most baseline results demonstrate enhancement, particularly for RF and XGB, but SAFS maintains leading compared to baselines, particularly on large-scale datasets such as ISOLET, SVHN, and CIFAR-10.

For the main results on Table 9, compared to the average performance of other baselines, SAFS can improve ranging from 3.08% to 26.01%. The improvements on datasets with high dimensions, e.g., HAR, ISOLET, SVHN, and Chiar. are especially significant. This fact highlights the effectiveness of SAFS on high-dimensional data.

Self-supervised solutions, A-SFS and SEFS, display no significant performance improvement since they typically depend on additional unlabeled data for self-supervision and offer limited enhancements to the feature weight generation module designed for the high-dimensional nature of the dataset.

These results suggest that the stacked architecture can significantly improve the ability to capture complex feature interactions compared to existing one-layer-based approaches.

4.8.2. Performances Under Different TopK Features

This section compares the performance of feature selection across varying numbers of features. Figure 11a–i shows the classification accuracy for nine different datasets.

It is evident that SAFS achieves consistent performance advantages over other baselines across nearly all evaluated TopK ranges. This merit is rather important in real-world FS applications, as it shows that SAFS generates high-quality feature weights for different features: both for highly related features and those less related but still valuable features.

4.8.3. Feature Selection in m > n Regime

The performance of SAFS is next evaluated under a more challenging regime where the number of features exceeds the number of samples ($\mathit{m}>\mathit{n}$). In some real-world scenarios, only limited samples are labeled. In this feature selection regime, only 10% of the samples are randomly selected from the raw datasets, with 70% allocated for training and the remaining 30% for testing. To better demonstrate overall performance, Figure 12 shows the performance of different baselines on different TopK. Across almost all ranges, SAFS achieves the best performance in this challenging regime with a few samples per class, and keeps a constant increment of performance with increasing K. This experiment shows that SAFS can effectively weight high-dimensional features with limited labels.

4.8.4. Detailed Ablation Results

To better understand the impact of components in SAFS, ablation studies are conducted on key components by systematically removing each one to observe performance changes across multiple datasets.

Table 11. Ablation studies. SAFS-s: Remove the stack architecture, do FS only use the SABlock; SAFS-bn: Remove the BN-layer of SABlock; SAFS-i: Remove the inertia-based weight update strategy; SAFS-c: Remove the feature skip connection. The best and second-best results are highlighted in bold and underline, respectively.

Datasets	SAFS	SAFS-s	SAFS-bn	SAFS-i	SAFS-c
SVHN	60.61 ± 0.74	57.03 ± 3.18	57.82 ± 3.23	59.40 ± 1.06	59.47 ± 1.43
CIFAR10	43.86 ± 1.31	41.48 ± 1.29	41.39 ± 1.76	43.61 ± 0.90	43.57 ± 1.33
MNIST	62.64 ± 2.37	64.04 ± 2.77	59.98 ± 4.12	63.65 ± 2.05	62.38 ± 1.74
ISOLET	80.84 ± 2.16	73.01 ± 3.68	65.73 ± 0.66	78.20 ± 2.68	79.09 ± 2.29
HAR	96.17 ± 0.83	94.85 ± 0.81	82.25 ± 5.84	94.25 ± 1.80	95.95 ± 0.92
DNA	89.33 ± 3.41	85.67 ± 6.67	87.22 ± 3.10	88.88 ± 3.44	88.11 ± 4.95
GAS	97.99 ± 0.39	96.83 ± 0.72	93.95 ± 2.07	97.08 ± 0.81	97.65 ± 0.59
SATIMAGE	84.33 ± 3.06	84.67 ± 2.42	81.33 ± 7.09	83.08 ± 3.07	82.37 ± 2.93
SEGMENT	96.75 ± 1.37	96.43 ± 0.71	96.64 ± 1.10	96.04 ± 1.86	96.17 ± 1.31

clearly shows that all designs are important for improving performance. When the stack architecture is removed, the performance drops significantly on high-dimensional datasets, and the BN-layer reduces the standard deviation. From the results, the feature skip connection design and the inertia-based update strategy also increase the stability of the SAFS.

5. Case Study: SAFS in Short-Term Wind Power Forecasting

• Background. With increasing energy demands and environmental concerns from fossil fuels, renewable energy sources are essential [45]. Wind energy, being pollution-free and abundant, has gained significant global attention. Accurate wind power prediction is crucial for integrating wind energy into power grids. The high number of input features of the forecasting system negatively affects the accuracy of performance and computational time. To address this issue, some studies apply feature selection methods in the wind power systems [46,47].
• Dataset. These dataset is publicly available on the Kaggle platform (https://www.kaggle.com/datasets/aymanlafaz/wind-energy-germany) (accessed on 4 November 2025), and it was collected continuously from 1 January 2011 00:00:00 to 30 December 2021 07:45:00, with data recorded every 15 min. The unit includes 76 sensor monitoring points, such as the temperature of the heat exchanger converter, blade torque, and line-to-line voltage (phase voltage), resulting in 7392 feature value entries per day, which is approximately 192 MB.

To evaluate the proposed feature selection method on time-series data, experiments were conducted using data generated between 1 January 2020 and 30 June 2020. The dataset was chronologically split into training and testing sets with an 8:2 ratio, comprising 20,889 samples for training and 5223 for testing. The statistical information of the collected wind power data are presented in

Table 12. Description of the wind power dataset.

Statistics	Mean	Std	Min	25%	50%	75%	Max
Power (kW)	1052.86	1083.11	−42.19	33.75	588.81	2218.21	2777.19

.

• Data cleaning. Features with a missing value ratio exceeding 50% and outliers were processed using previous value imputation to maximally preserve working condition invariance. For features with missing values below 50%, the local sample relationships were considered by applying K-NN imputation (K = 5) to estimate missing values. Finally, min–max normalization was applied to each feature to facilitate faster convergence of SAFS.
• Experiment. The experiment aims to predict wind power in different short-term periods, LightGBM regressor is used to evaluate the feature subsets found by different FS methods.

Figure 13 shows the wind power prediction results using different feature selection methods. The results on the left show that compared with using all features (Baseline), the RMSE and MAE are improved when only TOP-10 features are selected for prediction, and STG and SAFS have better results. This indicates that through feature selection, the impact of redundant/irrelevant features on the prediction model can be reduced. From the curve in Figure 13 (right), compared with STG and Baseline, the gap between the TOP-10 prediction curve of SAFS and the true curve is smaller, suggesting that SAFS is beneficial for improving the accuracy of wind power prediction.

6. Limitation and Future Work

While the SAFS method proposed in this paper demonstrates promising results in feature selection, it is imperative to acknowledge certain limitations and outline potential directions for future research.

Firstly, the application of SAFS has not yet been extensively tested in industrial settings where domain-specific knowledge plays a critical role. This highlights the need for more empirical studies to validate and refine SAFS in various real-world scenarios, ensuring its adaptability and effectiveness across different industrial domains.

Secondly, the theoretical underpinnings of SAFS are still in nascent stages. Our future work will develop a comprehensive theoretical analysis that can provide insight into the convergence properties, optimality conditions, and scalability aspects of SAFS.

Lastly, the architectural choices within SAFS present another avenue for exploration. It remains an open question whether alternative structural configurations or enhancements could further improve its performance and efficiency. Investigating different construction strategies and integrating advanced techniques could potentially lead to more robust versions of SAFS.

Addressing these limitations will not only strengthen the credibility and applicability of SAFS but also contribute to the broader field of feature selection in machine learning.

7. Conclusions

This paper proposes SAFS, a novel feature selection framework that can evaluate the importance of features. Specifically, a new FS module, SABlock, is designed to capture complex feature interactions. This architecture is designed to be stackable, and SAFS can be built based on this fundamental block. Then, feature skip connection and inertia-based weight update method are designed to enhance the performance of SAFS. Experiments on nine real-world datasets from diverse domains validate that the proposed model discovers features that deliver superior prediction performance for classification tasks. Analysis of the weight distribution across layers provides explanatory insights into the effectiveness of the stacked architecture. Further experiments prove that SAFS can effectively select features in multiple regimes, such as few samples and noise disturbance, which is important for practical applications. The ablation and sensitivity analyses illustrate the effectiveness of our design. One of our future works will be to verify its applicability in real-world applications with domain experts.