Supervised Feature Selection Method Using Stackable Attention Networks

Mainstream DNN-based feature selection methods share a similar design strategy: employing one specially designed feature selection module to learn the importance of features along with the model-training process. While these works achieve great success in feature selection, their shallow structures, which evaluate feature importance from one perspective, are easily disturbed by noisy samples, especially in datasets with high-dimensional features and complex structures. To alleviate this limitation, this paper innovatively introduces a Stackable Attention architecture for Feature Selection (SAFS), which can calculate stable and accurate feature weights through a set of Stackable Attention Blocks (SABlocks) rather than from a single module. To avoid information loss from stacking, a feature jump concatenation structure is designed. Furthermore, an inertia-based weight update method is proposed to generate a more robust feature weight distribution. Experiments on twelve real-world datasets, including multiple domains, demonstrate that SAFS produced the best results with significant performance edges compared to thirteen baselines.