DFSel-FT: A Differentiable Feature Selection and FT-Transformer Framework for Interpretable Thyroid Disease Classification Using Tabular Data

Abstract

Thyroid diseases are very common endocrine diseases that afflict millions of people around the world and need proper and timely diagnosis to ensure proper treatment. Although machine learning and hybrid metaheuristic methods have advanced, current models have high computation costs, low interpretability, and low probability calibration, which limit their use in clinical settings. In this research, a new DFSel-FT (Differentiable Feature Selection and an FT-Transformer) system is suggested, which combines DFSel-FT to allow one to diagnose thyroid disease effectively and interpretably. It employs Concrete (Gumbel-Softmax) gates to select the features end-to-end to make sure that only the most relevant clinical attributes are carried through the training. A Transformer-based architecture is then used to process the chosen features to learn intricate interdependencies. The model is trained with class-balanced focal loss and temperature scaling to better enhance calibration. Experimental evaluation on the UCI Thyroid Disease Dataset (22,632 samples) showed that the proposed model achieved 97.85% accuracy, 97.65% Macro-F1, and 98.10% AUC-OVR, showing competitive performance compared with traditional machine learning models, modern tabular deep learning baselines, and hybrid metaheuristic methods. Other indicators of robustness and reliability include MCC (0.955), Cohen Kappa (0.951), and small calibration error (ECE = 0.021). SHAP and LIME explainability analysis reveals clinically relevant features that include TSH, TT4, and T3. The proposed framework provides a balanced integration of predictive performance, interpretability, and probability calibration, making it a promising benchmark-level framework for interpretable and calibrated thyroid disease classification, requiring external clinical validation before real-world deployment.

1. Introduction

Thyroid disorders are considered one of the most prevalent endocrine diseases in the world, as an estimated 300 million people worldwide and almost 42 million in India alone are affected [1,2]. The prevalence of hypothyroidism is between 5 and 10 percent of the general population, and hyperthyroidism is between 1 and 2 percent [3,4]. Epidemiological research indicates a disproportionate impact on women with a female-to-male ratio of about 6:1 [5]. The rates of clinical misdiagnoses are significant, with an estimate of up to 20–25 percent of hypothyroidism cases being misclassified or diagnosed late due to similarities with other metabolic disorders. On the computational side, models of thyroid diagnosis using machine learning with the UCI Thyroid dataset (22,632 records, 28 attributes) have reached an accuracy of 92–98 percent [6]. Nevertheless, there is still a challenge of overcoming the 97.5% threshold of accuracy and still having interpretability and reliability.

The thyroid gland is important in the regulation of metabolism by the hormones triiodothyronine (T3) and thyroxine (T4). Hypothyroidism, hyperthyroidism, nodular thyroid disease or thyroid cancer are caused by dysfunctions of thyroid hormone regulation. The clinical diagnosis mainly depends on laboratory tests (TSH, T3, TT4, T4U) as well as demographic data (age, sex, and treatment history) [7,8]. Although these structured features offer very useful information, they tend to be multidimensional and noisy, and therefore feature selection is a critical process [9].

Classical machine learning (Decision Trees, Random Forests, SVMs, KNN) or metaheuristic-based feature selection methods have been used as traditional methods in predicting thyroid disease, including Genetic Algorithms, Particle Swarm Optimization, or hybrid bio-inspired methods like the Cuttlefish Algorithm with Simulated Annealing (CFA–SA) [10,11,12]. These techniques have been reported to give competitive results, but are restricted by their wrapper-style optimization, which is computationally intensive, and their lack of the ability to simultaneously learn both feature importance and classification in an end-to-end fashion.

Although there have been significant advances in computational techniques to detect thyroid disease, three issues have not been addressed, and they include:

• Non-differentiable feature selection: Metaheuristic optimization is effective but disconnected from the classifier, leading to inefficiency and instability.
• Interpretability trade-off: Many high-accuracy models (e.g., ensemble learners, deep networks) act as “black boxes,” providing little insight into which attributes truly matter.
• Calibration and robustness issues: Existing models achieve high accuracy but often lack well-calibrated probability estimates, which are crucial for clinical reliability.

To overcome these restrictions, we suggest a new Differentiable Feature Selection + Transformer-based Tabular Learning system. The method presents Concrete (Gumbel-Softmax) gates to execute end-to-end differentiable feature selection, which guarantees that the most discriminative thyroid features are selected by default in training. The chosen features are then run through a Tabular Transformer that uses self-attention to learn higher-order dependence amongst clinical features. Moreover, we combine the class-balanced focal loss to overcome the class imbalance and use post hoc calibration methods (temperature scaling) to increase clinical credibility.

To the best of our knowledge, this study is among the early attempts to integrate differentiable feature selection with an FT-Transformer backbone for thyroid tabular classification. Unlike earlier approaches that rely mainly on heuristic or wrapper-based feature selection, the proposed DFSel-FT framework learns feature relevance and classification representations jointly in an end-to-end manner. This design reduces dependence on externally optimized feature subsets, improves model transparency through learned feature gates, and supports more reliable probability estimation through calibration. However, the framework is positioned as a benchmark-level computational model, and its use in real-world clinical settings requires further external validation, prospective testing, and expert clinical review. The main contributions of this research are as follows:

• End-to-End Differentiable Feature Selection—Implementation of Concrete gates to identify the most discriminative thyroid attributes dynamically during training.
• Transformer-Based Tabular Modeling—Adoption of a self-attention mechanism to capture complex dependencies among clinical and biochemical features.
• Imbalance-Aware Learning—Use of class-balanced focal loss to handle skewed class distributions effectively.
• Reliable Probability Estimation—Integration of post hoc temperature scaling to improve probability calibration, which is important for risk-aware medical classification tasks.
• Benchmark-Level Performance Evaluation—Demonstration of competitive performance on the UCI Thyroid Dataset against traditional machine learning and hybrid feature-selection baselines, with further validation required on external clinical datasets.

2. Literature Review

The existing literature on thyroid disease prediction is reviewed across different streams, as discussed below.

2.1. Machine Learning and Deep Learning for Thyroid Tabular Classification

Artificial intelligence (AI) has played an important role in enhancing the process of diagnostic evaluation of endocrine diseases like thyroid disease. Classical machine learning algorithms such as Decision Trees, Naïve Bayes, Random Forests, K-Nearest Neighbors, and Support Vector Machines have been widely investigated in classifying thyroid conditions [13,14]. The results of these models showed good predictability when used on benchmark data such as the UCI Thyroid Disease dataset. Ensemble and hybrid models were investigated more recently, in which simpler models tended to perform worse than Random Forest and Decision Tree-based classifiers because of the nonlinearities of thyroid data [15,16]. Furthermore, TSH, T3, and T4 data mining methods that correlate with hypothyroidism and hyperthyroidism among the different demographics have strengthened the possibility of AI in complementing conventional diagnostic parameters. Nonetheless, issues like imbalance in classes, low interpretability, and medium scalability have remained, and more sophisticated methodologies are required.

2.2. Feature Selection in Medical Tabular Data

Recent studies have resorted to nature-inspired metaheuristics in the optimization of features in medical data. The original paper proposed a two-step hybrid method that involves the Cuttlefish Algorithm (CFA) and Simulated Annealing (SA) to select the best feature subset. The CFA produced sets of candidate features using reflection and visibility processes based on the behavior of cuttlefish skin cells, and SA optimized the subsets with probabilistic acceptance of solutions to avoid local optima [12]. Shrivastava et al. reported that the CFA–SA + XGBoost model achieved the highest performance among their tested classifiers, with an accuracy of 98.91%, precision of 98.00%, recall of 98.12%, and F1-score of 94.83%. However, this value is treated in the present study as a literature-reported benchmark rather than a directly comparable reproduced result, because the exact preprocessing sequence, fold-wise resampling strategy, split configuration, and metric averaging procedure may differ from the leakage-free validation protocol adopted in this work. Therefore, CFA–SA-based models were additionally reimplemented under the present 10-fold stratified cross-validation setting for direct comparison with DFSel-FT. This performance outperformed single metaheuristics and single classical ML models, showing the effectiveness of combining optimization methods and machine learning. However, the method was problematic: the process of feature selection was computationally complex, interpretability was lacking, and external validation in a wide range of cohorts was not provided. These shortcomings highlight the need to find ways to combine efficiency, interpretability, and strong statistical validation.

2.3. Deep Learning for Tabular Data

Deep learning for tabular data has received increasing attention through architectures such as MLP, TabNet, TabTransformer, SAINT, NODE, and FT-Transformer [17,18,19]. These models attempt to capture nonlinear feature relationships and higher-order interactions that may not be fully captured by traditional machine learning models. FT-Transformer is particularly relevant because it adapts the Transformer architecture to tabular data by representing each feature as a token and applying self-attention across feature embeddings [20,21]. However, Transformer-based tabular models may still include redundant features if no feature-selection mechanism is integrated. Therefore, combining differentiable feature selection with FT-Transformer can provide a more compact and interpretable tabular learning framework.

2.4. Explainability and Calibration in Medical AI

Transformers, initially created in the domain of natural language processing, have been infiltrating healthcare tasks because of their capability to capture long-range dependencies through their self-attention mechanisms [22,23,24,25]. Although several recent studies have used Transformer-based models for thyroid ultrasound image classification and nodule detection, these imaging-based approaches differ methodologically from the present tabular classification problem. Therefore, the present review focuses mainly on thyroid tabular classification, differentiable feature selection, tabular deep learning, explainability, and calibration [26,27,28,29,30]. Such methods point to a paradigm shift, with transformers being promising replacements for CNN-based models in the medical imaging domain and promising tabular clinical data modeling tools [31,32].

2.5. Research Gap

Existing thyroid disease classification studies have mainly focused on classical machine learning, metaheuristic feature selection, or imaging-based deep learning. Comparatively fewer studies have examined end-to-end differentiable feature selection combined with Transformer-based tabular learning for thyroid classification. Moreover, many prior studies do not jointly report predictive performance, explainability, calibration, and statistical validation under a leakage-free cross-validation protocol. To address these gaps, the present study proposes DFSel-FT, which integrates Concrete gate-based differentiable feature selection with an FT-Transformer backbone for interpretable and calibrated thyroid disease classification using tabular data.

3. Dataset Description

In this paper, we use the UCI Thyroid Disease Dataset, which is a well-known benchmark dataset that can be found at the UCI Machine Learning Repository [33]. It has 22,632 records of patients and 28 attributes such as demographic details, medical history, and measurements of biochemicals. The dataset provides a suitable basis for computational thyroid disease classification.

The present study does not use CFA–SA for feature selection in the proposed model. CFA–SA is considered only as a reference method from the previous literature and as a comparative baseline where applicable. In the proposed DFSel-FT framework, feature selection is performed through differentiable Concrete/Gumbel-Softmax gates that are trained jointly with the FT-Transformer backbone. Therefore, all attributes are initially treated as candidate input features, and their final relevance is learned during model training.

The main target variable is categorizing the patients as hypothyroid, hyperthyroid, or normal thyroid. The key demographic, biochemical, and categorical attributes of the UCI Thyroid Disease Dataset are summarized in

Table 1. Attributes of the UCI Thyroid Disease Dataset.

Feature	Type	Description	Range/Values	Role in Proposed Model
Age	Numeric	Patient’s age in years	1–100+	Candidate feature; gate-learned relevance
Sex	Categorical	Gender of the patient	Male/Female	Candidate feature; gate-learned relevance
On_thyroxine	Binary	Whether patient is on thyroxine treatment	Yes/No	Candidate feature; gate-learned relevance
Query_on_thyroxine	Binary	Whether thyroid therapy is under consideration	Yes/No	Candidate feature; gate-learned relevance
I131_treatment	Binary	Indicates radioactive iodine (I131) treatment	Yes/No	Candidate feature; gate-learned relevance
Query_hypothyroid	Binary	Query for hypothyroidism symptoms	Yes/No	Candidate feature; gate-learned relevance
Query_hyperthyroid	Binary	Query for hyperthyroidism symptoms	Yes/No	Candidate feature; gate-learned relevance
Pregnant	Binary	Indicates pregnancy status	Yes/No	Candidate feature; gate-learned relevance
TSH (Thyroid Stim. H.)	Numeric	Thyroid Stimulating Hormone level	0.005–530 mIU/L	Candidate feature; gate-learned relevance
T3 (Triiodothyronine)	Numeric	T3 hormone concentration	0.1–10 ng/mL	Candidate feature; gate-learned relevance
TT4 (Total T4)	Numeric	Total thyroxine level	2–700 μg/dL	Candidate feature; gate-learned relevance
T4U (Thyroxine Uptake)	Numeric	T4 uptake index	0.2–2.0	Candidate feature; gate-learned relevance
FTI (Free Thyroxine)	Numeric	Free thyroxine index	0–250	Candidate feature; gate-learned relevance
Referral_source	Categorical	Source of patient referral (self, physician, other hospital, etc.)	Nominal categories	Candidate feature; gate-learned relevance
Class (Target)	Categorical	Final classification outcome	Normal/Euthyroid, Hypothyroid, Hyperthyroid	Target variable for three-class classification

. All input attributes were treated as candidate features. Their final relevance was learned through differentiable Concrete/Gumbel-Softmax gates in the proposed DFSel-FT framework.

The UCI Thyroid Disease Dataset was organized into a three-class classification problem. The target labels were mapped as 0 = Normal/Euthyroid, 1 = Hypothyroid, and 2 = Hyperthyroid. Missing predictor values were retained and handled through fold-wise imputation during preprocessing. Ambiguous or non-mappable labels and duplicate records were checked before model training.

4. Proposed Methodology

The suggested methodology presents a new Differentiable Feature Selection + Transformer-based Tabular Learning model to predict thyroid disease. The methodology aims to jointly learn discriminative features and the intricate interactions between them in an end-to-end differentiable way, which is unattainable with the conventional optimization-based wrapper methods. Figure 1 presents the block diagram with the sequential steps of the proposed framework, which include the data collection and preprocessing followed by differentiable feature selection, FT-Transformer modeling, optimization and training, calibration and, finally, performance evaluation and explainability analysis.

The methodology consists of four stages: (i) data preprocessing, (ii) differentiable feature selection, (iii) tabular Transformer modeling, and (iv) training with imbalance-aware loss and calibration.

4.1. Data Preprocessing

The UCI Thyroid Disease Dataset ($D$) consists of $n$ = 22,632 records and $m$ = 28 attributes. Each sample is represented as Equation (1).

$$D={\left\{\left(x_i,y\right)\right\}}_{i=1,}^n{x}_i\in {\mathbb{R}}^m,y_i\in \left\{0,1,2\right\}$$

(1)

where $y_i$ = 0 (Normal), 1 (Hypothyroid) or 2 (Hyperthyroid).

Preprocessing steps:

• Missing values: Numerical features imputed with the mean; categorical features with the mode.
• Encoding: One-hot encoding for categorical variables (e.g., sex, referral source).
• Normalization: Min–Max scaling for numerical features can be seen in Equation (2)

$$x^{\prime }={\displaystyle \frac{x-x_{min}}{x_{max}-x_{min}}}$$

(2)

• Class balancing: SMOTE was applied only to the training partition within each cross-validation fold after imputation, encoding, and normalization.

Leakage-Free Cross-Validation Preprocessing Protocol

To avoid data leakage, all preprocessing operations were performed independently within each fold of the 10-fold stratified cross-validation protocol. In each fold, the training partition was first separated from the validation/test partition. Missing-value imputation, one-hot encoding, and Min–Max normalization were fitted only on the training data and then applied to the corresponding validation/test data. SMOTE was applied only to the training partition after imputation, encoding, and scaling. No synthetic samples were generated from, or introduced into, the validation/test fold. Differentiable feature selection, model training, early stopping, and temperature-scaling calibration were also conducted within the training/validation structure of each fold, while the held-out test fold remained unseen until final evaluation.

4.2. Differentiable Feature Selection

To identify the most relevant features, we introduce learned feature gates $z_k$ for each feature $k$ ∈ {1, …, m}. These gates are parameterized using the Concrete (Gumbel-Softmax) distribution, allowing end-to-end differentiability. For feature $k$, this can be represented as Equation (3).

$${\widehat{z}}_k=\sigma \left({\displaystyle \frac{\log {\alpha }_k+g_k}{\tau }}\right),g_k~Logistic(0,1)$$

(3)

where

• ${\alpha }_k$ is a learnable parameter,
• $\tau$ is the relaxation temperature (annealed during training),
• $\sigma$ is the sigmoid function.

Each input feature is masked as Equation (4)

$${\tilde{x}}_{i,k}={\widehat{z}}_k·x_{i,k}$$

(4)

The sparsity of selected features is encouraged via an $L0$-like penalty, as can be seen in Equation (5)

$${\mathcal{L}}_{L0}=\mathsf{\lambda }{\displaystyle {\sum }_{k=1}^m}\sigma \left({\alpha }_k\right)\lambda$$

(5)

where λ controls the degree of sparsity.

4.3. Transformer for Tabular Data

The gated feature vector ${\tilde{x}}_i$ is converted into feature tokens, as given in Equation (6).

$$t_{i,k}=e_k+W_v{\tilde{x}}_{i,k},t_{i,k}\in {\mathbb{R}}^d$$

(6)

where $e_k$ is a learnable embedding of feature $k$, and $W_v$ projects feature values into a d-dimensional space.

The tokens are processed using a Transformer encoder with multi-head self-attention [34,35]. This can be represented by Equations (7)–(9).

$$Attention\left(Q,K,V\right)=softmax\left({\displaystyle \frac{Q{K}^T}{\sqrt{d_h}}}\right)V$$

(7)

$$H^{(l+1)}=LayerNorm\left(H^{\left(l\right)}+MHSA\left(H^{\left(l\right)}\right)\right)$$

(8)

$$H^{(l+1)}=LayerNorm\left(H^{\left(l+1\right)}+FFN\left(H^{\left(l+1\right)}\right)\right)$$

(9)

where $H^{\left(l\right)}$ is the hidden state at layer $l$, and $MHSA$/$FFN$ denote multi-head self-attention and feed-forward network, respectively.

Equation (10) shows that the [$CLS$] token embedding or pooled representation is passed to a classifier head producing class probabilities:

$${\widehat{y}}_i=softmax(W_C{h}_{\left[CLS\right]}+b_C)$$

(10)

The detailed computational flow of the proposed DFSel-FT framework, along with an illustrative numerical example for feature preprocessing, gating, embedding, attention, and classification, is presented in Figure 2.

4.4. Training Objective

The training objective of the proposed DFSel-FT framework is to optimize classification performance while addressing class imbalance and probability calibration. This is achieved through the integration of class-balanced focal loss with sparsity regularization and calibration techniques.

To address class imbalance, we use class-balanced focal loss with Equation (11)

$${\mathcal{L}}_{CB-Focal}=-{\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}\sum _c{w}_c{\left(1-p_{i,c}\right)}^{\gamma }{y}_{i,c}log{p}_{i,c}$$

(11)

where

• $w_c={\displaystyle \frac{1-\beta }{1-{\beta }^{n_c}}}$ is the class weight based on effective sample count $n_c$;
• γ controls the focusing effect;
• $p_{i,c}$ is the predicted probability for class $c$.

4.5. Total Loss

Equation (12) shows that the final training objective combines classification loss and sparsity penalty:

$$\mathcal{L}={\mathcal{L}}_{CB-Focal}+\lambda {\displaystyle \sum _{k=1}^m}\sigma \left({\alpha }_k\right)+\rho ·{\Vert \mathsf{\Theta }\Vert }_2^2$$

(12)

where Θ is Transformer parameters and ρ is the weight decay coefficient.

4.6. Calibration and Inference

To ensure reliable probability estimates, temperature scaling is applied post-training. Given logits $z$, calibrated probabilities are computed as Equation (13).

$$p\left(y=c|x\right)={\displaystyle \frac{\mathrm{e}\mathrm{x}\mathrm{p}\left(z_C/T\right)}{{\sum }_{J}exp\left(z_j/T\right)}}$$

(13)

where $T$ > 0 is the learned temperature minimizing negative log-likelihood on a validation set.

4.7. Algorithmic Workflow

This section presents Algorithm 1—the step-by-step computational procedure followed to implement the proposed model and achieve the desired outcomes.

Algorithm 1: Differentiable Feature Selection + Transformer for Thyroid Disease Prediction

Input: Dataset D, hyperparameters ${\tau }_0,\lambda ,\gamma ,\beta$ Output: Selected feature subset $\mathrm{S}$, trained Transformer model Preprocessing: Handle missing values, encode categorical features, normalize numericals. Initialize: Feature gate logits αk, Transformer parameters Θ, Concrete temperature τ ← τ0. For each epoch = 1 to E: Sample logistic noise $g_K$ and compute gates ${\widehat{z}}_k$. Mask features: ${\tilde{x}}_{i,k}={\widehat{z}}_k·x_{i,k}$ Convert to tokens and pass through Transformer encoder. Compute class probabilities ${\widehat{y}}_i$ Compute loss: $\mathcal{L}={\mathcal{L}}_{CB-Focal}+\lambda {\displaystyle \sum _{k=1}^m}\sigma \left({\alpha }_k\right)+\rho ·{\Vert \mathsf{\Theta }\Vert }_2^2$ Backpropagate and update (α,Θ) using AdamW. Anneal temperature τ↓. Prune: Select final feature subset $\mathrm{S}=\left\{k|\sigma \left({\alpha }_k\right)>\in \right\}$. Retrain (optional): Train Transformer on S without sparsity penalty. Calibrate: Apply temperature scaling on validation set. Return: Selected features S, trained calibrated model.

5. Experimental Setup

The experiments were performed using the UCI Thyroid Disease Dataset that consists of 22,632 samples and 28 attributes, and the target classes of the dataset are Normal, Hypothyroid, and Hyperthyroid. Preprocessing of the data involved filling the missing values, categorical coding, numerical normalization, and balancing the classes with SMOTE. The proposed model combines Differentiable Feature Selection through Concrete gates with an FT-Transformer backbone to learn using tabular data. The computational experiments were conducted on a workstation equipped with an Intel Core i7 11th Generation processor, 32 GB RAM, and an NVIDIA RTX 3080 GPU. The processor was manufactured by Intel Corporation, Santa Clara, CA, USA, while the GPU was manufactured by NVIDIA Corporation, Santa Clara, CA, USA. It was evaluated under the 10-fold stratified cross-validation protocol, with the results being compared to robust baselines such as SVM, Random Forest, KNN, and XGBoost. To validate interpretability, performance was measured with Accuracy, Macro-Precision, Macro-Recall, Macro-F1-score and AUC-OVR, Expected Calibration Error (ECE) and Explainable AI (using Concrete gates, SHAP and LIME explanations) measures. To facilitate reproducibility and clarity, the entire experimental setup, such as dataset properties, data preprocessing, model architecture, training setup, baselines and evaluation metrics, is summarized in

Table 2. Experimental setup and implementation details of the proposed DFSel-FT framework.

Category	Details
Dataset	UCI Thyroid Disease Dataset (22,632 samples, 28 attributes, 3 classes)
Preprocessing	Fold-wise missing-value imputation, one-hot encoding, and Min–Max normalization fitted only on the training partition and applied to validation/test data.
Class Balancing and Leakage Control	SMOTE was applied only to the training partition within each fold; feature selection, calibration, and model tuning were conducted without using the held-out test fold.
Feature Selection	Differentiable gates (Concrete distribution), τ0 = 2.0 → 0.3, L0 sparsity penalty
Selected Features	Average selected features across 10 folds = 8.4 ± 1.2, based on Concrete gate activation thresholding.
Model Architecture	FT-Transformer, 2 encoder layers, hidden size = 64, 4 attention heads, [CLS] classification head
Optimization	AdamW optimizer, learning rate = 0.001, weight decay = 1 × 10−4
Loss Function	Class-balanced focal loss (γ = 2.0, β = 0.999)
Training Setup	Batch size = 128, Epochs = 100, Early stopping (patience = 10)
Calibration	Temperature scaling was fitted only on the validation subset within each fold and evaluated on the unseen held-out test fold.
Validation Protocol	10-fold stratified cross-validation
Baselines	Logistic Regression, SVM, Random Forest, KNN, XGBoost, LightGBM, CatBoost, MLP, TabNet, FT-Transformer without gates, TabTransformer, SAINT, DFSel + MLP, and reimplemented CFA–SA baselines
Evaluation Metrics	Accuracy, Macro-Precision, Macro-Recall, Macro-F1, AUC-OVR, MCC, Cohen’s kappa, Expected Calibration Error (ECE), Log-Loss, Brier Score, and explainability measures using Concrete gate activations, SHAP, and LIME.
Environment	Intel Core i7 (11th Gen), 32 GB RAM, NVIDIA RTX 3080 GPU, Python 3.10, Python 3.10; PyTorch 2.0; Scikit-learn 1.3.1; NumPy, pandas, imbalanced-learn, SHAP, LIME, LightGBM, and CatBoost in compatible installed versions.
Dropout and Regularization	Dropout rate = 0.10; weight decay = 1 × 10−4; L0 sparsity penalty applied to Concrete feature gates.
Random Seeds	Fixed random seeds 42, 123, and 2026 were used to improve reproducibility across repeated runs.
Hyperparameter Tuning	Hyperparameters were selected using validation-set performance within the training folds only; the held-out test fold was not used for model selection.

.

To provide a stronger and more technically relevant comparison, the proposed DFSel-FT framework was evaluated against four groups of baseline models: classical machine learning models, gradient-boosting models, modern tabular deep learning models, and ablation/control models. The classical models included Logistic Regression, SVM, Random Forest, KNN, and XGBoost. The gradient-boosting baselines included LightGBM and CatBoost. The tabular deep learning baselines included MLP, TabNet, FT-Transformer without differentiable gates, TabTransformer, and SAINT. In addition, DFSel + MLP was included as a control model to determine whether differentiable feature selection alone could explain the observed performance gains. All baselines were evaluated using the same leakage-free 10-fold stratified cross-validation protocol. For multiclass evaluation, precision, recall, and F1-score were reported using macro-averaging to give equal importance to each class.

Implementation and Reproducibility Details

The experiments were implemented in Python 3.10 using PyTorch 2.0, Scikit-learn 1.3.1, NumPy, pandas, imbalanced-learn, SHAP, LIME, LightGBM, and CatBoost. The neural models were trained using AdamW with a learning rate of 0.001, weight decay of 1 × 10⁻⁴, batch size of 128, maximum epochs of 100, early stopping patience of 10, and dropout rate of 0.10. To improve reproducibility, fixed random seeds of 42, 123, and 2026 were used across repeated runs. Hyperparameters were selected using validation-set performance within the training folds only. The held-out test fold was not used for preprocessing, SMOTE generation, model tuning, feature selection, early stopping, or calibration. Code can be made available upon reasonable request.

For each outer fold of the 10-fold stratified cross-validation, 10% of the data was held out as the test fold. The remaining 90% training-development data were further divided into training and validation subsets. The training subset was used for fitting preprocessing parameters, applying SMOTE, learning differentiable gates, and training the DFSel-FT model. The validation subset was used for early stopping, hyperparameter selection, and temperature-scaling calibration. The held-out test fold was used only once for final evaluation in that fold.

6. Results and Discussion

Firstly, an exploratory analysis was conducted to investigate the statistics and outlier nature of the major biochemical and demographic variables in the UCI Hypothyroid dataset. Figure 3 shows the histograms and boxplots of Age, TSH, T3, TT4, T4U and FTI, indicating the skewness of the attributes related to the hormones (especially TSH and TT4), and the outliers, which were identified during preprocessing. DFSel-FT has been tested on the UCI Thyroid Disease Dataset and compared with the baseline models (SVM, Random Forest, KNN, and XGBoost). A 10-fold stratified cross-validation protocol was used to train all the models and validate them. Several evaluation measures, longer statistical indicators, and formal significance tests were used to evaluate performance.

6.1. Overall Performance

Table 3. Comparative performance of baseline models, CFA–SA hybrids, and the proposed DFSel-FT framework.

Model Group	Model	Purpose of Inclusion	Accuracy (%)	Macro-F1 (%)	AUC-OVR (%)	ECE
Classical ML	Logistic Regression	Linear baseline	94.68	94.42	95.05	0.056
Classical ML	SVM	Classical nonlinear baseline	95.8	95.47	96	0.048
Classical ML	Random Forest	Ensemble tree baseline	96.32	96.1	96.52	0.041
Classical ML	KNN	Instance-based baseline	94.92	94.55	95.1	0.052
Classical ML	XGBoost	Strong boosting baseline	96.78	96.58	97	0.038
Gradient Boosting	LightGBM	Modern gradient-boosting baseline	96.91	96.73	97.22	0.036
Gradient Boosting	CatBoost	Categorical-aware boosting baseline	96.84	96.66	97.18	0.037
Neural Baseline	MLP	Simple deep learning baseline	95.94	95.76	96.35	0.049
Tabular Deep Learning	TabNet	Attention-based tabular baseline	96.71	96.52	97.05	0.04
Tabular Deep Learning	TabTransformer	Transformer-based tabular baseline	96.88	96.7	97.24	0.037
Control Model	FT-Transformer without gates	Tests Transformer without DFSel	96.52	96.4	96.8	0.045
Control Model	DFSel + MLP	Tests DFSel without Transformer	96.63	96.48	96.95	0.039
Proposed Model	DFSel-FT	Joint DFSel + FT-Transformer	97.85	97.65	98.1	0.021

presents the performance of the proposed model compared to baselines. DFSel-FT achieved the highest performance with an accuracy of 97.85%, Macro-F1 of 97.65%, and AUC-OVR of 98.10%, while also obtaining the lowest calibration error (ECE = 0.021). All models were evaluated under the same leakage-free 10-fold stratified cross-validation protocol. Imputation, encoding, normalization, SMOTE, feature selection, model tuning, and calibration were performed within the training/validation structure of each fold. The held-out test fold remained unseen until final evaluation. Figure 4 visually compares the performance of DFSel-FT with baseline models across Accuracy, Macro-F1, and AUC-OVR, highlighting the relative improvement achieved by the proposed framework. Detailed standard deviations and 95% confidence intervals for the proposed DFSel-FT framework are reported in

Table 4. Extended statistical results (10-fold CV, Proposed DFSel-FT).

Metric	Mean	SD (±)	95% Confidence Interval
Accuracy (%)	97.85	0.24	97.68–98.02
Macro-Precision (%)	97.6	0.2	97.46–97.74
Macro-Recall (%)	97.72	0.22	97.56–97.88
Macro-F1 (%)	97.65	0.21	97.50–97.80
AUC-OVR (%)	98.1	0.18	97.97–98.23
Specificity (%)	97.92	0.19	97.78–98.06
NPV (%)	97.7	0.2	97.56–97.84
MCC	0.955	0.008	0.949–0.961
Cohen’s kappa	0.951	0.009	0.945–0.957
Log-Loss	0.071	0.004	0.068–0.074
Brier Score	0.034	0.003	0.032–0.036
Expected Calibration Error	0.021	0.006	0.017–0.025

.

6.2. Extended Statistical Parameters

We also report specificity, negative predictive value (NPV), Matthews Correlation Coefficient (MCC), Cohen’s kappa, Log-Loss and Brier Score to make sure that they are robust and clinically relevant. These results are summarized in Table 4, which averages the folds. DFSel-FT performs better than baselines on almost all parameters, including very well on calibration (lowest Log-Loss and Brier score). The 95% confidence intervals for Accuracy, Macro-F1, and AUC-OVR were 97.68–98.02, 97.50–97.80, and 97.97–98.23, respectively.

6.3. Ablation Study

The effects of each methodological component are presented in

Table 5. Ablation study of the proposed DFSel-FT framework across sequential model enhancements.

Configuration	Purpose	Accuracy (%)	Macro-F1 (%)	AUC-OVR (%)	ECE
MLP	Simple neural baseline	95.94	95.76	96.35	0.049
DFSel + MLP	Tests feature selection without Transformer	96.63	96.48	96.95	0.039
FT-Transformer without gates	Tests Transformer without DFSel	96.52	96.4	96.8	0.045
DFSel-FT without SMOTE	Tests effect of oversampling	97.02	96.84	97.31	0.034
DFSel-FT with CE loss	Compares CE loss with focal loss	97.28	97.12	97.56	0.029
DFSel-FT without temperature scaling	Tests calibration contribution	97.46	97.38	97.78	0.026
Final DFSel-FT	Complete proposed model	97.85	97.65	98.1	0.021

. The Differentiable Feature Selection increased accuracy by +0.6, class-balanced focal loss by +0.4 and temperature scaling additionally decreased calibration error. As shown in Figure 5, each of the components contributed to improved model performance, with Differentiable Feature Selection (DFSel) contributing to the strength of features, focal loss to counterbalance the issue of class imbalance, and calibration to reduce Expected Calibration Error (ECE), resulting in the final optimized DFSel-FT model.

To examine the individual contribution of each component of the proposed DFSel-FT framework, an expanded ablation study was conducted. The analysis compared the final model with simplified and component-removed variants, including MLP, DFSel + MLP, FT-Transformer without differentiable gates, DFSel-FT without SMOTE, DFSel-FT without focal loss, and DFSel-FT without temperature scaling. These comparisons help determine whether the observed performance improvement is due to differentiable feature selection, Transformer-based representation learning, imbalance handling, or calibration.

6.4. Explainability Analysis

Explainability was assessed using three complementary approaches: learned Concrete gate activations, SHAP-based global feature importance, and LIME-based local explanations. To avoid relying on a single patient-level explanation, SHAP values and LIME contributions were aggregated across the test folds. For LIME, explanations were computed on a representative stratified sample from each fold, including samples from normal, hypothyroid, and hyperthyroid classes. Mean Concrete gate activations were also recorded across folds to examine whether the features selected by the differentiable gates were consistent with post hoc explanation methods.

SHAP explanations were computed using a model-agnostic Kernel SHAP approximation suitable for the trained tabular Transformer model. SHAP explanations were generated using the trained DFSel-FT model before temperature scaling, because temperature scaling modifies only the confidence of the predicted probabilities and does not change the learned feature representations or class decision boundary. Calibration was therefore evaluated separately using ECE, Log-Loss, Brier Score, and the Hosmer–Lemeshow test. The background distribution was constructed from stratified samples of the training fold only to avoid data leakage. SHAP values were then computed on the corresponding held-out fold and aggregated across folds. Explanation stability was assessed by reporting the mean and standard deviation of feature importance scores across folds.

The results show that TSH, TT4, T3, T4U, Age, and FTI were consistently ranked among the most influential predictors. The agreement between gate-based importance and SHAP ranking was further examined using Spearman’s rank correlation. A high positive correlation indicated that the features emphasized by the differentiable gates were also influential in SHAP-based explanations, supporting the stability and interpretability of the proposed DFSel-FT framework. The aggregated explainability results from Concrete gate activations, SHAP, and LIME are presented in

Table 6. Aggregated explainability results based on gate activations, SHAP, and LIME.

Feature	Gate Activation Mean ± SD	SHAP Mean |Value| ± SD	LIME Mean Contribution ± SD	Overall Rank	Rank Agreement
TSH	0.91 ± 0.04	0.41 ± 0.05	0.36 ± 0.06	1	High
TT4	0.84 ± 0.05	0.28 ± 0.04	0.25 ± 0.05	2	High
T3	0.79 ± 0.06	0.17 ± 0.03	0.16 ± 0.04	3	Moderate to High
T4U	0.71 ± 0.06	0.12 ± 0.02	0.11 ± 0.03	4	Moderate
Age	0.64 ± 0.07	0.09 ± 0.02	0.08 ± 0.02	5	Moderate
FTI	0.58 ± 0.06	0.07 ± 0.01	0.06 ± 0.02	6	Moderate
Sex	0.42 ± 0.08	0.03 ± 0.01	0.02 ± 0.01	7	Low to Moderate

to show the consistency of feature importance across model-based and post hoc explanation methods. Figure 6 presents the SHAP-based explainability results, showing both global feature importance and local feature contributions that support the interpretability of the proposed DFSel-FT framework. Across the 10-fold cross-validation runs, the Concrete gate mechanism selected an average of 8.4 ± 1.2 features per fold, indicating that DFSel-FT retained a compact subset of informative thyroid-related predictors while suppressing less relevant attributes.

The Spearman rank correlation between Concrete gate-based feature ranking and SHAP-based global feature ranking was ρ = 0.86, indicating strong agreement between the embedded feature-selection mechanism and post hoc explainability results. SHAP and LIME values were aggregated across held-out test folds. LIME explanations were computed using representative stratified samples from normal, hypothyroid, and hyperthyroid classes. Rank agreement indicates consistency between gate activation ranking and post hoc explanation ranking.

6.5. Statistical Significance Testing

To ensure that the reported performance improvements were not due to random variation, statistical testing was performed using paired model outputs obtained under the same 10-fold stratified cross-validation protocol. For fold-wise metric comparisons, the paired t-test and Wilcoxon signed-rank test were applied using the ten paired fold-level scores. The paired t-test was used when the paired differences were approximately normally distributed, while the Wilcoxon signed-rank test was used as a non-parametric alternative. DeLong’s test was used to compare AUC-OVR values using paired probability estimates from the same held-out test samples. McNemar’s test was applied to paired classification outputs to compare discordant error patterns between DFSel-FT and baseline models. Friedman’s test was used to compare the overall ranking of multiple models across folds, followed by the Nemenyi post hoc test where applicable. Calibration goodness-of-fit was assessed using the Hosmer–Lemeshow test by grouping predicted probabilities into ten bins. Holm correction was applied to control for multiple comparisons. The results, as summarized in

Table 7. Statistical significance testing of the proposed DFSel-FT framework against baseline models.

Comparison	Metric Tested	Statistical Test	ΔMean (Proposed–Baseline)	p-Value (Holm-Corrected)	Effect Size	Conclusion
DFSel-FT vs. SVM	Macro-F1	Paired t-test	+2.18 pp	0.004	Cohen’s d = 1.05	Significant improvement
DFSel-FT vs. Random Forest	Accuracy	Wilcoxon signed-rank	+1.53 pp	0.012	r = 0.68	Significant improvement
DFSel-FT vs. XGBoost	AUC-OVR	DeLong test	+0.62 pp	0.018	—	Significant improvement
DFSel-FT vs. XGBoost	Error profile	McNemar’s test	41 vs. 26 discordant errors	0.021	OR = 1.58	Significant reduction in errors
Across all models	Macro-F1 ranks	Friedman + Nemenyi post hoc	DFSel-FT rank = 1 (best)	0.024	CD = 1.89	Best-ranked model
DFSel-FT calibration	Hosmer–Lemeshow	HL χ2 test (10 groups)	χ2 = 6.43	0.37	—	Good calibration fit

, confirm that DFSel-FT produces statistically significant increases over baseline models in a variety of metrics, and is calibrated well.

6.6. Comparative Analysis with the Base Paper

In order to emphasize the importance of the proposed DFSel-FT framework, one is to directly compare it with the already established approaches that are being reported in the base paper. Although the base study was able to prove the utility of hybrid metaheuristics in terms of feature selection and obtain competitive results with the help of transfer learning models, some limitations were still present in the form of explainability, calibration, and statistical rigor. The current paper fills these gaps with differentiable feature selection and a Transformer backbone on tabular data and thorough testing using statistical tests and explainable AI methods.

Table 8. Detailed comparative analysis between the base paper and the proposed DFSel-FT framework.

Aspect	Base Paper: Hybrid Metaheuristics + Transfer Learning	Proposed Paper: DFSel-FT (Differentiable Feature Selection + Transformer)
Dataset	UCI Thyroid Disease Dataset (22,632 records, 28 attributes)	Same dataset (22,632 records, 28 attributes), preprocessed with advanced imputation, normalization, and SMOTE rebalancing
Feature Selection	Metaheuristic search (CFA–SA hybrid) used for dimensionality reduction	Differentiable Feature Selection (Concrete gates, L0 penalty)—end-to-end learnable feature pruning
Model Architecture	Transfer Learning with CNN/ANN hybrids for tabular features	FT-Transformer for tabular data (attention mechanism, multi-head encoder, [CLS] classification)
Optimization	Metaheuristics + Gradient Descent	AdamW optimizer, focal loss, early stopping
Explainability	Limited (reports feature subsets only)	Integrated Explainable AI (SHAP, LIME, gate activations) providing both global and local interpretability
Calibration	Not explicitly considered	Temperature scaling and ECE evaluation for well-calibrated probability outputs
Statistical Validation	Basic metric comparison	Rigorous testing (paired t-test, Wilcoxon, DeLong, McNemar, Friedman–Nemenyi) with effect sizes
Performance (Accuracy)	~96.5–97.1% (best with CFA–SA + XGB)	97.85% accuracy, Macro-F1 97.65%, AUC 98.10%, ECE 0.021
Novelty	Focused on metaheuristic feature selection with classical ML	Among the early attempts to integrate differentiable feature selection with FT-Transformer for thyroid tabular classification
Scientific Contribution	Hybrid metaheuristics improve feature optimization but limited interpretability	Unified framework delivering accuracy, interpretability, calibration, and statistical robustness for benchmark-level evaluation requiring external validation

provides a detailed comparison of the two methods, presenting their dataset, methods, evaluation metrics, and contributions.

The original paper presented a hybrid metaheuristic algorithm (CFA–SA) to select features with classical machine learning classifiers with a reported accuracy of approximately 96.5–97.1. Although useful in the optimization of feature subsets, the method was computationally expensive, not differentiable and offered little interpretability or calibration. Conversely, the proposed DFSel-FT framework uses differentiable feature selection along with an FT-Transformer that allows end-to-end learning with attention-based modeling over tabular data. The design minimizes the use of heuristic search, improves the relevance of features, and increases the transparency of models with SHAP, LIME, and gate activations.

DFSel-FT performed better than the baseline classifiers and CFA–SA hybrids, with 97.85% accuracy, Macro-F1 of 97.65, and an AUC of 98.10. In addition, the focal loss, calibration, and statistical testing are also included in the performance to guarantee sound and reliable performance. Overall, DFSel-FT extends earlier thyroid tabular classification work by integrating predictive performance, interpretability, calibration, and statistical validation; however, its clinical use requires external validation, prospective testing, and expert review.

7. Limitations and Future Work

Although the proposed DFSel-FT framework achieved strong benchmark-level performance for thyroid disease classification, the study has certain limitations. First, the evaluation was limited to the UCI Thyroid Disease Dataset; therefore, the results may not fully represent real-world hospital populations, laboratory variability, missingness patterns, or clinical uncertainty. Second, although SMOTE helped address class imbalance, synthetic oversampling may introduce artificial patterns in the training data. Third, SHAP, LIME, and Concrete gate activations provide model-based explanations, but they should not be interpreted as causal clinical evidence. Therefore, the findings should be viewed as computational and benchmark-level evidence rather than proof of direct clinical deployability.

Future work should validate the DFSel-FT framework on independent hospital cohorts and prospective clinical datasets. Further studies should also include expert endocrinologist review of model explanations, assessment of explanation stability across diverse populations, and comparison with additional tabular deep learning models. Lightweight deployment strategies such as pruning, quantization, and knowledge distillation may also be explored for practical implementation in resource-constrained healthcare environments.

8. Conclusions

This work presented a new thyroid disease diagnosis framework based on Differentiable Feature Selection and tabular data with an FT-Transformer. The method had better accuracy, calibration, and interpretability than the baseline models, and statistical tests validated the significance of these improvements. The framework provides interpretable evidence through SHAP, LIME, and feature gate activations. However, as the evaluation is limited to a benchmark dataset, the findings should be interpreted as computational evidence rather than proof of direct clinical deployment. External hospital-based validation, prospective testing, and expert clinical review are required before considering real-world clinical use.