A New Alternating Suboptimal Dynamic Programming Algorithm with Applications for Feature Selection

Abstract

Feature selection is predominantly used in machine learning tasks, such as classification, regression, and clustering. It selects a subset of features (relevant attributes of data points) from a larger set that contributes as optimally as possible to the informativeness of the model. There are exponentially many subsets of a given set, and thus, the exhaustive search approach is only practical for problems with at most a few dozen features. In the past, there have been attempts to reduce the search space using dynamic programming. However, models that consider similarity in pairs of features alongside the quality of individual features do not provide the required optimal substructure. As a result, algorithms, which we will call suboptimal dynamic programming algorithms, find a solution that may deviate significantly from the optimal one. In this paper, we propose an iterative dynamic programming algorithm, which invertsthe order of feature processing in each iteration. Such an alternating approach allows for improving the optimization function by using the score from the previous iteration to estimate the contribution of unprocessed features. The iterative process is proven to converge and terminates when the solution does not change in three successive iterations or when the number of iterations reaches the threshold. Results in more than 95% of tests align with those of the exhaustive search approach, being competitive and often superior to the reference greedy approach. Validation was carried out by comparing the scores of output feature subsets and examining the accuracy of different classifiers learned on these features across nine real-world applications, considering different scenarios with various numbers of features and samples. In the context of feature selection, the proposed algorithm can be characterized as a robust filter method that can improve machine learning models regardless of dataset size. However, we expect that the idea of alternating suboptimal optimization will soon be generalized to tasks beyond feature selection.

1. Introduction

Nowadays, in the era of the Internet of Things, social media platforms, Earth observation, crowdsourcing, medical imaging equipment, various biomedical signals measurement devices, wearable sensors, digital twins, etc., we are flooded with vast amounts of data. The measurable characteristics used as attributes or input variables to describe an object of interest are called features, while individual data points (objects of interest) represent feature vectors. Each feature thus corresponds to a dimension in the vector. Vast amounts of data allow the creation of a large repertoire of features, but usually, not all features are relevant for further processing objects of interest. They often slow down the model, direct it towards a wrong solution, or even make reaching a solution infeasible. In order to address these challenges, feature selection approaches were introduced that select a subset of the most informative features while discarding irrelevant or redundant ones [1]. Feature selection plays a vital role in model construction in statistical analysis, dimensionality reduction, signal processing, pattern recognition, data visualization, and, particularly, in various machine learning tasks, such as classification, regression, and clustering. Its aim is to improve the model’s performance, including its accuracy, generalizability, and interpretability, and reduce overfitting and computational cost [2].

Feature selection methods can be grouped into three categories [1]. Filter methods evaluate candidate subsets with independent criteria that exploit essential characteristics of the training data. They are fast, but the solution may deviate significantly from the optimal one. A wrapper approach uses a learning algorithm for subset evaluation, such as a classifier or regressor. Its performance is usually better but also much slower than the filter approach. Embedded methods interact with a learning algorithm but at a lower computational cost than the wrapperapproach. They use independent criteria to identify optimal subsets for a known cardinality. The learning algorithm is then used to select the final optimal subset across different cardinalities [3].

Regardless of the approach chosen, feature selection can be viewed as an optimization problem as it searches for the best-evaluated feature subset [4]. Different search strategies can be used, including sequential search (greedy approach), exponential search (exhaustive search, beam search, or branch and bound), and random search [3]. Conversely, dynamic programming (DP) is not as commonly applied to feature selection as other methods. This popular optimization approach breaks a problem into smaller subproblems and uses their solutions to construct the solution to the larger problem. An optimal solution can be found if the problem exhibits optimal substructure. This means that an optimal solution to the problem contains optimal solutions to subproblems [5,6]. However, DP is usually computationally demanding, so for reasons of feasibility, acceptable speed, and availability to handle problems with higher dimensionality, it is also required that the number of subproblems is not too high and that the subproblems overlap, suggesting that it makes sense to record their solutions in a table and reuse them [6].

In this paper, we highlight the possibilities of using DP in feature selection, analyze the difficulties of existing (rare) approaches, and propose alternative solutions. An evaluation criterion based on feature quality, correlation, and/or statistics does not generally provide an optimal substructure since, e.g., the union of two optimal subsets is not necessarily optimal due to possible high correlations between pairs of features, one from each subset. It is possible to achieve an optimal solution for specific problems by adapting the evaluation criterion, but this spoils generality (e.g., wrappers or embedded selection methods are tied to specific machine learning models and prone to overfitting [2]), which is among our primary goals. We thus focused on finding the best possible suboptimal solution. We studied approximate (ADP) [7] and iterative [8] dynamic programming (IDP) methods and developed a solution that we called alternating suboptimal dynamic programming (ASDP). It inverts the order of feature processing in each iteration and improves the optimization criterion by using the score from the previous iteration to estimate the contribution of unprocessed features. Its contributions are as follows:

• A better or at least the same evaluation score of the final solution set compared to the score after a single iteration. Furthermore, the solution found in each iteration is never worse than the one found in the previous iteration.
• Optimal solution according to the evaluation score found in more than 95% of cases.
• Polynomial worst-case time complexity ($O\left(n^4\right)$) allows significantly larger input feature sets to be considered compared to the exhaustive search approach.
• Comparable and, in some cases, better classification accuracy on the basis of the feature set selected by the new method than when using our previous graph-based greedy feature selection method. In this respect, we have already demonstrated the competitiveness of the latter in our previous work [9] compared to state-of-the-art classification approaches and applied feature selection methods.

The rest of the paper is structured as follows. In Section 2, we survey existing solutions in feature evaluation and selection, the use of DP in feature selection, and suboptimal DP algorithms. In the most research-intensive Section 3, we first summarise our preliminary filter method for feature selection based on graph cuts, which can be used alone or as a preprocessing for the new alternating suboptimal DP method presented afterward. In Section 4, we show and analyze the results, and, finally, in Section 5, we discuss the work carried out, its strengths, and some weaknesses that pose challenges for future research.

2. Related Works

As the topic presented here combines several challenges, the state-of-the-art review must address several areas. First, in Section 2.1, we address feature evaluation, i.e., procedures and metrics to assess the contribution of individual features and/or a feature subset to a machine learning model. Feature evaluation is the basis for feature selection, which we review in Section 2.2. The goal is to optimally select a subset of the input features that solve a given machine-learning task. We wanted to approach the problem using dynamic programming, so in Section 2.3, we review the use of this software design strategy in feature selection. However, such methods are rare, time-demanding, and practically always offer partial solutions only. Consequently, the solution proposed in this paper is suboptimal; thus, Section 2.4 briefly reviews the use of suboptimal DP for various problems, including feature selection.

2.1. Feature Evaluation

Feature evaluation is a critical step in the feature selection process. It includes assessing the contribution of input features to a machine learning model performance [2]. Features that contribute the most information to the predictive model can improve the model’s performance, reduce overfitting, and accelerate the learning process [2,10,11]. For classification purposes, feature evaluation can be achieved directly by evaluating the classification models built for each feature [2]. The choice of classifier strongly influences the evaluation results, while its learning is often time-consuming. The latter is particularly evident in cases of a large number of features [2,10,11]. Similar drawbacks are also noted for regression purposes, as feature evaluation can be achieved using computationally demanding regression approaches built for each feature.

To avoid using a computationally demanding classifier, techniques for analyzing the discriminatory power of features are introduced. Early approaches focused on the ratio between the distances of samples of different classes and samples of the same class [12]. Examples of these include the Fisher criterion [13], the maximum margin criterion [14], and the Laplacian score [15]. In the case of regression tasks, techniques are based on calculating the correlation coefficients (e.g., Pearson’s or Spearman’s) between the feature’s values and the continuous target variable [2,10]. However, the mentioned techniques for classification and regression tasks cover only linear interdependencies between feature values and the target variable.

Approaches that capture nonlinear dependencies are based on the information contribution. For classification purposes, this technique evaluates features according to the ratio between the classes’ entropy and the feature values’ conditional entropy [16,17,18]. Regarding accuracy, similar results can be obtained using the computationally more efficient Gini impurity [19,20,21]. In the case of multi-class classification, the Gini impurity is biased towards majority classes and prone to overfitting [21]. Conversely, techniques based on mutual information capture nonlinear relationships between features and the target variable successfully, utilizing both classifications [22] and regression [23], respectively. However, such metrics favor features with many different values, which can lead to overfitting.

2.2. Feature Selection

Overfitting negatively affects the power of machine learning methods and cripples predictive accuracy. Irrelevant features lower the predictive power of the model. Feature selection methods can overcome both limitations [24]. We divide these methods into three groups:

• Filters;
• Wrappers;
• Embedded methods.

Filtering is usually performed using a threshold value. Although such methods are computationally very efficient, their classification power largely depends on the feature evaluation techniques [2,10,25]. The latter often consider only pairwise dependencies between feature values and the target variable, ignoring correlations between features [2,12]. As a result, the prediction efficiency is thus limited.

In [26,27], a method calculates the efficiency of separation between different classes in the local neighborhood of selected samples to evaluate the feature. This enables low execution times because it does not use all of the samples contained in the dataset. However, calculations are usually inaccurate due to the limited number of considered samples. The method also does not consider the correlation between features. In [28], the authors proposed an approach that selects features highly correlated with the class labels and in low correlation with each other. A similar method is proposed in [29] but for regression purposes. The only difference is that it selects features highly correlated with the target variable. However, neither technique considers the interaction between features and only considers the linear interdependence between feature values and target variables.

In [30], the authors propose a two-stage feature selection method in which the evaluation is based on the calculation of mutual information while at the same time considering the correlation between pairs of features. In [23], the adequacy of the mutual information for regression is considered. However, in the case of a small number of training samples, inaccurate estimates of mutual information may appear, and the method is biased towards features with a large number of different values due to the use of this metric. In [31], a feature selection framework for large datasets was proposed based on a cascade of methods capable of detecting nonlinear relationships between two features and designed to achieve a balance between accuracy and speed.

Conversely, wrapper methods select a subset of features that maximizes the performance of a given classifier or regressor [2]. Wrappers are considered a multi-criteria optimization problem that maximizes machine-learning-method performance while minimizing the number of selected features. This can be addressed with several optimization techniques [2,32], such as sequential selection algorithms or nature-inspired algorithms, such as the evolutionary and genetic algorithm [33,34], particle swarm optimization [33,35], and the bees algorithm [36].

Early wrappers were based on sequential selection [37]. This starts with an empty set, adding features, and evaluating the prediction performance. The feature that gives the best results is permanently included in the set. The selection continues by adding features one by one again and keeping those that contribute the most to improving the prediction performance. The algorithm terminates when a predetermined threshold of acceptable results is reached or when a sufficient number of features are selected. In [38], the authors propose an inverse procedure where features are removed from the input set. The main limitation of these algorithms is that they do not consider the correlation between features. This limitation is eliminated in the adaptive version of the algorithm [39]. However, such a search for an optimal set with a more significant number of features soon grows into an exponentially time-consuming process [12,37].

Therefore, methods that find a suboptimal solution were proposed [33,34]. Examples of the latter are algorithms based on nature-inspired concepts. Similar to sequential feature selection methods, the evaluation function of evolutionary algorithms represents the performance of a model, while the features represent the population. The best-performing subsets are combined to achieve the desired result [34,35]. The biggest problem with these methods is their computational complexity, as the process involves the model evaluating features for each specific subset over a large number of iterations to obtain useful results [2,40].

An alternative to wrapper methods is embedded methods, which have lower execution times [2,41]. These approaches perform the selection of a subset of features in interaction with the model in the learning phase of the model and are thus tied to the selected machine learning algorithm [2]. Decision trees achieve feature selection based on mutual information evaluation, while support vector methods use LASSO (Least Absolute Shrinkage and Selection operator) regression analysis with Ll [41] or ridge regression with L2-regularization [42] to rank features during learning. This significantly reduces the computational complexity of both methods, as it allows multiple repetitions of the machine learning algorithm learning process to be avoided. At the same time, support vector methods are inexplicable and sensitive to many input parameters. Similar to support vector methods, the selection of features can also be achieved with neural networks, whereby learning the network, we set the weights for individual features according to their suitability [2].

2.3. Dynamic-Programming-Based Feature Selection

DP can be employed since feature selection can be formulated as an optimization problem. In this context, DP solves the exhaustive search of a feature subset by breaking it down into simpler searches while storing the suboptimal subsets of features to avoid redundant calculations.

As early as 1998, Nelson and Levy [43] presented a method where optimality is defined in terms of a particular measure, the Fisher return function, providing the features were uncorrelated. In [44], a method for selecting the best subset of features for classification purposes is presented. It uses DP for divergence analysis of the feature distributions regarding given classes, selecting the subset of the most informative features. However, this method does not consider the interaction between features and thus can choose redundant ones. The identical drawback can be noted in [45]. The approach is similar to the sequential forward feature selection, starting with an empty set of features and adding the best-performing one in each iteration. The main difference is that it can remove any previously selected features if it improves the performance of a set. In [46], the authors presented a method that uses rough sets theory and DP in order to remove redundant features from the input set while maintaining high classification performance. However, this approach is computationally intensive, especially for large datasets with a high number of features.

2.4. Suboptimal Dynamic Programming

In the literature, there are several partially overlapping concepts addressing suboptimal DP.

• Approximate Dynamic Programming (ADP) is a sophisticated variant of traditional DP, representing a compromise between the method’s accuracy and computational feasibility. It aims to find near-optimal solutions to DP problems, achieved by approximating the value or policy functions using function approximation techniques such as neural networks, linear approximators, or interpolation methods [47,48]. A policy represents a strategy or set of rules that dictate the decision-making process, while the value corresponds to the expected return or benefit from following a particular policy. ADP iteratively refines both, converging towards an optimal or near-optimal solution [48]. The concept has proven itself, particularly in solving large-scale discrete-time multistage stochastic control processes, i.e., complex Markov Decision Processes (MDPs), and found applications in different fields, such as inventory control systems, financial optimization problems, robot path planning, information theory, and decision-making processes in learning agents, i.e., reinforcement learning (RL) [49,50]. Refs [7,51] consider feature selection in RL and MDPs, while [52] addresses feature discovery, also known as feature construction, in the context of ADP and RL. Note that adaptive dynamic programming with the same acronym ADP is sometimes found in the literature for practically the same concept [53].
• Iterative dynamic programming (IDP) involves solving a problem by iteratively refining an initial solution through DP techniques. The definition can be interpreted in different ways. Most often, IDP is used to solve real-valued optimization problems in a manner that reduces the state and control quantization to an arbitrarily small amount by first searching over a relatively coarse but large set of system inputs and states using DP, and then successively generating a denser and narrower search range centered about the previous result. Luus [54] defined this principle as reducing dimensionality by iteratively varying grid resolutions, while Lock and McKelvey [55] applied it on different time scales. In [8], IDP was used to optimize queries in database systems. IDP was also referred to as a counterweight to recursive DP, while value and policy iteration represents an intersection point of IDP with ADP [48]. Note that IDP can be either optimal or suboptimal.
• Relaxed dynamic programming reduces the complexity by relaxing the demand for optimality. The distance from optimality is kept within prespecified error bounds, and the size of the bounds determines the computational complexity [56]. The bounds are chosen by the user, who can then effectively trade-off between solution time and accuracy [57]. By controlling the error in the processes of relaxed value iteration and approximate policy iteration, the relaxed DP concept is closely related to ADP and IDP [58].

3. Materials and Methods

In this section, we present a new method for feature selection based on suboptimal DP. There are exponentially many subsets of a given feature set, all of which are candidates for the feature selection solution, so the exhaustive search approach is only practically applicable to problems with a few dozen features. Our method processes, e.g., 200 features in 5 s, but for larger input sets, it makes sense to preprocess the features with some faster filtering. We use our efficient and reliable graph-cut-based feature selection [9], summarised in Section 3.1. In Section 3.2, we discuss the idea of using DP and the encountered difficulties and introduce an iterative suboptimal alternating solution, where the order of feature processing is inverted in each iteration. We conclude the chapter with proof of convergence and a theoretical analysis of time and space complexity.

3.1. Graph-Cut-Based Feature Selection

While wrapper feature selection methods, like the sequential search, nature-inspired algorithms, or binary teaching–learning-based approaches bypass the need for explicit feature evaluation to yield results that are close to optimal, their effectiveness is tied to the specific classification model being used. Additionally, these methods are highly computationally intensive, which can limit their applicability. Similarly, embedded methods incorporate an iterative cycle of evaluating and selecting features as a part of the model training process, which can also demand significant computational resources. Furthermore, the performance of embedded methods is likewise influenced by the choice of the classification model. As an alternative to the discussed wrapper and embedded feature selection techniques, as well as those filter methods that are unable to deal with correlated features, in this section, we presented a graph-cut-based feature selection strategy outlined in our work [9] that enables the selection of a subset of high-quality dissimilar features while providing superior results. Depending on the defined feature estimation measurement, it can be used for both classification and regression purposes. Graph vertices represent features with associated weights that define their quality (as proposed in [9]), while graph edge weights define similarities between them. The method relies on two input parameters, $T_{\Delta }$ and $T_p$, used for graph definition. The former defines the necessary level of feature quality (i.e., maximal allowed class overlap) to be included in the output feature space, and the latter determines the minimal level of dissimilarity between them.

Let $FS$ denote an input feature space $FS=⟨f_i⟩$. A feature $f_i$, referred to by an index $i\in [1,n]$, is given as a mapping function $f_i:\mathbb{Z}\to \mathbb{R}$. An index $m\in \left[1,M\right]$ refers to a sample, i.e., a feature vector defined as ${\overrightarrow{x}}_m=⟨f_{i,m}⟩$. An undirected graph used for feature selection is defined as $G=(F,E)$, where a set of vertices F is defined as $F=\left\{f_i\in FS;\Delta \left(f_i\right)\le T_{\Delta }\right\}$, while an unordered set of edges $E=\left\{e_{i,j};P\left(e_{i,j}\right)\ge T_p\right\}$ is given by $e_{i,j}=(f_i,f_j)$ for all $f_i,f_j\in F$, such that $i\ne j$. A vertex-weighting function is given by $\Delta \left(f_i\right)$, as defined in [9], and the edge-weighting function is given by the absolute Pearson correlation coefficient $P:E\to [0,1]$, formally described by Equation (1).

$$P\left(e_{i,j}\right)=\left|\frac{{\sum }_{m=1}^M(f_{i,m}-{\mu }_i)(f_{j,m}-{\mu }_j)}{{\sigma }_i{\sigma }_j}\right|,$$

(1)

where ${\mu }_i$ denotes mean, while standard deviation ${\sigma }_i$ of feature values is defined as ${\sigma }_i=\sqrt{\frac{1}{M}{\sum }_{m=1}^M\left(f_{i,m}-{\mu }_i\right)}$. Both functions, $\Delta$ and P, are designed such that lower values (closer to 0) are more favorable for selection than higher values (closer to 1).

According to the theoretical framework introduced in [9], we use the following definitions of elementary properties:

• Vertices $f_i\in F$ and $f_j\in F$ are adjacent in a graph G if there exists an edge $e_{i,j}\in E$.
• A path from $f_{i_0}$ to $f_{i_N}$ is an ordered sequence of vertices ${\prod }_{i_0,i_N}=⟨f_{i_0},f_{i_1},\dots ,f_N⟩$, such that $f_{i_j}$ and $f_{i_{j+1}}$ are adjacent for all $j\in [0,N-1]$.
• A graph G is connected if $\forall f_i,f_j\in F$ there exists a path ${\prod }_{i,j}$.
• A graph $G^{\prime }=(F^{\prime },E^{\prime })$ is subgraph of G if $F^{\prime }\subseteq F$ and $E^{\prime }\subseteq E$.
• A neighbourhood $Z\left(f_i\right)$ of a vertex $f_i$ in graph G is the subset of vertices F, defined by all the adjacent vertices of $f_{ik}$, namely, $Z\left(f_i\right)=\left\{f_j\right\};f_j\in F;\exists e_{i,j}$, where $i\ne j$.

We say that a set of vertices $CUT\left(F\right)\subseteq F$ is a vertex-cut if its removal separates graph G into at least two non-empty and pairwise disconnected connected components. Obviously, $Z\left(f_i\right)$ is a graph-cut, as it separates a singleton $\left\{f_i\right\}$ (i.e., an individual vertex) from the rest of the graph, thus creating a subgraph $G^{\prime }=(F^{\prime },E^{\prime })$, whose vertex- and edge-sets are given formally by Equation (2).

$$\begin{array}{c}{F}^{\prime }=F\setminus (Z\left(f_i\right)\cup \left\{f_i\right\}),\\ E^{\prime }=\{e_{h,l}\in E;f_h,f_l\in F^{\prime }\mathrm{and}h\ne l\}.\end{array}$$

(2)

An example of vertex-cut feature selection is presented in Figure 1. Figure 1a shows an undirected graph $G=(F,E)$, constructed over a set of features $FS=\{f_1,f_2,\cdots ,f_9\}$, with thresholds $T_{\Delta }=0.6$ and $T_p=0.6$ applied on the associated vertex- and edge-weighting functions $\Delta$ and P, accordingly. To ensure the preservation of the overall informativeness of selected features, a feature of the highest quality $\widehat{f_r}={arg\; min}_{f_m\in G}\Delta \left(f_m\right)$ is selected first by a vertex-cut of its neighborhood $Z\left(\widehat{f_r}\right)$. The selected feature $f_6$ is colored green. All of highly correlated adjacent features $Z\left(\widehat{f_6}\right)=\{f_2,f_3,f_8\}$ are marked red and removed from G. This results in $G^{\prime }$, as defined by Equation (2), and a disconnected singleton $\left\{f_6\right\}$ (see Figure 1b). The same process is then repeated on $G^{\prime }$, separating the feature of the highest quality, namely $f_1$, from the remaining graph $G^{\prime \prime }$ by removal of $Z\left(f_1\right)=\{f_4,f_7\}$. The final cut is performed on the graph $G^{\prime \prime }$ separating $f_5$ (in green) from the remaining (empty) graph $G^{\prime \prime }$ by removal of $Z\left(f_5\right)=\left\{f_9\right\}$ (in red), as shown in Figure 1c. Thus, the output subset of high-quality dissimilar features, namely $\{f_1,f_5,f_6\}$, is obtained, as shown in Figure 1d.

Figure 1. Vertex-cut-based feature selection: (a) graph G where the feature of the highest quality (coloured green) is selected and its neighbourhood (red) is removed, (b) repeating the same procedure on subgraph $G^{\prime }$, and (c) subgraph $G^{\prime \prime }$. (d) The output result $\{f_1,f_5,f_6\}$ (in green) is obtained.

3.2. New Suboptimal Dynamic Programming Algorithm

The new method combines the advantages of iterative and approximate dynamic programming. It does not seek a global optimum but instead adopts a suboptimal (approximate) solution, which it iteratively improves. It is based on a graph like the graph-cut-based filtering from Section 3.1. We thus use the same notation, but we will extend it throughout this subsection with additional algorithm parameters and graph vertex attributes. The graph is undirected, i.e., $P\left(e_{i,j}\right)=P\left(e_{j,i}\right)$. The input is the feature set $FS=⟨f_i⟩$, $0<i\le n$, which is processed in index order, i.e., from $f_1$ to $f_n$, so we will sometimes also speak of a sequence of features. At both ends of this sequence, the guard vertices $f_0$ and $f_{n+1}$ are added, which do not change during the execution of the algorithm, but they simplify the implementation. There is no edge between the two guards, while the guard vertices and the edges between a guard and any other vertex are given weights 0. We stress this in the form of an Equation (Equation (3)).

$$\begin{array}{c}\Delta \left(f_0\right)=0,\\ \Delta \left(f_{n+1}\right)=0,\\ P\left(e_{0,n+1}\right)=\infty ,\\ P\left(e_{0,i}\right)=0,0<i\le n,\\ P\left(e_{i,n+1}\right)=0,0<i\le n.\end{array}$$

(3)

Each graph vertex $f_i$ contains, in addition to the weight $\Delta \left(f_i\right)$, a set $S_i$ that stores the “optimal” subset (feature selection result) of the vertices already processed, and the score $s_i$ of this subset, which is obtained by the evaluation criterion. Their initialization is described by Equation (4) and is important for the convergence proof in Section 3.3. The evaluation criterion described in Equation (5) seeks a minimum for all vertices, except the guards, i.e., $0<i\le n$.

$$\begin{array}{c}{S}_i=\varnothing ,0\le i\le n+1,\\ s_i=0,0\le i\le n+1.\end{array}$$

(4)

$$s_i=\underset{0\le j<i}{min}(s_j+\Delta \left(f_i\right)+\sum _{k\in S_j}P\left(e_{k,i}\right))$$

(5)

Let r be the value of j where the minimum was identified. The corresponding $S_i$ is calculated by Equation (6).

$$S_i=S_r\cup \left\{i\right\}$$

(6)

The final score $score$ and feature selection result $Solution$ are given by Equation (7). 1.5

$$\begin{array}{c}score={\displaystyle \underset{0<i\le n}{min}}{s}_i,\\ solution=i,\mathrm{where}score\mathrm{was}\mathrm{found},\\ Solution=S_{solution}.\end{array}$$

(7)

Figure 2a shows the situation immediately before Equations (5) and (6) are applied to vertex $f_i$, and Figure 2b shows the situation immediately after the equations are applied. Green indicates the graph vertices that have already been processed, and white indicates those that are being or will be processed. The red text indicates vertex attributes modified during the observed $f_i$ processing.

Figure 2. The concept of feature selection based on dynamic programming: (a) partial solution to be stored in $f_i$ considers the solutions stored in all its predecessors; (b) the situation after updating the status of $f_i$. $S_i$ and $s_i$ are calculated with Equations (6) and (5), respectively.

To date, everything seems straightforward, but there are, in fact, three serious problems in the process that need to be addressed. The first is that the importance of vertices and edges might differ. For this reason, we introduce a weight w, $0\le w\le 1$. This modifies the evaluation criterion Equation (5) into Equation (8).

$$s_i=\underset{0\le j<i}{min}(s_j+w·\Delta \left(f_i\right)+(1-w)·\sum _{k\in S_j}P\left(e_{k,i}\right))$$

(8)

The second problem is that Equation (5) in its present form always leads to a trivial solution from Equation (9). Since the weights of the graph vertices and edges are all non-negative, the minimum consists of a single vertex (without incident edges) with the lowest weight.

$$score=\underset{0<j\le n}{min}(w·\Delta \left(f_j\right))$$

(9)

To prevent this, we first modified the model by replacing the decreasing vertex evaluation function $\Delta$ with the increasing ${\Delta }^{\prime }\left(f\right)=1-\Delta \left(f\right)$. The idea was to award high vertex weights and penalize high edge weights. This resulted in the optimization function Equation (10):

$$s_i=\underset{0\le j<i}{max}(s_j+w·{\Delta }^{\prime }\left(f_i\right)-(1-w)·\sum _{k\in S_j}P\left(e_{k,i}\right)),$$

(10)

which does not tend towards the trivial solution. However, to retain complementarity with the graph-cut-based method, we preferred to choose an alternative approach, which decrements all vertex and edge weights (except those of guards and their incident edges) for user-defined non-negative values $shf{t}_{\Delta }$ and $shf{t}_P$, respectively, (see Equation (11)). Furthermore, these two additional parameters provide new possibilities for tuning, as demonstrated in Section 4.

$$s_i=\underset{0\le j<i}{min}(s_j+w·(\Delta \left(f_i\right)-shf{t}_{\Delta })+(1-w)·\sum _{k\in S_j}(P\left(e_{k,i}\right)-shf{t}_P))$$

(11)

The third problem is the most demanding. Even if all partial solutions $S_j$, $0<j<i$ were optimal, there is no guarantee that this will be the case after adding $f_i$ to any of these solutions. It is enough that $f_i$ is over-correlated with a single feature from each $S_j$, and the optimum will likely be missed. In other words, optimization defined in this way does not guarantee an optimal substructure, one of the two fundamental assumptions of dynamic programming, along with overlapping subproblems [6]. Of course, when considering $f_i$, we can no longer refresh its predecessors’ attributes $S_j$ and $s_j$. We tried to mitigate this problem by extending the evaluation criterion by predicting the contribution of vertices not yet visited and, most importantly, considering the correlation between the visited and predicted parts. The need to predict the contribution of unvisited nodes led us to a simple idea, which later turned out to be very successful, namely to reverse the graph traversing direction after arriving at $f_n$. As G is an undirected graph, the status from the previous traversal can simply be used to estimate the score $s_i$ and partial solution $S_i$. The updated evaluation criterion is given by Equation (12).

$$s_{fwd}=\underset{n+1\ge j>i}{min}(s_j+s_i+(1-w)·\sum _{k\in S_j,h\in S_i}(P\left(e_{k,h}\right)-shf{t}_P))$$

(12)

When the reverse traversal reaches $f_1$, the direction of visiting the vertices is inverted again. The evaluation criterion Equation (12) is slightly modified to Equation (13), corresponding to the forward direction from $f_1$ towards $f_n$. The only difference between the two equations is, of course, the direction and boundaries of the vertices’ traversal, written under the min function label.

$$s_{fwd}=\underset{0\le j<i}{min}(s_j+s_i+(1-w)·\sum _{k\in S_j,h\in S_i}(P\left(e_{k,h}\right)-shf{t}_P))$$

(13)

The modified evaluation criterion significantly impacts the choice of vertex $f_r$ (r is the value of j, providing the minimum) and thus indirectly affects the calculation of $s_i$ and $S_i$. Let r be the value of j in Equation (12) or (13) where the minimum was identified. The score $s_i$ is then calculated by using Equation (14), while Equation (6) representing the solution subset $S_i$ remains applicable.

$$s_i=s_r+w·(\Delta \left(f_i\right)-shf{t}_{\Delta })+(1-w)·\sum _{k\in S_r}(P\left(e_{k,i}\right)-shf{t}_P)$$

(14)

However, $s_i$ and $S_i$ should not be directly refreshed by $s_{fwd}$ and $S_r\cup S_i$, since in the treatment of subsequent vertices, we assume that $s_i$ and $S_i$ can only refer to vertices that were visited before $f_i$ in the current iteration. Conversely, it would be a pity not to make better use of the great potential that Equations (12) and (13) certainly have. Fortunately, they can be used to predict the attributes of another vertex instead of $f_i$, namely $f_{i_{end}}$, which represents the last vertex in the set $S_i$ (the one with the lowest index in the reverse direction traversal or with the highest index in the forward traversal). However, we should not update $s_{i_{end}}$ and $S_{i_{end}}$ when we process $f_i$ because we will need the values from the previous iteration when we process $f_{i_{end}}$ later. As a consequence, we extend each vertex $f_k$ with additional attributes $pr\left(s_k\right)$ and $pr\left(S_k\right)$ ($pr$ stands for prediction), which store the aforementioned estimates of the score and the solution set. At the beginning of each iteration, the initialization $pr\left(s_k\right)=\infty ,0<k\le n$, is performed. Algorithm 1 shows the processing of vertex $f_i$, which is further explained in Figure 3. For simplicity, we assume that all the variables in Algorithm 1 are global, except i and $forward$. The score $s_i$ is determined as the minimum between the previously stored $pr\left(s_i\right)$ and $s_i$ computed by Equation (14). In the former case, the set $pr\left(S_i\right)$ is assigned to $S_i$, while in the latter case, $S_i$ is determined by Equation (6). Note that $pr\left(s_i\right)$ and $pr\left(S_i\right)$ can be refreshed multiple times in the same iteration since multiple sequences $S_i$ at different i can terminate with the same vertex $f_{i_{end}}$.

Algorithm 1 Processing a Considered Graph Vertex
1: function ProcessVertex(i, $forward$)
2:     if $forward$ then	▹ Forward direction graph traversal.
3:         $i_{end}={max}_{f_k\in S_i}k$;
4:         $s_{fwd}={min}_{0\le j<i}(s_j+s_i+(1-w)·{\sum }_{k\in S_j,h\in S_i}(P\left(e_{k,h}\right)-shf{t}_P))$;	▹ (13)
5:     else	▹ Reverse direction graph traversal.
6:         $i_{end}={min}_{f_k\in S_i}k$;
7:         $s_{fwd}={min}_{n+1\ge j>i}(s_j+s_i+(1-w)·{\sum }_{k\in S_j,h\in S_i}(P\left(e_{k,h}\right)-shf{t}_P))$;	▹ (12)
8:     end if
9:     $r=$ the value of j, where the minimum in line 4 or 7 was achieved;
10:     $s_i=s_r+w·(\Delta \left(f_i\right)-shf{t}_{\Delta })+(1-w)·{\sum }_{k\in S_r}(P\left(e_{k,i}\right)-shf{t}_P)$;	▹ (14)
11:     $S_i=S_r\cup \left\{i\right\}$;	▹ (6)
12:     if $(pr\left(s_i\right)<s_i)$ then	▹ Update the vertex with predictions from the same iteration.
13:         $s_i=pr\left(s_i\right)$;
14:         $S_i=pr\left(S_i\right)$;
15:     end if
16:     if $(s_{fwd}<pr\left(s_{i_{end}}\right))$ then	▹ Update the predictions of a not yet processed $f_{i_{end}}$.
17:         $pr\left(s_{i_{end}}\right)=s_{fwd}$;
18:         $pr\left(S_{i_{end}}\right)=S_r\cup S_i$;
19:     end if
20:     return	▹ No value returned—all the variables are global, except i and $forward$.
21: end function

Figure 3. The concept of feature selection based on alternating suboptimal dynamic programming: the situation (a) before processing $f_i$ during the reverse direction traversal; (b) after processing $f_i$ during the reverse direction traversal; (c) before processing $f_i$ during the forward direction traversal; and (d) after processing $f_i$ during the forward direction traversal.

Figure 3a,b show the situation immediately before and after Equations (6), (12) and (14) are applied to vertex $f_i$, respectively. The graph traversal is performed in the reverse direction. The obvious difference between the straightforward non-iterative solution from Figure 2 is that here, $S_i$ does not contain the initial vertex $f_i$ only, but the partial solution from the previous iteration instead. As a consequence, there is a double loop in the sum calculation. The green color indicates the graph vertices that have already been processed in the observed iteration, and the yellow color indicates those that were processed in the previous iteration (and are or will be processed later in the current iteration). Note that these yellow vertices contain the predictions (colored cyan), which might be updated earlier in the ongoing iteration. The red text indicates vertex attributes modified during the observed $f_i$ processing. Analogously, Figure 3c,d show the processing of vertex $f_i$ when the graph is passed in the forward direction. Equation (13) replaces Equation (12) in this case.

The pseudocode in Algorithm 2 describes the overall structure of the alternating suboptimal dynamic programming method for feature selection. As mentioned, 200 features can still be processed relatively fast, but for larger input sets, it makes sense to preprocess the features with graph-cut-based feature selection filtering (line 2). The initialization in line 3 sets up the guard vertices using Equation (3). Partial solution sets candidates and their scores are initialized using Equation (4), which is needed in lines 4, 7 and 11 of Algorithm 1 within the first-iteration calls of ProcessVertex (line 11 of Algorithm 2). The value $finalScore$ is set to some high value (∞) to provide the first comparison in line 16, and $MaxIterations$ is set to a user-defined value or default 100. In line 8, all predicted scores are set to a high value (∞) at the beginning of each iteration, which is needed in line 16. The main work is done in the ProcessVertex function, which is called sequentially in line 11 for each feature $f_i$ except for the guard vertices. The direction of traversing the features is inverted in each iteration (line 23). The process terminates when the identical score is obtained three times in a row, or the number of iterations reaches $maxIterations$ (line 24). If there are two (or more) solutions with the same score, the algorithm may find one during the forward direction traversal and a different one in the reverse direction traversal. In this case, it will return the last of the two solutions found.

3.3. Convergence and Complexity Analysis

The solution found is generally suboptimal but often better than that found in the one-pass method, as will be confirmed by the results in the next section. In any case, the solution after several passes is not worse than the one-pass solution since the result can only improve from iteration to iteration or remain unchanged (after three consecutive such iterations, the algorithm terminates), which is confirmed by Proposition 1 below.

Proposition 1. 

The score in each iteration of the proposed alternating suboptimal dynamic programming algorithm can only be lower (better) or equal to the score in the previous iteration but never higher (worse).

Algorithm 2 Alternating Suboptimal Dynamic Programming
1: function ASDP($\Delta$, P, n)
2:     $(\Delta ,P,n)=$ GraphCutBasedFeatureSelection($\Delta ,P,n$);	▹ Optional filtering
3:     $(\Delta ,P,s,S,finalScore,maxIterations)=$ Init($\Delta ,P,n$);
4:     $solutionRepeated=0$;    $iteration=0$;
5:     $start=1$;    $end=n$;
6:     repeat	▹ iterations of ASDP
7:         for $i\leftarrow start$ to $end$ do	▹ for all features
8:            $pr\left(s_i\right)=\infty$;
9:         end for	▹ for all features
10:         for $i\leftarrow start$ to $end$ do	▹ for all features
11:            ProcessVertex(i, $start<end$);
12:         end for	▹ for all features
13:         $score={min}_{0<i\le n}{s}_i$;	▹ (7): this and the next two lines
14:         $solution=i$, where $score$ was found;
15:         $Solution=S_{solution}$;
16:         if $(score<finalScore)$ then
17:            $finalScore=score$;
18:            $solutionRepeated=0$;
19:         else
20:            $solutionRepeated=solutionRepeated+1$;
21:         end if
22:         $iteration=iteration+1$;
23:         $(start,end)=$ Swap$(start,end)$;
24:     until $(iteration=maxIterations)\vee (solutionRepeated=3)$;
25:     return ($finalScore$, $Solution$)
26: end function

Proof 

(Proof). The proof is conceptually straightforward since we will show that the score $s_i$ from the previous iteration is also considered a candidate for the minimum in the observed iteration. Namely, this score is obtained in the evaluation criterion in line 7 of Algorithm 1 at $j=0$ or in line 4 at $j=n+1$. The algorithm does not modify the parameters of the two guards, so $s_j=0$ and $S_j=\varnothing$ in both cases. Consequently, only $s_i$ remains from the expression on the right of (13) or (12). If $s_i$ is also the minimum in the current iteration, then $s_{fwd}=s_i$ will be written first to $pr\left(s_{i_{end}}\right)$ in line 17 of Algorithm 1, then to $s_i$ in line 13 of Algorithm 1, to $score$ in line 13 of Algorithm 2, and finally to $finalScore$ in line 17 of Algorithm 2. Conversely, if $s_i$ is not the minimum in the current iteration, then it can only be replaced with a lower score in some of the aforementioned lines of Algorithm 1 or Algorithm 2. This completes the proof. □

Based on Proposition 1, it makes sense to modify the initialization (line 3 of Algorithm 2). The proven convergence allows us to use the input feature set instead of the empty set as an initial solution candidate. Equation (15) introduces a recursive definition of initial values, which replaces Equation (4). Note that the last two lines of Equation (15) were derived from Equations (6) and (14) by setting $r=i-1$.

$$\begin{array}{c}{S}_0=S_{n+1}=\varnothing ,s_0=s_{n+1}=0,\\ S_1=\left\{1\right\},s_1=\Delta \left(f_1\right),\\ S_i=S_{i-1}\cup \left\{i\right\},2\le i\le n,\\ s_i=s_{i-1}+w·(\Delta \left(f_i\right)-shf{t}_{\Delta })+(1-w)·{\displaystyle \sum _{k\in S_{i-1}}}(P\left(e_{k,i}\right)-shf{t}_P),2\le i\le n.\end{array}$$

(15)

Propositions 2–4 consider the time and space complexity of the graph-cut-based and the alternating suboptimal dynamic programming feature selection approaches.

Proposition 2. 

The graph-cut-based feature selection method has the worst-case time complexity $O\left(n^2\right)$, where n is the number of features, i.e., graph vertices.

Proof 

The algorithm gradually selects features $\widehat{f_r}$ with the highest quality, which requires at most $O\left(n\right)$ steps. In each step, a neighborhood $Z\left(\widehat{f_r}\right)$ is considered, which contains at most $O\left(n\right)$ features. This results in $O\left(n\right)·O\left(n\right)=O\left(n^2\right)$ worst-case time complexity. Note that the method removes the considered features and their highly correlated neighborhood from the graph G in each step and, consequently, the expected time complexity is much closer to $O(n·logn)$, which corresponds to sorting the vertices according to their qualities. □

Proposition 3. 

The proposed alternating suboptimal dynamic programming feature selection approach runs in $O\left(n^4\right)$ in the worst case, where n is the number of graph vertices (features).

Proof 

A double sum in lines 4 and 7 of Algorithm 1 contributes $O\left(n^2\right)$ time. In both cases, it is performed within the min function, which considers $O\left(n\right)$ values. The ProcessVertex function thus requires $O\left(n\right)·O\left(n^2\right)=O\left(n^3\right)$ time. It is called $O\left(n\right)$ times in line 11 of Algorithm 2, resulting in $O\left(n^4\right)$ time per a single iteration. Although the number of iterations (loop of lines 6–24) is by default set to 100, it rarely exceeds ten and practically never 15, so its time consumption may be considered constant, i.e., $O\left(1\right)$, and the overall worst-case time complexity is proven $O\left(n^4\right)$. □

Proposition 4. 

Both considered approaches to feature selection, i.e., the graph-cut-based and the alternating suboptimal dynamic programming algorithm, require $O\left(n^2\right)$ space, where n is the number of graph vertices (features).

Proof 

In the graph-cut-based approach, the graph contains n vertices and at most $\frac{n·(n-1)}{2}$ edges. Similarly, there are $n+2$ vertices and $\frac{(n+2)·(n+1)}{2}-1$ edges in the ASDP approach. Furthermore, $n+2$ sets $S_i$ and $pr\left(S_i\right)$, each with $O\left(n\right)$ elements, also do not exceed $O\left(n^2\right)$ space. The overall space complexity is thus $O\left(n^2\right)$. □

4. Results

4.1. Validation Setup

The proposed method based on alternating suboptimal dynamic programming (ASDP) and the exhaustive search algorithm (brute force, BF) was implemented using C++, while the graph-cut-based feature selection (Graph-FS) was implemented using Python 3.11.5 on the Microsoft^® Windows 11 operating system. All experiments were conducted on a workstation with an Intel^® Core™ i5 CPU and 16 GB of main memory. The algorithms are not yet integrated into a common application, but the results of the Graph-FS prefiltering are imported into the ASDP and BF methods via text files. The reproducibility of classification experiments is provided through the scikit-learn 1.4.1 implementation of machine learning methods. Classifiers were implemented with the following settings:

• K-Nearest neighbors classifier (KNN) was assessed using default settings, where $K\in \{2,3,\cdots ,8\}$ were tested;
• Naive Bayes classifier (NBC) was used with the default settings;
• Random Forest (RF) was of maximal depth from the range $\left\{2,4,8,16,20\right\}$, while the maximal number of iterations was from $\left\{5,10,15,20,25,30\right\}$;
• XGBOOST was of maximal depth from the range $\left\{2,4,8,16,20\right\}$, while the maximal number of iterations was from $\left\{5,10,15,20,25,30\right\}$.

The ASDP and BF evaluation and the classification accuracy assessment were conducted on nine well-known benchmark datasets, available at the UCI machine learning repository [59]. Table 1 summarises the characteristics of each dataset, including its name and the number of features, classes, and samples contained.

Table 1. Description of test datasets.

Dataset ID	Dataset Name	# Features	# Samples	# Classes
Ds1	Abalone	8	4177	2
Ds2	Credit Approval	15	690	2
Ds3	Diabetes	8	768	2
Ds4	Ionosphere	34	351	2
Ds5	Letters	16	20,000	26
Ds6	Sonar	60	208	2
Ds7	Spambase	57	4601	2
Ds8	Vehicle	18	946	4
Ds9	Wisconsin Brest Cancer Diagnostic	30	569	2

These datasets were chosen to demonstrate the diversity of real-world applications of the proposed methods. For example, while Ds2 presents the utility of feature selection for financial institutions, Ds3 and Ds9 show that feature selection is also beneficial for medical research. Furthermore, to prove the proposed method’s efficiency across different datasets and scenarios, examples with various numbers of features and samples were considered. In continuation, we will also show the consistency and robustness of the proposed methods, as the results will not deviate from the expected ones either in the case of Ds6, which contains 60 features, with only 208 samples, or in the case of Ds5 (it contains 16 features and 20,000 samples).

Each run of the ASDP and BF evaluation test consists of 125 experiments by employing $5·5·5$ triplets of parameters $(w,shf{t}_{\Delta },shf{t}_P)$, where $w\in \{0,0.25,0.5,0.75,1\}$, $shf{t}_{\Delta }\in \{0,me{d}_{1/4}(\Delta ),med(\Delta ),me{d}_{3/4}(\Delta ),1\}$, and $shf{t}_P\in \{0,me{d}_{1/4}\left(P\right),med\left(P\right),$$me{d}_{3/4}\left(P\right),1\}$. Here, $med(\Delta )$ is the median of $\Delta \left(f_i\right),0<i\le n$, while $me{d}_{1/4}(\Delta )$ and $me{d}_{3/4}(\Delta )$ are the medians of the lower and higher half-sequences of $\Delta \left(f_i\right)$, respectively. In a similar manner, the medians $me{d}_{1/4}\left(P\right)$, $med\left(P\right)$, and $me{d}_{3/4}\left(P\right)$ are determined.

4.2. Assessment of Scores of the Alternating Suboptimal Dynamic Programming Algorithm

The main question with respect to the ASDP method development was how much could it improve the solution compared to a single iteration of suboptimal dynamic programming (SDP-1). At the same time, it is reasonable to compare the extent to which ASDP and SDP-1 achieve the global optimum provided by the BF approach. The results of the analysis are summarized in Table 2. The three main conclusions are listed below the table.

Table 2. Comparison of scores obtained by BF, SDP-1, and the ASDP method.

Dataset ID	# Tests	SDP-1 Score = BF Score [%]	ASDP Score = BF Score [%]	SDP-1 Score = ASDP Score [%]	Max. # Iterations	Avg. # Iterations
Ds1	125	58.4	100.0	58.4	5	3.4
Ds2	125	68.8	100.0	68.8	7	3.6
Ds3	125	76.8	100.0	76.8	7	3.3
Ds4	125	/	/	65.6	8	3.6
Ds5	125	54.4	94.4	54.4	6	3.4
Ds6	125	/	/	52.0	7	3.9
Ds7	125	/	/	50.4	11	4.4
Ds8	125	54.4	84.8	54.4	7	3.7
Ds9	125	/	/	72.0	8	3.7
Total *	1125	62.6	95.8	61.4 (62.6)		3.7

* The Tests column contains the sum, and the others contain average values.

• The third column shows that SDP-1 reaches the global optimum in 62.6% of the tests. The fourth column then shows that ASDP significantly raises this percentage to 95.8.
• The degree of match (61.4%) between the SDP-1 and ASDP scores in the fifth column should not be below that between SDP-1 and BF (62.6%) since ASDP never degrades the score from the first iteration, according to Proposition 1. Indeed, if we ignore rows Ds4, Ds6, Ds7, and Ds9, where we could not evaluate BF, we also obtain 62.6% for ASDP (in brackets). Interestingly, at least for the tests performed, a conclusion can be drawn that whenever ASDP fails to reach the global optimum in the first iteration, it improves the score at least a little in subsequent iterations.
• The last two columns confirm the empirical finding of the proof of Proposition 3 that the number of iterations of ASDP is within $O\left(1\right)$, since in the tests performed, it does not exceed 11, and on average it is only $3.7$, barely above the termination condition of 3 consecutive iterations with the unchanged score.

In order to further improve the results and, in particular, the feasibility in situations with a larger number of features, we preprocessed ASDP with fast and highly accurate, though still suboptimal, Graph-FS. The results are shown in Table 3, and the critical observations are listed immediately below.

Table 3. Comparison of scores of Graph-FS filtering used alone or postprocessed by BF or ASDP.

Dataset ID	# Features Selected by Graph-FS	# Tests	BF Score = Graph-FS Score [%]	ASDP Score = Graph-FS Score [%]	ASDP Score = BF Score [%]	Max. # Iterations	Avg. # Iterations
Ds1	1 or 2	250	74.0	74.0	100.0	3	3.0
Ds2	2 to 15	1250	29.8	29.8	99.1	8	3.3
Ds3	1 to 8	500	50.2	50.2	99.8	6	3.1
Ds4	2	125	32.0	32.0	100.0	3	3.0
Ds5	10 to 13	625	32.6	32.6	91.7	8	3.5
Ds6	1 to 11	625	43.7	43.7	98.7	6	3.5
Ds7	12 to 56	1375	27.2 *	22.0	94.4 *	12	4.0
Ds8	5 to 10	500	26.0	26.0	93.2	5	3.0
Ds9	1 to 7	375	53.6	53.3	99.7	5	3.3
Total **		5625	38.7	34.8	98.0		3.4

* Only 125 tests used due to the limited number of features in BF. ** The Tests column contains the sum, and the others contain average values.

• The second column confirms a significantly lower number of features than before the use of Graph-FS (see Table 1).
• The fourth column shows that BF did not change the Graph-FS results in 38.7% of tests. In other words, it obtains a better score in 61.3% of cases.
• The fifth column gives the first impression that ASDP performs significantly worse (34.8% vs. 38.7) compared to BF. However, eliminating all tests on the Ds7 dataset, where BF was not viable, made both scores equal. Since ASDP cannot, according to Proposition 1 and the initialization from Equation (15), spoil the initial score, we may also conclude here that the score was strictly improved in the remaining 61.3% of tests. However, a better ASDP score obtained with Equation (14) does not necessarily imply better results in practical applications. We will show this in Section 4.3 by matching the ASDP score with the classification accuracy.
• The sixth column shows that preprocessing of ASDP with Graph-FS raises the proportion of solutions reaching the global optimum from 95.8% in Table 2 to 98%.
• The last two columns show a maximum number of iterations of 12 and a lower number of iterations of 3.4 compared to 3.7 from Table 2.

4.3. Assessment of the Use of the Proposed Approach in Classification Tasks

In this section, we demonstrated the usability of Graph-FS and ASDP for feature selection for classification tasks on the real benchmark datasets displayed in Table 1. For this purpose, we compared the classification performance of the selected features for both presented methods and their combination (Graph-FS + ASDP) with the performance of the same classifiers when learning about the input feature set. The results are shown in Table 4, Table 5, Table 6 and Table 7 for each specific classifier used. All tests were conducted by ten-fold cross-validation [60], using average accuracy $acc$ to indicate the method’s efficiency. The accuracy is defined by (16):

$$acc=\frac{\mathrm{number} \mathrm{of} \mathrm{correctly} \mathrm{classified} \mathrm{samples}}{\mathrm{number} \mathrm{of} \mathrm{all} \mathrm{classified} \mathrm{samples}}.$$

(16)

Table 4. Accuracies for RF classifier after the feature selection with Graph-FS, ASDP, their combination, or when using all input features.

	Graph-FS				ASDP		Graph-FS + ASDP		Input Data
Dataset ID	# Selected Features	$\mathit{acc}$	${\mathit{T}}_{\Delta }$	${\mathit{T}}_{\mathit{p}}$	# Selected Features	$\mathit{acc}$	# Selected Features	$\mathit{acc}$	$\mathit{acc}$
Ds1	1	46.17	0.35	0.4	5	53.11	1	46.17	53.34
Ds2	7	98.55	0.4	0.4	3	97.10	3	97.10	97.10
Ds3	3	80.51	0.3	0.4	2	74.02	2	74.02	71.42
Ds4	2	100	0.3	0.4	8	100	2	100	100
Ds5	13	95.85	0.3	0.65	15	94.95	11	88.25	95.20
Ds6	10	100	0.3	0.7	14	100	6	100	100
Ds7	56	96.52	0.45	0.55	14	90.67	13	85.46	94.79
Ds8	8	78.82	0.3	0.8	13	78.82	5	77.64	74.11
Ds9	7	78.94	0.35	0.8	1	78.94	5	75.43	75.43

Table 5. Accuracies for XGBOOST classifier after Graph-FS, ASDP, Graph-FS + ASDP, or when using all input features.

	Graph-FS				ASDP		Graph-FS + ASDP		Input Data
Dataset ID	# Selected Features	$\mathit{acc}$	${\mathit{T}}_{\Delta }$	${\mathit{T}}_{\mathit{p}}$	# Selected Features	$\mathit{acc}$	# Selected Features	$\mathit{acc}$	$\mathit{acc}$
Ds1	1	55.02	0.35	0.4	5	54.54	1	55.02	53.34
Ds2	7	98.55	0.4	0.4	3	97.10	3	97.10	95.65
Ds3	3	72.72	0.3	0.4	2	71.45	2	71.42	71.42
Ds4	2	100	0.3	0.4	8	100	2	100	100
Ds5	13	95.70	0.3	0.65	15	95.70	11	90.00	95.35
Ds6	10	100	0.3	0.7	14	100	6	100	100
Ds7	56	96.52	0.45	0.55	14	89.15	13	85.68	94.79
Ds8	8	75.29	0.3	0.8	13	74.11	5	70.58	75.29
Ds9	7	78.94	0.35	0.8	1	78.94	4	77.19	75.43

Table 6. Accuracies for NBC after the feature selection with Graph-FS, ASDP, their combination, or when using all input features.

	Graph-FS				ASDP		Graph-FS + ASDP		Input Data
Dataset ID	# Selected Features	$\mathit{acc}$	${\mathit{T}}_{\Delta }$	${\mathit{T}}_{\mathit{p}}$	# Selected Features	$\mathit{acc}$	# Selected Features	$\mathit{acc}$	$\mathit{acc}$
Ds1	1	55.02	0.35	0.4	5	54.06	1	55.02	53.34
Ds2	7	98.55	0.4	0.4	3	97.10	3	97.10	95.65
Ds3	3	75.32	0.3	0.4	2	72.72	2	72.72	71.42
Ds4	2	100	0.3	0.4	8	100	2	100	100
Ds5	13	63.40	0.3	0.65	15	62.60	11	52.6	61.45
Ds6	10	100	0.3	0.7	14	100	6	100	100
Ds7	56	90.60	0.45	0.55	14	91.32	13	97.61	59.65
Ds8	8	52.94	0.3	0.8	13	56.47	5	50.58	49.41
Ds9	7	78.94	0.35	0.8	1	78.94	4	77.19	75.43

Table 7. Accuracies for KNN classifier after Graph-FS, ASDP, Graph-FS + ASDP, or when using all input features.

	Graph-FS				ASDP		Graph-FS + ASDP		Input data
Dataset ID	# Selected Features	$\mathit{acc}$	${\mathit{T}}_{\Delta }$	${\mathit{T}}_{\mathit{p}}$	# Selected Features	$\mathit{acc}$	# Selected Features	$\mathbf{acc}$	$\mathit{acc}$
Ds1	1	51.67	0.35	0.4	5	55.02	1	51.67	53.34
Ds2	7	95.65	0.4	0.4	3	95.65	3	95.65	78.26
Ds3	3	71.42	0.3	0.4	2	75.32	2	75.32	71.42
Ds4	2	100	0.3	0.4	8	100	2	100	100
Ds5	13	94.85	0.3	0.65	15	94.70	11	86.8	94.45
Ds6	10	100	0.3	0.7	14	100	6	100	100
Ds7	56	91.54	0.45	0.55	14	92.62	13	84.38	50.45
Ds8	8	67.05	0.3	0.8	13	72.94	5	63.52	69.41
Ds9	7	78.94	0.35	0.8	1	78.94	4	77.19	75.43

Note that the $acc$ values in the tables represent the highest achieved classification results. Namely, in all test cases, all combinations of the classifier’s parameter values (see Section 4.1) were tested, except for the NBC. The latter is a non-parametric method and it was used with the default settings. We also report the number of features selected and parameters $T_{\Delta }$ and $T_p$ used in the Graph-FS and Graph-FS + ASDP methods while obtaining the listed highest results. Since identical results were typically obtained for different combinations, we do not list ASDP parameters w, $shf{t}_{\Delta }$, and $shf{t}_p$. Table 1 gives the number of input features. The highest accuracy for each dataset is emphasized in bold. Here, we considered that the same accuracy can be achieved across different methods, regardless of selected features.

Analysis shows an improvement in accuracy on the original dataset for all test cases except in the case of Ds1 for classifier RF. Furthermore, Graph-FS and ASDP achieved similar classification scores. However, Graph-FS showed slightly higher accuracy for Ds2, Ds3, Ds5, and Ds8 for classifier RF, while the same results as ASDP are shown in the case of Ds4, Ds5, Ds6, and Ds9. For classifier XGBOOST, similar results are obtained, where Graph-FS is slightly better in classification accuracy than ASDP in cases Ds1, Ds2, Ds3, Ds7, and Ds8. In the case of classifier NBC, Graph-FS achieved the best results in cases Ds2, Ds3, Ds5, and Ds8, while for Ds8, ASDP provides the most informative feature subset, achieving the highest accuracy among those in the comparison. We observed different results for the last classifier, KNN, with the ASDP showing superior performance. It achieved highest accuracy in cases Ds1, Ds3, Ds7, and Ds8.

Conversely, when comparing ASDP and Graph-FS + ASDP, we noticed improved classification performance of selected classifiers in some cases. For example, in the case of Ds4 for classifier RF, we achieved the highest classification accuracy with Graph-FS + ASDP for a selected feature subset that contains only two features, while Graph-FS and ASDP achieved the same results when subsets of 10 and 14 features were selected, respectively. Similar results can be found in the case of Ds2 and Ds6 across all classifiers, Ds1 for NBC, and Ds2 and Ds3 for the KNN classifier, where the combination of Graph-FS and ASDP achieved the highest measured accuracy but with a smaller number of features than Graph-FS and ASDP individually. The most interesting result is that for Ds7 for NBC, where Graph-FS and ASDP combined achieved the highest accuracy among all the measured results.

Finally, the results demonstrate the robustness of both approaches, as no significant deviations regarding the improvements were displayed in experiments with various datasets with different numbers of features or samples. Both, ASDP and Graph-FS + ASDP, achieved comparable results regardless of the number of features, which can be low (e.g., Ds1 and Ds3) or high (e.g., Ds7 and Ds9). In addition, both approaches showed improvements in classification accuracy in datasets containing both small and large numbers of samples.

5. Discussion

This paper introduces an alternating suboptimal dynamic programming (ASDP) algorithm, primarily aimed at improving feature selection, at least in some cases, and being competitive in others. It iteratively considers individual features and inverts the processing order in each iteration. This allows the optimization function to be improved by using the score from the previous iteration to estimate the contribution of yet unprocessed features in the current one. We proved that convergence is achieved and that the time complexity displays a polynomial ($O\left(n^4\right)$) relationship. Results on nine well-known benchmark datasets for machine learning tasks demonstrated that a single iteration suboptimal dynamic programming found the global optimum in $62.6\%$ cases, which was significantly improved to $95.8\%$ by ASDP in only $3.7$ iterations on average (and never above 12). Although ASDP is relatively slow and thus limited to 200–300 input features, we have extended its usability by preprocessing it with our fast and highly accurate graph-cut-based feature selection (Graph-FS) method. This raised the proportion of solutions reaching the global optimum to $98\%$ and reduced the average number of iterations to $3.4$.

We have also shown the practicality of using ASDP and the Graph-FS + ASDP combination in classification. The latter was slightly behind or equal to the Graph-FS alone when using the RF or XGBOOST classifiers and sometimes slightly better when using the NBC. The former seems contradictory to the proven convergence of ASDP, but the optimization criterion of ASDP and the classification accuracy of the used classifiers do not guarantee the perfect consistency of results. Surprisingly, the ASDP method without Graph-FS prefiltering performed best when using the KNN classifier. Finally, in all but one case for RF, the presented methods achieved better classification accuracy than the classifiers learned from the complete input feature set. Note that the superior performance of Graph-FS in comparison to state-of-the-art approaches was already demonstrated in [9]. We may thus conclude that ASDP and Graph-FS + ASDP are also entirely competitive.

The four contributions of the proposed method, listed in Section 1, were justified as follows. The first was confirmed by the proof of Proposition 1 and by the results in Table 2. Table 2 also confirmed the second promised contribution, which was further exceeded by the results in Table 3. The third contribution was confirmed by the proof of Proposition 3, as well as by the fact that the BF score in some cases in Table 2 could not be determined due to excessive time complexity. The fourth contribution was confirmed by the experiments in Section 4.3, in particular by the results in Table 6 and Table 7.

A disadvantage of using ASDF without preprocessing is that a larger number of features makes the method too slow or, depending on the implementation, even infeasible. It processes 200 features in 5 s on a regular PC and becomes useless at 500 features. This represents a significant improvement compared to the exhaustive search approach, which achieves such performance at a very modest 25 and 30 features, respectively. However, for larger input sets, it makes sense to preprocess ASDF with some faster filtering. Conversely, Graph-FS + ASDF restricts the solution search space to subsets of the Graph-FS solution. We will try to achieve a compromise by cascading Graph-FS over 2–5 iterations, in which in each iteration, we will gradually lower the thresholds $T_{\Delta }$ and $T_P$ and extend the selected set with features chosen from those not yet in the solution. We would also like to evaluate the use of ASDF in regression tasks in the future. In addition, we expect that the idea of alternating suboptimal optimization will soon be generalized to tasks beyond feature selection as well. In general, graph nodes can represent a wide variety of entities, and edges can represent any bilateral operation, such as distance, similarity, or correlation.