GFLASSO-LR: Logistic Regression with Generalized Fused LASSO for Gene Selection in High-Dimensional Cancer Classification

Abstract

Advancements in genomic technologies have paved the way for significant breakthroughs in cancer diagnostics, with DNA microarray technology standing at the forefront of identifying genetic expressions associated with various cancer types. Despite its potential, the vast dimensionality of microarray data presents a formidable challenge, necessitating efficient dimension reduction and gene selection methods to accurately identify cancerous tumors. In response to this challenge, this study introduces an innovative strategy for microarray data dimension reduction and crucial gene set selection, aiming to enhance the accuracy of cancerous tumor identification. Leveraging DNA microarray technology, our method focuses on pinpointing significant genes implicated in tumor development, aiding the development of sophisticated computerized diagnostic tools. Our technique synergizes gene selection with classifier training within a logistic regression framework, utilizing a generalized Fused LASSO (GFLASSO-LR) regularizer. This regularization incorporates two penalties: one for selecting pertinent genes and another for emphasizing adjacent genes of importance to the target class, thus achieving an optimal trade-off between gene relevance and redundancy. The optimization challenge posed by our approach is tackled using a sub-gradient algorithm, designed to meet specific convergence prerequisites. We establish that our algorithm’s objective function is convex, Lipschitz continuous, and possesses a global minimum, ensuring reliability in the gene selection process. A numerical evaluation of the method’s parameters further substantiates its effectiveness. Experimental outcomes affirm the GFLASSO-LR methodology’s high efficiency in processing high-dimensional microarray data for cancer classification. It effectively identifies compact gene subsets, significantly enhancing classification performance and demonstrating its potential as a powerful tool in cancer research and diagnostics.

1. Introduction

Currently, advancements in technology have made massive datasets, like DNA microarray data, fundamental to statistical analysis [1,2]. High-dimensional data pose challenges in classification tasks and gene subset identification [3]. DNA microarrays are vital in medicine and biology for various research purposes, including gene co-regulation, clinical diagnosis, and differential gene expression [4,5]. Microarray technology, crucial in cancer research, identifies genes critical to tumor growth and aids in developing diagnostic systems [6]. However, microarray data analysis is complex due to computational and biological challenges. The high cost of microarray experiments limits sample size, creating datasets with many genes but few samples [7,8], leading to computational instability and the curse of dimensionality [9,10]. Consequently, gene selection becomes a crucial yet difficult task.

Scientists aim to identify markers for lab testing to create effective disease treatments. Efficient gene selection improves understanding of gene–disease relationships and classifier quality, addressing the curse of dimensionality [11,12,13]. Gene selection, a significant challenge due to the gene-sample number disparity, requires advanced optimization methods [14,15]. Despite numerous gene selection methods aiming to enhance classification accuracy, further research is needed, particularly for diseases like cancer [16,17]. High classification accuracy is essential for personalized medicine, enabling better physician decision-making and potentially saving lives. Gene [18,19] selection algorithms include filter, wrapper, and embedded methods [20].

Filter methods evaluate genes independently, assigning scores to select high-scoring genes. Though easy to implement, they overlook gene interactions. Examples include the Fischer score, t-test, signal-to-noise ratio, information gain, and ReliefF [21,22,23,24,25]. Wrapper methods use classifiers to select gene subsets, testing candidate subsets for performance. Despite achieving high performance, they are computationally intensive and risk overfitting [20]. Embedded methods integrate selection into the learning algorithm, offering efficiency and performance without extensive classifier execution [26]. These methods consider gene dependencies but are algorithm-specific and computationally complex [27,28,29].

Filter methods evaluate genes independently, assigning scores to select high-scoring genes. Though easy to implement and computationally efficient, they have significant limitations: primarily, they overlook gene interactions, which can lead to suboptimal gene selection in complex biological processes where gene interactions play a crucial role. Examples include the Fischer score, t-test, signal-to-noise ratio, information gain, and ReliefF. Their simplicity and neglect of gene–gene interactions can lead to a lack of robustness in capturing the multifaceted nature of genetic influences on diseases [21,22,23,24,25]. Wrapper methods, on the other hand, use classifiers to select gene subsets, evaluating the performance of candidate subsets. While these methods can achieve high performance by considering gene interactions and utilizing the predictive power of classifiers, they come with their own set of limitations. They are computationally intensive, making them impractical for very large datasets, and they carry a significant risk of overfitting, especially when the number of genes vastly exceeds the number of samples. This can lead to models that perform well on training data but poorly on unseen data [20]. Embedded methods offer a middle ground by integrating gene selection directly into the learning algorithm, thus offering efficiency and potentially high performance without necessitating the execution of separate classifiers for feature selection. These methods can account for gene dependencies through their integration with the learning model, offering a more nuanced approach to gene selection. However, their algorithm-specific nature means that each method is tailored to a particular model or set of assumptions about the data, which can limit their applicability. Moreover, they can still be computationally complex, posing challenges for their use in large-scale genomic studies [27,28,29]. These limitations highlight the ongoing need for innovative solutions that balance accuracy, computational efficiency, and the ability to handle high-dimensional genomic data.

Regularized methods, particularly penalized logistic regression (PLR) with LASSO, are gaining traction for cancer classification [30,31,32,33,34,35,36,37,38]. While LASSO is effective for feature selection and model simplification, it has limitations such as potential bias in coefficient estimation, difficulty in handling highly correlated predictors, and a tendency to select only one variable from a group of highly correlated variables. These limitations prompted the development of the Elastic-Net (EN) penalty by Zou and Hastie [39], which combines the $L_1$ and $L_2$ penalties to better handle correlated predictors, and the adaptive LASSO (ALASSO) by Zou et al. [40], which introduces weights for penalization within the $L_1$ norm to improve variable selection accuracy.

To overcome the deficiencies in achieving the oracle property, Zou introduced ALASSO [40], which theoretically possesses the oracle property under certain conditions. However, the performance of ALASSO can be sensitive to the choice of weights, which may require additional tuning. Structured penalties, like Fused LASSO [41,42], further enhance LASSO by considering gene relationships, aiding in feature reduction and meaningful gene selection [43]. Despite its advantages, Fused LASSO can be computationally intensive for large datasets and may not perform well if the true underlying model does not adhere to its assumption of spatial or sequential coherence. The Fused LASSO and its theoretical foundations [44,45] have led to the Generalized Fused LASSO (GFLASSO) within linear regression [46,47], which penalizes coefficient differences based on a gene graph to encourage sparse and connected solution vectors [47]. GFLASSO further extends the applicability to more complex data structures but faces challenges in scalability and the need for a well-defined gene graph, which may not always be available or accurately represent biological relationships.

The $L_1$ norm penalties’ absolute values complicate minimization due to non- differentiability at zero, challenging analytical solution derivation. Initial solutions for linear regression included FLSA [48] and other algorithms [49,50,51]. For logistic regression, fewer algorithms exist, with Hofling et al. [46] applying quadratic approximation iteratively to the log-likelihood function.

Table 1. Comparison of Contributions in DNA Microarray Data Analysis.

Reference	Methodological Approach	Application Area	Challenges Addressed	Notable Contributions
Li et al. [1]	High-dimensional data analysis	DNA microarray data	Statistical analysis difficulty in classification tasks	Developed approaches for gene subset identification
Feng et al. [2]	High-dimensional data analysis	DNA microarray data	High-dimensional data challenges in gene selection	Improved understanding of gene–disease relationships
Mehrabi et al. [3]	Efficient gene selection methods	DNA microarray data	Curse of dimensionality in microarray data analysis	Enhanced classification accuracy for diseases like cancer
Syu et al. [4]	Microarray technology analysis	Medicine and Biology	Clinical diagnosis and gene co-regulation	Identified critical genes for tumor growth and diagnostic systems development
Caraffi et al. [5]	Microarray data analysis	Cancer research	Computational and biological challenges	Developed diagnostic systems to aid in cancer treatment
Ghavidel et al. [7]	Machine learning approaches	Microarray data analysis	Computational instability and sample size limitation	Addressed the curse of dimensionality with computational methods
Birjmel et al. [12]	Optimization methods for gene selection	DNA microarray data	Gene-sample number disparity	Advanced optimization methods to improve classifier quality
Yaqoob et al. [16]	Gene selection for personalized medicine	Cancer research	Need for high classification accuracy	Optimized gene selection methods for better physician decision-making
Alharthi et al. [30]	Regularized methods, PLR with LASSO	Cancer classification	Limitations of LASSO	Introduced Elastic-Net (EN) and adaptive LASSO (ALASSO) for cancer classification
Zou et al. [40]	Adaptive LASSO	Statistical analysis	Oracle property deficiencies in LASSO	Introduced weights for penalization within the $L_1$ norm
Hofling et al. [46]	Generalized Fused LASSO (GFLASSO)	Linear regression	Challenges in $L_1$ norm penalties minimization	Penalized coefficient differences based on a gene graph for sparse solutions

presents a comparison of Contributions in DNA Microarray Data Analysis.

In the wake of comprehensive literature analysis, it becomes evident that the frontier of gene selection for cancer research is ripe for innovation. Current methodologies, while robust, exhibit limitations in computational efficiency, model overfitting, and the nuanced understanding of gene interdependencies. This literature underscores the essential balance between gene selection’s computational demands and the need for precise, interpretable models capable of navigating the vast, complex landscape of genomic data. Notably, regularized regression methods like LASSO and its derivatives represent a significant stride towards addressing these challenges. However, their effectiveness is contingent upon the integration of gene relational structures and overcoming inherent mathematical complexities.

This study introduces an innovative algorithm, termed GFLASSO, which leverages sub-gradient methods and unfolds in two distinct phases: initially, a preprocessing step where a preliminary subset of genes is selected via a univariate filter method that utilizes Pearson’s correlation coefficient; followed by the application of an improved Generalized Fused Lasso technique, which also employs Pearson’s correlation coefficient to assess inter-gene relationships. The primary objective of this research is to showcase the effectiveness of our gene selection methodology in enhancing the classification of high-dimensional data, particularly within the context of cancer research. Overall, the main contributions of this study are summarized as follows:

• Introduced an improved approach for dimension reduction and important gene selection in cancer research using DNA microarray technology.
• Integrated gene selection with classifier training into a single process, enhancing the efficiency of identifying cancerous tumors.
• Formulated gene selection as a logistic regression problem with a generalized Fused LASSO (GFLASSO) regularizer, incorporating dual penalties to balance gene relevance and redundancy.
• Utilized a sub-gradient algorithm for optimization, demonstrating the algorithm’s objective function is convex, Lipschitzian, and possesses a global minimum, meeting necessary convergence conditions.
• Showed that the GFLASSO-LR method significantly improves cancer classification from high-dimensional microarray data, yielding compact and highly performant gene subsets.

The remainder of this paper is structured as follows: Section 2 provides a detailed presentation of the proposed method, beginning with preliminary concepts before delving into the sub-gradient method and its application for estimating ${\widehat{\beta }}_{\mathrm{GFLASSO}}$. Subsequently, Section 3 evaluates the performance of our proposed approach using four well-known gene expression classification datasets, detailing both the parameter settings and the numerical results obtained. Finally, Section 4 concludes the paper and outlines directions for future work.

2. Methods

2.1. Preliminaries

In Microarray data, each value of $x_{ij}$ is the measure of the expression level of the $j^{th}$ gene in the $i^{th}$ sample, this array contains real values, and there are no missing values. Where $\mathrm{N}$ is the number of genes and $\mathrm{M}$ is the number of samples.

DNA microarray analysis for cancer classification is formulated as a supervised classification problem. The limited number of samples that can be collected due to the high cost of preparing DNA microarrays presents a challenge for statistical learning techniques. Linear models, specifically logistic regression, are commonly used for discriminant analysis in supervised learning. In binary logistic regression, it is usual to consider the values of the variable to be explained as belonging to the set $\{0,1\}$; this is a binary classification.

Let ${\Pi }_i=\mathbb{P}\left(y_i=1/x_i=x_{i1},\dots ,x_{iN}\right)$ denote the probability that the class of $x_i$ is 1, for any sample $x_i$, $i=1,\dots ,M$, of the dataset matrix.

It can then be deduced that $\mathbb{P}\left(y_i=0/x_i=x_{i1},\dots ,x_{iN}\right)=1-{\Pi }_i$. The logistic regression model is written [52] as follows:

$${\Pi }_i=\frac{e^{{\beta }_0+{\sum }_{j=1}^N{\beta }_j{x}_{ij}}}{1+e^{{\beta }_0+{\sum }_{j=1}^N{\beta }_j{x}_{ij}}}$$

where $\mathit{\beta }={\left({\beta }_0,{\beta }_1,\dots ,{\beta }_N\right)}^T$ denotes the vector of regression coefficients including a constant ${\beta }_0$. This vector will be estimated from the training data. Without loss of generality, it was assumed that the genes are centered:

${\sum }_{i=1}^M{x}_{ij}=0$ and $\frac{1}{M}{\sum }_{i=1}^M{x^2}_{ij}=1,\forall j\in \{1,2,\dots ,N\}.$

For clarity and simplicity, let us adopt the following notation. Let $i=1,2,\dots ,M$ denote

• $x_i^{\prime }=(1,x_i)=(1,x_{i1},\dots ,x_{iN})$, or simply for any vector $\mathbf{x}=(x_1,\dots ,x_N)$ of ${\mathbb{R}}^N$, it was noted that $x^{\prime }=(1,x)=(1,x_1,\dots ,x_N)$
• $x_i^{\prime }\beta ={\beta }_0+{\sum }_{j=1}^N{\beta }_j{x}_{ij}$ is the usual scalar product.

Moving on, for $i=1,\dots ,M$,

$${\Pi }_i=\frac{e^{x_i^{\prime }\beta }}{1+e^{x_i^{\prime }\beta }}\mathrm{et}ln\left(\frac{{\Pi }_i}{1-{\Pi }_i}\right)=x_i^{\prime }\beta$$

The probability of observing the response $y_i$ for a sample $x_i$ is written in a more compact way:

$$\mathbb{P}\left(y_i/x_i\right)={\Pi }_i^{y_i}{(1-{\Pi }_i)}^{1-y_i}$$

Let f be the “sigmoid” function used in the logistic regression defined by $f\left(z\right)=\frac{1}{1+e^{-z}}$ for any real z; the latter has a value in $[0,1]$. It can be noticed then that ${\Pi }_i=f\left(x_i^{\prime }\beta \right)$.

The classification rule for a new example x via logistic regression is therefore defined by

$$Class\left(x\right)=\left\{\begin{array}{cc}0\hfill & \mathrm{if}f\left(x^{\prime }\beta \right)<0.5\hfill \\ 1\hfill & \mathrm{if}f\left(x^{\prime }\beta \right)⩾0.5\hfill \end{array}\right.$$

The methods for estimating the $\beta$ coefficients include the statistical method called “maximum likelihood estimation”, which allows us to determine the values of the $\beta$ parameter of the model, which renders the following maximum value

$$L\left(\beta \right)=\prod _{i=1}^M{\Pi }_i^{y_i}{\left(1-{\Pi }_i\right)}^{1-y_i}$$

This is equivalent to finding $\beta$ in ${\mathbb{R}}^{N+1}$, which maximizes the log-likelihood function.

$$ln\left(L\left(\beta \right)\right)=\sum _{i=1}^M{y}_{i}ln\left({\Pi }_i\right)+\left(1-y_i\right)ln\left(1-{\Pi }_i\right)$$

Maximizing $ln\left(L\right(\beta \left)\right)$ is the same as minimizing $-ln\left(L\right(\beta \left)\right)$. Let us then note $g\left(\beta \right)=-ln\left(L\right(\beta \left)\right)$ for all $\beta$ of ${\mathbb{R}}^{N+1}$. After some elementary calculations, we arrive at

$$g\left(\beta \right)=-\sum _{i=1}^M\left(y_i{x}_i^{\prime }\beta -ln\left(1+e^{x_i^{\prime }\beta }\right)\right)$$

In cancer classification, logistic regression is used to model the relationship between a set of independent variables (genes) and a binary dependent variable (tumor or normal class) by minimizing the function $g\left(\beta \right)$ in Equation (1).

$$\begin{array}{c}{\widehat{\beta }}_{LR}=arg\underset{\beta \in {\mathbb{R}}^{N+1}}{min}g\left(\beta \right)=arg\underset{\beta \in {\mathbb{R}}^{N+1}}{min}-\sum _{i=1}^M(y_i{x}_i^{\prime }\beta \\ -ln\left(1+e^{x_i^{\prime }\beta }\right))\end{array}$$

(1)

However, logistic regression may be limited in classifying high-dimensional data as the minimization problem may not have a unique solution, as shown by Albert et al. [53]. They found that when the dataset is completely or nearly completely separated, the estimators of (1) become infinite and non-unique, while in the presence of overlap in the data, the minimum of (1) exists and is finite.

In microarray data, the high number of genes can lead to overfitting and multicollinearity in the estimators ($\beta$) since only a few genes are actually associated with the response variable. To enhance classification accuracy, it is important to develop gene selection methods that eliminate irrelevant genes. One such method is penalized logistic regression (PLR), which adds a positive penalty to the likelihood function g, causing some coefficients to approach zero, a technique known as regularization. The penalized log-likelihood can be expressed as shown in Equation (2):

$$PLR=g\left(\beta \right)+\lambda P\left(\beta \right)$$

(2)

where $P\left(\beta \right)$ represents the penalty term and $\lambda$ is the regularization parameter that determines the extent of the penalty. For $\lambda =0$, this can lead to the solution of (1) if it exists. On the other hand, for high values of $\lambda$, the regularization term has a greater impact on the coefficient estimates. The selection of the tuning parameter is critical in fitting the model, and if it is chosen via cross-validation, the classifier can achieve satisfactory classification accuracy. These penalized methods are commonly used in gene selection and classification of high-dimensional data, as reported in [54].

In the following, we denote ${\widehat{\beta }}_{\lambda }$ as the vector of size $N+1$ that represents a solution to the minimization problem (2), i.e.,

$${\widehat{\beta }}_{\lambda }=\underset{\beta \in {\mathbb{R}}^{N+1}}{argmin}\left(g\left(\beta \right)+\lambda P\left(\beta \right)\right)$$

(3)

Several penalty terms have been explained and applied in the literature, namely, Ridge ($L_2$) [55], LASSO ($L_1$) [56], Elastic net [39], Adaptive LASSO [40], Fused LASSO and Generalized Fused LASSO [47], Relaxed LASSO [57], Group LASSO [58], Random LASSO [59], and many others. These penalties, along with logistic regression, can be used successfully to obtain high classification rates. We are specifically interested in the Generalized Fused LASSO penalty. This focus sets the stage for introducing the sub-gradient method.

2.2. Sub-Gradient Method

The sub-gradient method is a technique for minimizing non-differentiable convex functions, similar to the ordinary gradient method for differentiable functions. However, it has some key differences, such as its ability to handle non-differentiable functions directly and the lack of guarantee for decreasing the function value.

These methods were introduced in the 1960s by N.Z. Shor [60] for the unconstrained minimization of convex functions in the field of network transport. The first convergence results are attributed to Polyak [61] and Nemirovski [62], and they have been further developed by various researchers since then.

Consider a function $f:\Omega \to \mathbb{R}$, where $\Omega \subset {\mathbb{R}}^n$ is a convex set, and $\overline{x}\in \Omega$. We define a subgradient of f at $\overline{x}$ as an element $v\in {\mathbb{R}}^n$ satisfying the following inequality:

$$\langle v,x-\overline{x}\rangle \le f\left(x\right)-f\left(\overline{x}\right)\mathrm{for}\mathrm{all}x\in \Omega$$

The set of all subgradients is termed the subdifferential of f at $\overline{x}$, denoted $\partial f\left(\overline{x}\right)$.

For a convex function $f:\Omega \to \mathbb{R}$ defined on a convex set $\Omega \subset {\mathbb{R}}^n$, several properties of its subdifferential have been well-established [63]:

• Let $\overline{x}\in int(\Omega )$, then $\partial f\left(\overline{x}\right)$ is a non-empty, convex, bounded set.
• Assuming differentiability of f at $\overline{x}\in int(\Omega )$, we have $\partial f\left(\overline{x}\right)=\nabla f\left(\overline{x}\right)$ and

  $$\langle \nabla f\left(\overline{x}\right),x-\overline{x}\rangle \le f\left(x\right)-f\left(\overline{x}\right)\mathrm{for}\mathrm{all}x\in \Omega$$
• Suppose $\Omega$ is a convex set in ${\mathbb{R}}^n$, and let $f_i:\Omega \to \mathbb{R}$ be convex functions with ${\lambda }_i>0$ for $i=1,\dots ,m$. If $\overline{x}\in \Omega$, then the subdifferential sum rule is expressed as follows:

  $$\partial \left(\sum _{i=1}^m{\lambda }_i{f}_i\right)\left(\overline{x}\right)=\sum _{i=1}^m{\lambda }_i\partial f_i\left(\overline{x}\right)$$

Let us now present some important examples of sub-differential calculation:

Example 1. 

The norm function (${\mathbf{L}}_{\mathbf{1}}$):

Let $f:x\in {\mathbb{R}}^n\mapsto {\parallel x\parallel }_1=\left|x_1\right|+\dots +\left|x_n\right|.$

The sub-differential of f is the Cartesian product of the following intervals:

$$\partial f\left(x\right)=I_1\times \cdots \times I_n,I_k=\left\{\begin{array}{cc}[-1,1]& x_k=0\\ \left\{1\right\}& x_k>0\\ \{-1\}& x_k<0\end{array}\right.$$

Example 2. 

Let n be an integer greater than or equal to $2,$ and $(k,l)\in 1,n^2$ with $k<l$.

Let $f_{k,l}$ be the function defined on ${\mathbb{R}}^n$ by $f_{k,l}:x\in {\mathbb{R}}^n\mapsto \left|x_k-x_l\right|$

The subdifferential of $f_{k,l}$ is the Cartesian product of the following intervals:

$$\partial f_{k,l}\left(x\right)=I_1\times \cdots \times I_n.$$

With:

$I_i=\left\{0\right\}$ for all $i\in 1,n\setminus \{k,l\}$$,I_k=\left\{\begin{array}{cc}[-1,1]& x_k-x_l=0\\ \left\{1\right\}& x_k-x_l>0\\ \{-1\}& x_k-x_l<0\end{array}\right.$ $,I_l=\left\{\begin{array}{cc}[-1,1]& x_k-x_l=0\\ \left\{1\right\}& x_k-x_l<0\\ \{-1\}& x_k-x_l>0\end{array}\right.$

2.2.1. Subgradient Algorithm

The classical subgradient method is designed to solve convex optimization problems without type constraints:

$$\mathrm{minimize}f\left(x\right)\mathrm{subject}\mathrm{to}x\in {\mathbb{R}}^n$$

(4)

where $f:{\mathbb{R}}^n\to \mathbb{R}$ represents a convex function. Let $\left\{{\alpha }_k\right\}$, where $k\in \mathbb{N}$, be a sequence of positive numbers. The subgradient algorithm generated by $\left\{{\alpha }_k\right\}$ is defined as follows: given an initial point $x_1\in {\mathbb{R}}^n$, the iterative procedure is as follows:

$$x_{k+1}:=x_k-{\alpha }_k{v}_k\mathrm{with}{v}_k\in \partial f\left(x_k\right),k\in \mathbb{N}$$

(5)

We presuppose that problem (4) possesses an optimal solution and that f is convex and Lipschitz on ${\mathbb{R}}^n$.

Several methodologies have been proposed in the literature, as documented in [61,63,64,65,66,67], for selecting the sequence of steps ${\alpha }_k$ to ensure convergence of the subgradient method. Note that

$$\overline{V}:=min\{f\left(x\right)\mid x\in {\mathbb{R}}^n\}\mathrm{and}{V}_k:=min\{f\left(x_1\right),\dots ,f\left(x_k\right)\}$$

There are three main results that have been found:

• Suppose ${\alpha }_k=ϵ$ for all $k\in \mathbb{N}$. Then there exists a positive constant $\mathrm{C}$ such that

  $$0\le V_k-\overline{V}<{\mathcal{l}}^2ϵ\mathrm{when}k>\frac{C}{{ϵ}^2}$$
• Suppose that

  $$\sum _{k=1}^{\infty }{\alpha }_k=\infty \mathrm{and}\underset{k\to \infty }{lim}{\alpha }_k=0$$

  (6)
  Then, we have convergence $V_k\to \overline{V}$ as $k\to \infty$, and

  $$\underset{k\to \infty }{lim\; inf}f\left(x_k\right)=\overline{V}$$
• Suppose the generated sequence $\left\{{\alpha }_k\right\}$ in (5) satisfies

  $$\sum _{k=1}^{\infty }{\alpha }_k=\infty \mathrm{and}\sum _{k=1}^{\infty }{\alpha }_k^2<\infty$$

  (7)
  Consequently, the sequence $\left\{V_k\right\}$ converges to the optimal value $\overline{V}$, and the iterative sequence $\left\{x_k\right\}$ generated by (5) converges to an optimal solution $\overline{x}$ of (4).

2.2.2. Generalized Fused LASSO

Proposed by Tibshirani and Al [42], the Fused LASSO is an extension of LASSO aiming to improve its properties when the features are ordered. It adds another penalty to $L_1$ to encourage zero differences between successive coefficients. The idea of the Fused LASSO has been generalized in the context of linear regression models (and generalized linear models) [46,47]. This is accomplished by constructing a graph $\mathcal{G}=(V,E)$ with $V=\{1,\dots ,N\}$ vertices (representing genes in DNA microarray data), each corresponding to a coefficient, and a set of edges E. In the generalized Fused LASSO, differences $\left|{\beta }_k-{\beta }_l\right|$ for $(k,l)\in E$ are penalized, with associated weights representing the degree of association between genes. The objective function is

$$\begin{array}{cc}\hfill {\widehat{\beta }}_{\mathrm{GFLASSO},{\lambda }_1,{\lambda }_2}& =arg\underset{\beta \in {\mathbb{R}}^{N+1}}{min}g\left(\beta \right)+{\lambda }_1\sum _{k=0}^N\left|{\beta }_k\right|\hfill \\ \hfill & +{\lambda }_2\sum _{(k,l)\in E,k<l}{w}_{k,l}\left|{\beta }_k-{\beta }_l\right|\hfill \end{array}$$

(8)

The penalties $w_{k,l}$ in the Fused LASSO regularizer (8) are typically determined from the initial data. Here, ${\lambda }_{k,l}={\lambda }_2\times w_{k,l}$ represents the penalty applied to the difference between genes $g_k$ and $g_l$. As $w_{k,l}$ increases, the penalty on $\left|{\beta }_k-{\beta }_l\right|$ also increases, making ${\beta }_k$ and ${\beta }_l$ more likely to be equal. The first penalty ensures many ${\beta }_j$ are zero, promoting sparsity. Therefore, ${\widehat{\beta }}_{\mathrm{GFLASSO},{\lambda }_1,{\lambda }_2}$ tends to have sparse and equal coefficients for connected genes in the graph. The choice of ${\lambda }_1$ and ${\lambda }_2$ is determined using appropriate cross-validation methods [68].

Due to the penalty (8), some coefficients of ${\widehat{\beta }}_{\mathrm{GFLASSO},{\lambda }_1,{\lambda }_2}$ are reduced to zero, indicating unused genes in the model. The parameters ${\lambda }_1$ and ${\lambda }_1$ control the degree of penalization, affecting the number of non-zero estimated coefficients. As ${\lambda }_1$ increases, the number of null components in ${\widehat{\beta }}_{\mathrm{GFLASSO},{\lambda }_1,{\lambda }_2}$ increases. The set of selected genes is defined as

$$\mathcal{S}\left({\lambda }_1,{\lambda }_2\right)=\left\{g_j:{\widehat{\beta }}_{GFLASSO,{\lambda }_1,{\lambda }_2,j}\ne 0;j=1,\dots ,N\right\}$$

where ${\widehat{\beta }}_{GFLASSO,{\lambda }_1,{\lambda }_2,j}$ is the penalized logistic regression coefficient (8) associated with the $g_j$ gene for the penalty parameters ${\lambda }_1$ and ${\lambda }_2$.

We formulate the problem of gene selection as a logistic regression problem (maximum log-likelihood) with a generalized Fused LASSO regularizer, a penalty that is rarely used in this type of problem. The selection is performed by minimizing (8).

To construct the graph $\mathcal{G}=(V,E)$ and assign weights $w_{k,l}$ to each edge $(k,l)$, we consider $V=\{1,\dots ,N\}$ as the set of vertices (genes) associated with our DNA microarray data. The interaction between genes is measured using the Pearson correlation coefficient. An edge is formed between vertices k and l if and only if the absolute correlation between genes $g_k$ and $g_l$ is greater than or equal to a threshold $r_0$. The weight of edge $(k,l)$ is defined as the absolute value of the correlation between $g_k$ and $g_l$: $w_{k,l}=\left|{\rho }_{g_k,g_l}\right|$. This way, the graph $\mathcal{G}=(V,E)$ and the weights $w_{k,l}$ are established. The correlation coefficient between $g_k$ and $g_l$ is defined as

$${\rho }_{g_k,g_l}=\frac{\mathrm{cov}(g_k,g_l)}{{\sigma }_{g_k}{\sigma }_{g_l}}$$

(9)

where $\mathrm{cov}(g_k,g_l)$ is the covariance between $g_k$ and $g_l$, ${\sigma }_{g_k}$ is the standard deviation of $g_k$, and ${\sigma }_{g_l}$ is the standard deviation of $g_l$.

The use of Fused LASSO is motivated by several factors [69]. One of the main advantages is its ability to handle high-dimensional data where the number of genes exceeds the number of samples. Traditional LASSO may struggle in such cases, while Elastic Net, though potentially outperforming LASSO, is relatively less efficient than Fused LASSO, especially when there are relationships among genes. Additionally, Fused LASSO explicitly regularizes the differences between neighboring coefficients using an $L_1$ norm regularizer, exploiting relationships between genes. This results in sparse coefficients, with non-zero components equal for some connected genes in the graph, balancing individual gene relevance and redundancy.

2.2.3. Structure of the Proposed Approach

The proposed approach consists of a two-step sequential process (as depicted in Figure 1). Firstly, a preprocessing phase eliminates irrelevant genes, selecting the top genes ranked by relevance determined by their correlation with the target class. Then, an embedded method is applied to the selected genes, involving the construction of a logistic regression classifier. The model is trained to find ${\widehat{\beta }}_{\mathrm{GFLASSO},{\lambda }_1,{\lambda }_2}$ by solving the minimization problem outlined in Equation (8) using the subgradient algorithm.

With respect to the estimation of ${\widehat{\beta }}_{\mathrm{GFLASSO},{\lambda }_1,{\lambda }_2}$ in (8), since the function in question is not differentiable due to the presence of the absolute value, we use the subgradient algorithm to solve the minimization problem (8). This algorithm has not been previously used to solve the logistic regression problem with the Generalized Fused LASSO penalty. In the following, we demonstrate the convergence of this algorithm for (8).

2.3. Sub-Gradient for Estimating ${\widehat{\beta }}_{\mathrm{GFLASSO},{\lambda }_1,{\lambda }_2}$

It is known that the sub-gradient algorithm can converge to the optimal solution of a convex minimization problem when an appropriate step sequence $\left({\alpha }_k\right)$ is chosen, as long as the function in question meets the following conditions:

• It is convex.
• It admits a global minimum.
• It is Lipschitz.

To apply this algorithm to our problem (8), we must first ensure that the function meets these three conditions.

2.3.1. Characteristics and Convexity Analysis of the GFLASSO Penalty Function

It is important to note that

$$\begin{array}{c}{h}_{{\lambda }_1,{\lambda }_2}\left(\beta \right)=g\left(\beta \right)+{\lambda }_1\sum _{j=0}^N\left|{\beta }_j\right|+{\lambda }_2\sum _{(k,l)\in E,k<l}{w}_{k,l}\left|{\beta }_k-{\beta }_l\right|\\ \forall \beta \in {\mathbb{R}}^{N+1}\end{array}$$

(10)

1.

The convexity of $h_{{\lambda }_1,{\lambda }_2}$:

First, there is

$$\begin{array}{c}g\left(\beta \right)=-\sum _{i=1}^M\left(y_i{x}_i^{\prime }\beta -ln\left(1+e^{x_i^{\prime }\beta }\right)\right)\hfill \\ =-\sum _{i=1}^M{y}_i{x}_i^{\prime }\beta +\sum _{i=1}^{M}ln\left(1+e^{x_i^{\prime }\beta }\right),\forall \beta \in {\mathbb{R}}^{N+1}\hfill \end{array}$$

Let us show that g is convex on ${\mathbb{R}}^{N+1}$:

• The function $\beta \mapsto -{\sum }_{i=1}^M{y}_i{x}_i^{\prime }\beta$ is convex; it is a linear function on ${\mathbb{R}}^{N+1}$.
• The function $\beta \mapsto x_i^{\prime }\beta$ is convex for all i in $\{1,\dots ,M\}$, so $\beta \mapsto ln\left(1+e^{x_i^{\prime }\beta }\right)$ is convex for all i in $\{1,\dots ,M\}$. (It is the composition of an increasing function convex on $\mathbb{R}$ and a convex function on ${\mathbb{R}}^{N+1}$).

The function g is therefore convex on ${\mathbb{R}}^{N+1}$, being the sum of $M+1$ convex functions. On the other hand, the function $\beta \mapsto {\lambda }_1{\sum }_{j=0}^N\left|{\beta }_j\right|$ is convex as the sum of $N+1$ convex functions.

Next, we show that the function $\beta \mapsto {\lambda }_2{\sum }_{(k,l)\in E,k<l}{w}_{k,l}\left|{\beta }_k-{\beta }_l\right|$ is convex.

Indeed, let $(k,l)\in {\{1,\dots ,N\}}^2$ with $k<l$. The function $\beta \mapsto {\lambda }_2{w}_{k,l}\left|{\beta }_k-{\beta }_l\right|=f_{k,l}\left(\beta \right)$ is convex. This is because $f_{k,l}$, defined in Example (2), is composed of an affine function and another convex one, with ${\lambda }_2{w}_{k,l}\ge 0$. Then the function $\beta \mapsto {\lambda }_2{\sum }_{(k,l)\in E,k<l}{w}_{k,l}\left|{\beta }_k-{\beta }_l\right|$ is convex as it is a sum of convex functions.

Finally, we can deduce that the function $h_{{\lambda }_1,{\lambda }_2}$ is convex as it is a sum of convex functions.

2.

The existence of a global minimum of $h_{{\lambda }_1,{\lambda }_2}$:

We recall that a real-valued function $f:{\mathbb{R}}^n\to \mathbb{R}$ is coercive if ${lim}_{\left|\mathbf{x}\right|\to \infty }f\left(\mathbf{x}\right)=\infty$.

We have

$$\forall \beta \in {\mathbb{R}}^{N+1},L\left(\beta \right)=\prod _{i=1}^M{\Pi }_i^{y_i}{\left(1-{\Pi }_i\right)}^{1-y_i}$$

and

$$g\left(\beta \right)=-ln\left(L\right(\beta \left)\right)$$

with ${\Pi }_i=\frac{e^{x_i^{\prime }\beta }}{1+e^{x_i^{\prime }\beta }}$ for all i in $\{1,\dots ,M\}$.

Additionally, we have

$\forall i\in \left[\right|1,M\left|\right],0<{\Pi }_i<1\mathit{and}{y}_i\in \{0,1\}$. So $0<L\left(\beta \right)<1\forall \beta \in {\mathbb{R}}^{N+1}$

Hence, $g\left(\beta \right)\in {\mathbb{R}}^{+*}\forall \beta \in {\mathbb{R}}^{N+1}$

Let us examine the expression for $h_{{\lambda }_1,{\lambda }_2}$. Given that the function g is positive on ${\mathbb{R}}^{N+1}$ and ${\lambda }_1$ in (10) is strictly positive, it follows that

$$h_{{\lambda }_1,{\lambda }_2}\left(\beta \right)⩾{\lambda }_1{\parallel \beta \parallel }_1\forall \beta \in {\mathbb{R}}^{N+1}$$

Based on the latest inequality, we can deduce that $h_{{\lambda }_1,{\lambda }_2}$ is a coercive function. Additionally, $h_{{\lambda }_1,{\lambda }_2}$ is continuous, so it attains a a global minimum at least once (see appendix in reference [70]).

3.

The function $h_{{\lambda }_1,{\lambda }_2}$ is ℓ-Lipschitz on ${\mathbb{R}}^{N+1}$:

We only have one condition left to establish to show that the sub-gradient algorithm converges to a global minimum of $h_{{\lambda }_1,{\lambda }_2}$; it is to prove that $h_{{\lambda }_1,{\lambda }_2}$ is ℓ-Lipschitzian on ${\mathbb{R}}^{N+1}$, i.e., find a positive ℓ such that

$$\forall \beta ,\gamma \in {\mathbb{R}}^{N+1},\left|h_{{\lambda }_1,{\lambda }_2}\left(\beta \right)-h_{{\lambda }_1,{\lambda }_2}\left(\gamma \right)\right|⩽\mathcal{l}\parallel \beta -\gamma \parallel$$

Note in passing that the Lipschitz constant ℓ depends on the choice of the norm on ${\mathbb{R}}^{N+1}$. Since all norms are equivalent on ${\mathbb{R}}^{N+1}$, whether a function is Lipschitz or not does not depend on the chosen norm.

Let us start by showing that the function g is Lipschitz. We first have

$$g\left(\beta \right)=-\sum _{i=1}^M\left(y_i{x}_i^{\prime }\beta -ln\left(1+e^{x_i^{\prime }\beta }\right)\right)\forall \beta \in {\mathbb{R}}^{N+1}$$

The function g is of class $C^1$ on ${\mathbb{R}}^{N+1}$. Find the partial derivatives of g:

Let $\beta \in {\mathbb{R}}^{N+1}$. We have

$$\forall j\in \left[\right|0,N\left|\right],\frac{\partial g\left(\beta \right)}{\partial {\beta }_j}=-\sum _{i=1}^M\left(y_i{x}_{ij}^{\prime }-{\Pi }_i{x}_{ij}^{\prime }\right)$$

Therefore,

${\parallel \nabla g\left(\beta \right)\parallel }_1\le {\sum }_{j=0}^N\left({\sum }_{i=1}^M\left(\left|y_i{x}_{ij}^{\prime }\right|+\left|x_{ij}^{\prime }\right|\right)\right)$

Let $M_1={\sum }_{j=0}^N\left({\sum }_{i=1}^M\left(\left|y_i{x}_{ij}^{\prime }\right|+\left|x_{ij}^{\prime }\right|\right)\right)$. Then we have ${\parallel \nabla g\left(\beta \right)\parallel }_1\le M_1$

As we are working on the finite dimensional normed vector space ${\mathbb{R}}^{N+1}$, there then exists some positive $M_2$ such that $\parallel \nabla g\left(\beta \right)\parallel \le M_2$

We can notice that the real $M_2$ does not depend on $\beta$; more precisely it only depends on the training set. So we get

$$\forall \beta \in {\mathbb{R}}^{N+1},\parallel \nabla g\left(\beta \right)\parallel \le M_2$$

(11)

So g is a $M_2$-Lipschitz function on ${\mathbb{R}}^{N+1}$ by the mean value inequality.

In the following, we will prove that the function $h_{{\lambda }_1,{\lambda }_2}$ is Lipschitz.

Let $\beta ,\gamma \in {\mathbb{R}}^{N+1}$. We have

$$\begin{array}{c}|{\lambda }_1\sum _{j=0}^N\left|{\beta }_j\right|-{\lambda }_1\sum _{j=1}^N\left|{\gamma }_j\right||={\lambda }_1\left|\sum _{j=0}^N\right(\left|{\beta }_j\right|\\ -\left|{\gamma }_j\right|\left)\right|\le {\lambda }_1\sum _{j=0}^N\left|\left|{\beta }_j\right|-\left|{\gamma }_j\right|\right|\le {\lambda }_1{\parallel \beta -\gamma \parallel }_1\end{array}$$

Similarly, since the weights $w_{k,l}\in [0,1]$,

$$\begin{array}{cc}& \left|{\lambda }_2\sum _{(k,l)\in E,k<l}{w}_{k,l}\left|{\beta }_k-{\beta }_l\right|-{\lambda }_2\sum _{(k,l)\in E,k<l}{w}_{k,l}\left|{\gamma }_k-{\gamma }_l\right|\right|\hfill \\ & ={\lambda }_2\left|\sum _{(k,l)\in E,k<l}{w}_{k,l}\left(\left|{\beta }_k-{\beta }_l\right|-\left|{\gamma }_k-{\gamma }_l\right|\right)\right|\hfill \\ & \le {\lambda }_2\sum _{(k,l)\in E,k<l}\left|({\beta }_k-{\beta }_l)-({\gamma }_k-{\gamma }_l)\right|\hfill \\ & \le {\lambda }_2\frac{N(N-1)}{2}{\parallel \beta -\gamma \parallel }_1\hfill \end{array}$$

Thereby, $\forall \beta ,\gamma \in {\mathbb{R}}^{N+1},|{\lambda }_1{\sum }_{j=0}^N\left|{\beta }_j\right|-{\lambda }_1{\sum }_{j=0}^N\left|{\gamma }_j\right||\le {\lambda }_1{\parallel \beta -\gamma \parallel }_1$

and

$$\begin{array}{c}\forall \beta ,\gamma \in {\mathbb{R}}^{N+1},\hfill \\ \multicolumn{1}{c}{|{\lambda }_2\sum _{(k,l)\in E,k<l}{w}_{k,l}\left|{\beta }_k-{\beta }_l\right|}\\ -{\lambda }_2\sum _{(k,l)\in E,k<l}{w}_{k,l}\left|{\gamma }_k-{\gamma }_l\right||\le {\lambda }_2\frac{N(N-1)}{2}{\parallel \beta -\gamma \parallel }_1\hfill \end{array}$$

By norm equivalence, we can deduce that there are some positive $M_3$ and $M_4$ such that

$$\forall \beta ,\gamma \in {\mathbb{R}}^{N+1},|{\lambda }_1\sum _{j=0}^N\left|{\beta }_j\right|-{\lambda }_1\sum _{j=0}^N\left|{\gamma }_j\right||\le M_3\parallel \beta -\gamma \parallel$$

(12)

$$\begin{array}{c}\forall \beta ,\gamma \in {\mathbb{R}}^{N+1},|{\lambda }_2\sum _{(k,l)\in E,k<l}{w}_{k,l}\left|{\beta }_k-{\beta }_l\right|\\ -{\lambda }_2\sum _{(k,l)\in E,k<l}{w}_{k,l}\left|{\gamma }_k-{\gamma }_l\right||\le M_4|\beta -\gamma \parallel \end{array}$$

(13)

It follows from (11), (12) and (13) that

$$\begin{array}{c}\forall \beta ,\gamma \in {\mathbb{R}}^{N+1},|h_{{\lambda }_1,{\lambda }_2}\left(\beta \right)\\ -h_{{\lambda }_1,{\lambda }_2}\left(\gamma \right)|⩽(M_2+M_3+M_4)\parallel \beta -\gamma \parallel \end{array}$$

Finally, it suffices to take $\mathcal{l}:=M_2+M_3+M_4$ to deduce that $h_{{\lambda }_1,{\lambda }_2}$ is ℓ-lipschitzian on ${\mathbb{R}}^{N+1}$.

2.3.2. Sub-Gradient Algorithm

After confirming the convergence of our sub-gradient algorithm for estimating $\beta$, let us now detail the method for the objective function $h_{{\lambda }_1,{\lambda }_2}$. The sub-differential sum rule states that $\partial h_{{\lambda }_1,{\lambda }_2}\left(\beta \right)=\partial g\left(\beta \right)+{\lambda }_1\partial \left|\left|\beta \right|1\right|+\lambda 2{\sum }_{(k,l)\in E,k<l}{w}_{k,l}\partial f_{k,l}\left(\beta \right)$. In this equation, $f_{k,l}$ is a function defined in the Example (2), and g is differentiable on ${\mathbb{R}}^{N+1}$; so $\partial g\left(\beta \right)=\nabla g\left(\beta \right)$. The subdifferentials of the norm $L_1$, and $f_{k,l}$ are provided in the Examples 1 and 2, respectively, which leads to the final expression

$$\begin{array}{c}\forall \beta \in {\mathbb{R}}^{N+1},\partial h_{{\lambda }_1,{\lambda }_2}\left(\beta \right)=\nabla g\left(\beta \right)+{\lambda }_1\partial \left|{\parallel \beta \parallel }_1\right|\\ +{\lambda }_2\sum _{(k,l)\in E,k<l}{w}_{k,l}\partial f_{k,l}\left(\beta \right)\end{array}$$

Thus, for a well-chosen sequence $\left({\alpha }_k\right)$, the search algorithm for ${\widehat{\beta }}_{\mathrm{GFLASSO},{\lambda }_1,{\lambda }_2}$ for problem (8) is as follows:

The use of the generalized Fused LASSO penalty in our classification model enables an embedded gene selection process. Genes associated with null coefficients in ${\beta }^{*}$ have no impact on the classification of new examples, as the classification rule in logistic regression is determined by the value of $\frac{1}{1+e^{-x^{\prime }{\beta }^{*}}}$. If this value is less than $0.5$, the example is classified as 0; otherwise, it is classified as 1. This approach fosters a more efficient and effective classification model by eliminating irrelevant genes.

3. Experiments and Results

This section presents the performance of our proposed approach on four well-known gene expression classification datasets (see

Table 2. Description of datasets (DNA microarray) used in GFLASSO-LR.

Dataset	# Genes	# Samples	# Classes	Source
Colon	2000	62	2	[71]
DLBCL	5469	77	2	[72]
Prostate_Tumor	10,509	102	2	[72]
Prostate	12,600	102	2	[73]

). The parameter settings and numerical results will be described in the following subsections. The implementation of our proposed approach was conducted using MATLAB R2018a, and the datasets were divided into training and test sets to assess the performance of gene subset selection.

3.1. Datasets

To evaluate our approach, we selected DNA microarray datasets related to cancer recognition, which are publicly available. These datasets have been utilized in multiple supervised classification studies (as shown in Table 2). They exhibit a variety of characteristics: some have a low number of samples, while others have a higher number. Additionally, all datasets have binary classes. Since our proposed method is designed for large-scale microarrays, all datasets consist of a high number of genes, ranging from 2000 to 12,600.

3.2. Settings

In our approach, we begin with a pre-processing step, which involves selecting the top 200 highest-ranked genes using the absolute correlation coefficient. Subsequently, we randomly split our dataset into a training set and a test set. Using the training set, we then construct our objective function (8). To minimize this function, we employ the sub-gradient algorithm (Algorithm 1). It is important to note that the performance of this algorithm is highly dependent on the choice of step size $\left({\alpha }_k\right)$. A well-chosen $\left({\alpha }_k\right)$ can significantly impact both the convergence rate and the quality of the estimate obtained.

Algorithm 1: Subgradient algorithm to minimize $h_{{\lambda }_1,{\lambda }_2}$.

Function: $GFLASSO-LR(X,i{t}_{max},{\lambda }_1,{\lambda }_2,r_0,\left({\alpha }_k\right))$ Inputs: X: Training set, $i{t}_{max}$: the maximum number of iterations, ${\lambda }_1$ and ${\lambda }_2$: the coefficients that control the penalties, $r_0$: the threshold for building an edge in the graph $\mathcal{G}=(V,E)$, $\left({\alpha }_k\right)$: the sequence that controls the steps in the subgradient algorithm. Output: ${\beta }^{*}$: an approximate value of ${\widehat{\beta }}_{\mathrm{GFLASSO},{\lambda }_1,{\lambda }_2}$. Construct the graph $\mathcal{G}=(V,E)$ and calculate the weights $w_{k,l}$ k = 1; Choose ${\beta }^1$ randomly from ${\mathbb{R}}^{N+1}$ ${\beta }^{*}={\beta }^1$, $h_{{\lambda }_1,{\lambda }_2}^{*}=h_{{\lambda }_1,{\lambda }_2}\left({\beta }^1\right)$ $k=2$  while$k<i{t}_{max}$do   ${\beta }^k={\beta }^{k-1}-{\alpha }_k\ast v_{k-1}$ with $v_{k-1}$ a randomly chosen vector of $\partial h_{{\lambda }_1,{\lambda }_2}\left({\beta }^{k-1}\right)$ if $h_{{\lambda }_1,{\lambda }_2}\left({\beta }^k\right)\le h_{{\lambda }_1,{\lambda }_2}^{*}$ then ${\beta }^{*}={\beta }^k$, $h_{{\lambda }_1,{\lambda }_2}^{*}=h_{{\lambda }_1,{\lambda }_2}\left({\beta }^k\right)$. $k=k+1$. Return ${\beta }^{*}$.

3.2.1. Choice of $\left({\alpha }_k\right)$

The choice of an appropriate $\left({\alpha }_k\right)$ step for the sub-gradient algorithm is an essential step in our study. To begin, we set ${\lambda }_1=1$, ${\lambda }_2=0.2$, and $r_0=0.7$. Then, the “Prostate_Tumor” dataset is utilized to conduct a parametric study.

We start by selecting the 200 best-ranked genes and split the resulting dataset into two disjoint sets: 80% for training and 20% for testing. Our objective function is constructed based on the training set. The goal of this subsection is to choose a step size $\left({\alpha }_k\right)$ that accelerates the convergence of our algorithm to the global minimum. At this stage, we are not concerned with classifier testing, as our focus is solely on observing the behavior of the sub-gradient algorithm in finding the minimum of $h_{{\lambda }_1,{\lambda }_2}$.

We initially compared the classic step ${\alpha }_k=\frac{1}{k}$ to the step proposed by Alber et al. in [74], ${\alpha }_k^{\prime }=\frac{1}{k}\ast \frac{1}{max(1,|v_{k-1}\left|\right)}$, where $v_{k-1}$ is a randomly chosen vector of $\partial h_{{\lambda }_1,{\lambda }_2}\left({\beta }^{k-1}\right)$ (as outlined in Algorithm 1). We set a maximum number of iterations at 10,000. It is evident that these steps ensure the convergence of our algorithm from the sub-gradient to the global minimum.

From the results depicted in Figure 2, it is evident that using ${\alpha }_k^{\prime }$ accelerates the convergence of the sub-gradient algorithm. This acceleration is attributed to the inclusion of the term $v_{k-1}$ in ${\alpha }_k^{\prime }$.

To determine the optimal step size, we drew inspiration from the work of Alber et al. in [74] and utilized steps of the form ${\alpha }_k^{\prime }={\alpha }_k\ast \frac{1}{max(1,|v_{k-1}\left|\right)}$, where $v_{k-1}$ represents a randomly chosen vector of $\partial h_{{\lambda }_1,{\lambda }_2}\left({\beta }^{k-1}\right)$ and $\left({\alpha }_k\right)$ denotes a sequence of positive terms satisfying

$$\sum _{k=1}^{\infty }{\alpha }_k=\infty \mathrm{and}\underset{k\to \infty }{lim}{\alpha }_k=0$$

The final steps ensure the convergence of the subgradient algorithm. In the following, we compare different forms of the step $\left({\alpha }_k\right)$, specifically those of the form $\frac{1}{k^{\gamma }}$ with $\gamma \in [0.05,1]$. Figure 3 illustrates how the objective function changes with the number of iterations for different values of $\gamma$. To enhance the interpretability of the data, we have plotted it on a logarithmic scale. Additionally,

Table 3. Comparison between ${\alpha }_k^{\prime }$ steps with different $\gamma$ for the minimization of $h_{{\lambda }_1,{\lambda }_2}$.

$\gamma$	0.05	0.15	0.25	0.35	0.45	0.55	0.65	0.75	0.85	0.95	1
The optimal value$h_{{\lambda }_1,{\lambda }_2}^{*}$	12.49	8.61	7.38	7.13	7.04	7.01	7.01	7.15	8.02	15.42	19.72

presents the optimal values of the objective function $h_{{\lambda }_1,{\lambda }_2}$ obtained using the subgradient algorithm with different steps ${\alpha }_k^{\prime }$ for various values of $\gamma$.

In this regard, Figure 3 and Table 3 demonstrate that the step size ${\alpha }_k^{\prime }=\frac{1}{k^{0.65}}\times \frac{1}{max(1,\parallel v_{k-1}\parallel )}$ leads to faster convergence compared to other step sizes, yielding superior results.

3.2.2. Choice of ${\lambda }_1$ and ${\lambda }_2$

After determining the optimal step size $\left({\alpha }_k^{\prime }\right)$, we proceed to select the best values for the parameters ${\lambda }_1$ and ${\lambda }_2$. We compare different combinations of these parameters by analyzing the gene set selected by minimizing $h_{{\lambda }_1,{\lambda }_2}$, aiming to find values of ${\lambda }_1$ and ${\lambda }_2$ that achieve the highest classification rate on the test set while minimizing the number of genes used in the classification.

The experiments were conducted using the “Prostate_tumor” dataset, and the accuracy rates were determined using a 5-fold cross-validation method. Initially setting ${\lambda }_2=0.2$, and by varying ${\lambda }_1$, the results are shown in

Table 4. Comparison between the ${\lambda }_1$ for the “Prostate_Tumor” dataset (aggregated values across the 5-fold Cross-validation with ${\lambda }_2=0.2$).

${\mathit{\lambda }}_1$	Accuracy (Test)	Number of Genes
0.2	96.14	55.6
0.4	96.1	36.8
0.6	96.14	32.6
0.8	96.14	29
1	96.14	24
1.2	97.1	28
1.4	96.14	26.2
1.6	97.1	25.4
1.8	96.14	25.6
2	96.14	22.6
2.2	96.14	24.6
2.4	96.14	23.8
2.6	95.14	22.2
2.8	96.14	20.4
3	95.14	20.8
3.2	94.19	20.4
3.4	94.19	20.4
3.6	94.19	20.4
3.8	95.14	19.6
4	94.19	19.2

for different values of ${\lambda }_1$ (with ${\lambda }_2=0.2$).

Typically, we can observe that when ${\lambda }_1=1.6$, the results are better. Specifically, this value of ${\lambda }_1$ results in high accuracy on test data ($97.1\%$), while only utilizing 25 genes. However, as we increase ${\lambda }_1$, the number of selected genes decreases, which is a result of the $L_1$ penalty. After determining that ${\lambda }_1=1.6$ is optimal, we proceed to determine the best value for ${\lambda }_2$.

Table 5. Comparison between the ${\lambda }_2$ for the “Prostate_Tumor” dataset (5 fold Cross-validation with ${\lambda }_1=1.6$).

${\mathit{\lambda }}_2$	Accuracy (Test)	Number of Genes
0	94.19	21.6
0.025	96.14	20
0.05	96.14	21.2
0.075	96.14	22.4
0.1	97.1	22
0.125	97.1	23.8
0.15	97.1	24
0.175	96.14	25
0.2	97.1	25.4
0.225	95.14	25.6
0.25	97.1	26.6
0.275	95.14	28.2
0.3	95.19	27.4
0.325	95.14	28.4
0.35	95.14	28.6
0.375	95.19	27.2
0.4	96.14	27.2
0.425	96.14	28.4
0.45	96.14	25.8
0.475	96.14	28.6
0.5	95.19	28.2
0.525	95.1	29.6
0.55	97.1	27
0.575	95.14	30.8
0.6	96.19	29.6
0.625	95.19	28.2
0.65	95.19	32.2
0.675	96.1	31.2
0.7	94.14	26.8
0.725	96.14	29.8
0.75	95.19	29.2

shows the results obtained for different values of ${\lambda }_2$, with ${\lambda }_1$ fixed at $1.6$. Specifically, it is evident that when ${\lambda }_2=0$, the method employed corresponds to the classic LASSO approach. However, setting ${\lambda }_2=0.1$ yields improved performance, achieving a high accuracy of 97.1% on the test data while utilizing only 22 genes. Consequently, based on these findings, the optimal values of ${\lambda }_1=1.6$ and ${\lambda }_2=0.1$ were selected for the final model.

3.3. Results and Comparisons

Firstly, to limit the search space and accelerate the convergence speed of our proposed approach, we selected the initial subset of genes based on the Pearson correlation filter. Subsequently, the GFLASSO-LR algorithm was applied to determine the optimal gene subset. The quality of this subset was assessed based on the accuracy of the test set and the number of genes selected. Due to the non-deterministic nature of our proposed method, each dataset was randomly divided into a training set comprising 70% of the original dataset and a test set comprising 30%. This process was repeated in 50 independent runs for each dataset to ensure reliable results.

Table 6. The results obtained over 50 executions, without and with selection via the LASSO and GFLASSO penalties.

Dataset	Performance	Without Selection		Pearson Filter		LASSO-LR	GFLASSO-LR
		1NN	SVM	1NN	SVM
Colon	Accuracy (%)	72.33	80.56	80.66	82.55	83.56	83.67
	Number of genes	2000		200		15.4	19.9
DLBCL	Accuracy (%)	81.04	95.65	95.47	97.91	90.78	90.87
	Number of genes	5469		200		21.24	20.4
Prostate_Tumor	Accuracy (%)	80	91.27	90.86	94.6	93.73	94.6
	Number of genes	10,509		200		20.66	21.34
Prostate	Accuracy (%)	80	91.27	91.73	94.26	93.8	95.27
	Number of genes	12,600		200		20.64	20.94

Accuracy: The average classification accuracy (70% training and 30% testing). Genes: The number of genes used in the classification. SVM: The support vector machine classifier using a linear kernel. LR: The logistic regression classifier with LASSO and GFLASSO regularization, respectively. 1NN: the $1-$ nearest neighbor classifier.

summarizes the outcomes of our study across four microarray datasets: Colon, DLBCL, Prostate_Tumor, and Prostate. These results stem from experiments employing a range of gene selection techniques to assess their impact on classification accuracy, thereby validating and demonstrating the effectiveness of our proposed method. Specifically, the table compares performance metrics for two classifiers, KNN and SVM, both without selection and with the application of the Pearson filter. Additionally, it includes outcomes using logistic regression enhanced by LASSO and GFLASSO penalties, offering a comprehensive view of the method’s efficacy in gene selection for cancer classification.

Our embedded methods (LASSO-LR and GFLASSO-LR) were implemented based on the following parameters:

• ${\lambda }_1=1.6,{\lambda }_2=0$, which corresponds to a LASSO-LR type penalty.
• ${\lambda }_1=1.6,{\lambda }_2=0.1$, which corresponds to a generalized Fused LASSO (GFLASSO-LR) type penalty.

The first column of the table displays the datasets used. The second column shows the performance measures, including accuracy (in percent) and the number of selected genes. The third and fourth columns show the results obtained using the 1NN and SVM classifiers, respectively, without gene selection. The fifth and sixth columns present the results obtained using the SVM and 1NN classifiers after having selected 200 genes via the Pearson correlation filter. The seventh and eighth columns present the results of experiments using our LASSO-LR and GFLASSO-LR approaches, respectively.

Figure 4, Figure 5 and Figure 6 show the average classification accuracy obtained from 50 experiments for different methods. According to these figures, the GFLASSO-LR approach shows good performance on the majority of datasets. More precisely, the highest average accuracy rates are obtained for the Colon, Prostate Tumor, and Prostate datasets.

From Table 6 and Figure 4, it is noteworthy that the GFLASSO-LR method outperforms the traditional LASSO-LR method in terms of accuracy for all datasets. This improvement is particularly significant for the datasets Prostate_Tumor and Prostate, which consist of a small number of genes ranging from 19 to 21. Additionally, both GFLASSO-LR and LASSO-LR methods achieve high classification accuracy (greater than or equal to 83.57%) using a small number of genes compared to the original number.

The effectiveness of our selection methods was compared to classifiers (1NN and SVM) both with and without selection, employing the Pearson filter. Figure 5 and Table 6 demonstrate that our GFLASSO-LR and LASSO-LR approaches outperform the 1NN classifier in both scenarios. Specifically, our methods succeed in enhancing accuracy for the majority of datasets. For instance, in the Colon dataset, an accuracy greater than or equal to 83.56% was achieved using less than 20 genes, indicating a significant improvement in classification accuracy (ranging from 3 to 11% difference) with fewer genes. The most notable improvement was observed for the Prostate dataset, where an accuracy of 94.6% was attained (with a 4–14% difference) using less than 21 genes.

Similarly, Figure 6 and Table 6 demonstrate the effectiveness of our gene selection approaches against the SVM classifier. Indeed, our methods show significant improvements in accuracy for the colon and prostate datasets, while using just fewer than 22 genes.

Afterward, we conducted a Kruskal–Wallis test to analyze the significance of the differences in accuracy obtained by our proposed GFLASSO-LR classifier compared to SVM and 1NN classifiers (without selection). The test revealed a statistically significant difference in performance among the classifiers (H-statistic $=6.55$, p-value $=0.038$), emphasizing the impact of classifier choice on classification accuracy.

The boxplot (Figure 7) illustrates the performance distribution of GFLASSO-LR compared to the other classifiers across various datasets. GFLASSO-LR consistently achieves higher classification accuracies, particularly noticeable in the Colon and Prostate_Tumor datasets. This highlights the effectiveness of incorporating GFLASSO regularization techniques in logistic regression, enhancing classification performance in specific contexts. These findings emphasize the importance of selecting the appropriate classifier to achieve optimal classification outcomes for individual datasets.

Overall, the results presented in the table show that the GFLASSO-LR algorithm can improve classification accuracy and decrease the number of selected genes. The proposed methods have important applications in the field of genomics and can advance the understanding of gene function and disease diagnosis. Based on the experiments we conducted, we can conclude that our gene selection approaches are well founded. Our methods achieved high classification accuracy in the datasets used in the study.

Comparison with Other Approaches

In this subsection, we compare our GFLASSO-LR method with algorithms referenced in the literature: [31,32,40,56,75,76]. All of these papers propose integrated methods based on different types of penalties and apply them to the gene selection problem. To make the comparison meaningful, experiments are performed under the same conditions for each algorithm. Our approach is run 50 times on each dataset, with the dataset randomly divided into two parts (70% for training and 30% for testing) in each run. The average accuracy on the test set and the average number of genes used in classification are chosen. It is noted that the authors of [31] performed 100 runs.

Table 7. Comparison between our method (GFLASSO-LR) and state-of-the-art methods.

Dataset	Performences	GFLASSO-LR (Our Method)	BLASSO [75]	LASSO [56]	SCAD [76]	ALASSO [40]	Nouvel ALASSO [31]	CBPLR [32]
Colon	Acccuracy (%)	83.67 < 2 >	93	79.53	79.51	78.4	82.91	89.5
	Number of genes	19.9	11	14	14	14	12	10
DLBCL	Acccuracy (%)	90.87 < 3 >	84	88.33	74.02	84.41	91.32	91.7
	Number of genes	20.4	17	24	24	24	22	17
Prostate	Acccuracy (%)	95.27 < 1 >	-	91.14	60.13	82.11	93.53	93.6
	Number of genes	20.94	-	29	28	28	24	16

< · >: Indicates the rank of our method among other algorithms in terms of average accuracy for each dataset. BLASSO = Bayesian Lasso quantile regression. LASSO = The classic LASSO method. SCAD = smoothly clipped absolute deviation. ALASSO = The classical Adaptive LASSO method. New ALASSO = A new Adaptive LASSO method based on the weighted Mahalanobis distance. CBPLR = A new method of Adaptive LASSO based on the correlation between genes.

summarizes the classification accuracy and the number of selected genes (from the original papers) for each approach. A dash (-) indicates that the result is not reported in the corresponding work. The results obtained by our approach are competitive with previous works.

For the Colon dataset, we achieve an accuracy rate of 83.67% using only 19 genes. Our method ranks second, outperforming the other five approaches, with the best performance achieved by the BLASSO approach (93%). For the DLBLC dataset, we obtain the third-best performance (90.87%) after the “New ALASSO” and “CBPLR” approaches, with a difference of less than 1%. For the “Prostate” dataset, we achieved the best performance (accuracy of 95.27%) followed by the “CBPLR” method with an accuracy of 93.6%. The number of genes selected by GFLASSO-LR is relatively similar to that of other methods, except for the Prostate dataset, which uses slightly more genes.

The results of this comparative analysis with previous embedded gene selection methods in cancer classification indicate that our improved generalized fused LASSO penalization approach is effective in the gene selection problem.

3.4. Time and Memory Complexity

3.4.1. Pre-Processing Step (Selecting the Top 200 Highest-Ranked Genes)

• Time Complexity: Computing correlation coefficients for each gene across all samples and selecting the top 200 genes: $O(N\times M)$ (as provided in the information).
• Memory Complexity: Storing microarray data and correlation coefficients: $O(N\times M)$.

3.4.2. Splitting-GFLASSO-LR Algorithm

• Time Complexity: $O\left(M\right)$ for randomly splitting the dataset into training and test sets.
• Memory Complexity: $O\left(M\right)$ for storing the training and test sets.

3.4.3. Building the Objective Function Based on the Training Data and the 200 Genes

Regarding the time complexity, in the worst case, where each gene is correlated with every other gene, the number of edges in the graph would be $\frac{200\times (200-1)}{2}$. Additionally, computing the absolute correlation values between each pair of genes is estimated using $O\left(\frac{200\times (200-1)}{2}\times M\right)$.

3.4.4. Sub-Gradient Algorithm

Time Complexity: The dominant term in the time complexity is represented as follows.

$$O\left(\frac{200\times (200-1)}{2}\times M+i{t}_{max}\times \frac{{200}^3}{2}\right)$$

(14)

Typically, the components and computational complexity of an optimization process can be summarized as follows:

• Constructing the graph and calculating weights: negligible compared to the overall complexity.
• Combining the subdifferentials: Once the subdifferentials of each component function are computed, they are combined according to the given expression. This involves a combination of nearly $\frac{{200}^2}{2}$ subdifferentials, resulting in a time complexity of $O\left(\frac{{200}^3}{2}\right)$.
• Iterating over the sub-gradient algorithm: $O\left(i{t}_{max}\right)$ iterations.
• Each iteration involves updating the coefficients using a sub-gradient: $O\left(\frac{{200}^3}{2}\right)$ operations.

Memory Complexity: Storing the graph structure and weights: $O\left(\frac{200\times (200-1)}{2}\right)$ in the worst case. Storing intermediate results and coefficients: $O\left(200\right)$ (since only one set of coefficients needs to be stored).

Overall, the analysis outlines a computationally intensive process with significant emphasis on handling gene interaction data efficiently. While time complexity primarily concerns the combinatorial interactions among genes and the iterative optimization process, memory complexity focuses on efficiently storing gene data, correlations, and algorithmic intermediates. The optimization aims to identify a subset of genes critical for further analysis, with considerations for computational and storage efficiencies.

3.5. Discussion

The discussion on the performance of the GFLASSO-LR approach compared to traditional and recent methods in gene selection for cancer classification underscores its practical utility and effectiveness. Our method’s superior or competitive performance across different datasets, particularly in achieving high classification accuracy with a relatively small number of genes, highlights its potential for advancing personalized medicine and genomic research. The ability to identify the most relevant genes from massive datasets with precision is crucial for understanding complex diseases like cancer. Our GFLASSO-LR approach not only simplifies the gene selection process but also ensures that the selected genes are highly indicative of the condition being studied, thus facilitating more accurate diagnoses and the development of targeted therapies.

Moreover, the consistency of our method’s performance across various datasets demonstrates its robustness and adaptability to different types of cancer, which is a significant advantage in the rapidly evolving field of genomics. The use of logistic regression with the GFLASSO penalty enhances the model’s ability to deal with the high dimensionality and multicollinearity inherent in microarray data, thereby overcoming some of the common challenges faced in genomic data analysis. However, it is worth noting that while our approach shows promise, further research is necessary to explore its full potential and limitations. Future studies could focus on refining the algorithm to improve efficiency and accuracy, as well as testing it on a wider range of datasets. Additionally, integrating our method with other data types, such as clinical and phenotypic information, could provide a more comprehensive understanding of the gene–disease relationships and further enhance its application in personalized medicine.

4. Conclusions

High-dimensional classification problems in microarray dataset analysis are a crucial area of research in cancer classification. In this paper, we propose and apply an improved method, GFLASSO-LR, for simultaneous gene coefficient estimation and selection to improve classification performance. We also demonstrate the convergence of the sub-gradient algorithm to solve the associated non-differential convex optimization problem.

The proposed method was evaluated based on the number of selected genes and classification accuracy on four sets of high-dimensional cancer classification data. The results consistently showed that GFLASSO-LR can significantly reduce the number of relevant genes and has superior accuracy compared to the classical LASSO method. Overall, the results demonstrate that GFLASSO is a promising method for accurately analyzing high-dimensional microarray data in cancer classification. The method can be applied to other types of high-dimensional classification data related to the medical domain. Future research could extend the present work to cover high-dimensional multiclass cancer data and focus on very high-dimensional microarray data for cancer classification.