Dynamic Online Label Distribution Feature Selection Based on Label Importance and Label Correlation

Abstract

Existing feature selection methods mainly target single-label learning and multi-label learning, and only a few algorithms are optimized for label distribution learning. In label distribution learning, the associated labels of each sample have different levels of importance. Therefore, multi-label feature selection algorithms cannot be directly applied to label distribution learning. Discretizing label distribution data into multi-label data will cause part of the supervision information to be lost. In most practical applications of label distribution learning, the feature space is undefined, and the features are in the form of flow features. To solve this problem, this paper applies fuzzy rough set theory and applies the flow feature framework to propose a dynamic label distribution feature selection algorithm that handles flow features. Experimental results show that the proposed method is more effective than six state-of-the-art feature selection algorithms on 12 datasets with respect to six representative evaluation metrics.

1. Introduction

In the field of machine learning and data mining [1], the problem of label ambiguity is currently a popular research topic. There are currently two relatively mature machine learning paradigms, namely, single-label learning (SLL) and multi-label learning (MLL) [2]. SLL achieves good results in the field of machine learning when the target instance has a clear single-class label. MLL is an extension of SLL. In real life, an object may be associated with several class labels. To intuitively reflect the characteristics of polysemous objects, the most obvious approach is to assign multiple category labels to each example of a polysemous object, that is, to use MLL [3,4].

Because it is constrained by label semantics, traditional multi-label classification assumes that relevant and irrelevant labels have the same degree of relevance to the description of the example, which obviously does not meet the complex cognitive needs of humans [5]. In layman’s terms, MLL only solves the label ambiguity problem of labels associated with the instance. However, in real life, we need to solve the more common and ambiguous scenario of “how to describe the instance”. For example, in facial-expression emotion recognition, it is more meaningful to understand the degree to which each emotion describes the instance. To solve these problems, a new learning paradigm has been proposed in recent years—label distribution learning (LDL) [6], which is an extension of MLL. The degree of LDL to which each label describes an instance is represented by the corresponding value of the label distribution, called the descriptive degree. This value explicitly indicates the relative importance of the label. Both SLL and MLL can be regarded as special cases of LDL. Figure 1 shows the three learning paradigms within the LDL framework.

Figure 2 shows an image described using a label distribution. The label descriptions of “HAP”, “SAD”, “SUR”, “ANG”, “DIS”, and “FER” in the image are shown in Figure 2. LDL has richer label information and clearly describes the importance of each label [7]. At present, it has been applied in multiple neighborhoods. For example, the adaptive facial recognition proposed by X. Geng et al., achieved good results by estimating the age of sample characters through LDL [8]. Z. Zhang et al., by collecting surveillance video data, used LDL to implement crowd counting applications over a period of time [9], and P. H. et al., used LDL to predict box office before movie release by collecting people’s opinions on movies [10]. LDL has also been widely used in fields such as emotion recognition and multilingual learning.

Similar to traditional SLL and MLL, LDL also faces many challenges. In terms of the data structure of label instances, the problems faced by LDL tasks include a large number of feature dimensions [11], large number of labels [12], label imbalance [13], and streaming features [14]. In LDL tasks, the dimensionality of the data is very large, with usually thousands or tens of thousands of dimensions [15,16]. This “curse of dimensionality” leads to a series of problems such as reduced classification accuracy and generalization ability in high-dimensional space, and it increases the computational cost of many learning algorithms. In addition, too many features may contain redundant or irrelevant features, which may lead to more problems such as the excessive consumption of computing resources and degradation of model performance. To this end, many dimensionality reduction methods have been proposed [17,18]. However, they are mainly designed for SLL and MLL algorithms, and are rarely suitable for LDL algorithms. Therefore, there is an urgent need to develop new and effective LDL dimensionality reduction methods.

There are two main methods for reducing the dimensionality of label distribution: feature extraction and feature selection. Feature extraction methods use space mapping techniques or space transformations to reduce the dimensionality of the feature space. However, these methods destroy the structural information of the original feature space, blur the physical meaning of the features, and lack semantic interpretation. In contrast to feature extraction methods, feature selection methods do not perform any feature space transformation or mapping; instead, they retain the original spatial structure. Feature selection methods sort the features in the original feature space by importance, select the subspace that best represents the semantic features of that space, and use this subspace to maximally represent the original feature space [19]. Therefore, feature selection methods well preserve the physical meaning of the feature space, which is an advantage of feature extraction methods [20]. From the perspective of interaction with learning algorithms, existing MLL feature selection algorithms are mainly divided into three types: filtering, wrapping, and embedding [21]. All types have different relative advantages. Filtering methods are typically more efficient, less computationally expensive, and more general than embedding and wrapping methods; therefore, this work focuses on filtering methods.

In the past few decades, many feature selection algorithms have been proposed for label distribution learning. In related studies [22,23], these methods were used to consider the correlation between features and labels based on fuzzy rough sets. In addition, because of the correlation between labels, the performance of label distribution models depends largely on label correlation. For example, the method in [24] considers label correlation to reduce data dimensionality, that is, it leverages the assumption that all samples share the same label correlation. However, for different sample groups in practical applications, the corresponding label correlation between random variables is an effective indicator for evaluating the discriminative ability of candidate features in the feature selection process. Alternatively, when dealing with problems related to uncertainty, rough set theory [25,26] naturally has many advantages and has been widely used in SLL or MLL feature selection algorithms [27,28]. Chen et al. [29] designed a parallel feature selection method based on neighborhood rough sets (NRSs), which considers the partial order between features and values. Yuan et al. [30] proposed a generalized unsupervised feature selection model based on fuzzy rough sets. However, to the best of our knowledge, there are relatively few feature selection methods based on rough set theory for label distribution. The main problem faced by rough sets in dealing with label distribution feature selection is how to deal with distributed labels.

In most practical applications of LDL, the feature space is usually uncertain, and the number of samples for each feature increases gradually over time, just like a stream of feature vectors. Such features are called streaming features. For example, on the social networking platform Twitter, hot topics change dynamically over time. When a hot topic appears, it is always accompanied by a set of new keywords. These fresh keywords can be used as key features to distinguish hot topics. Streaming feature selection assumes that features arrive dynamically over time [31] and performs feature selection when each feature arrives to always maintain the optimal subset of features [32,33]. So far, there have been many methods for processing streaming features. For example, Chen et al. [34] proposed using global features to process streaming feature data, theoretically analyzed the pairwise correlation between features in the currently selected feature subset, and solved the streaming feature problem using online pairwise comparison techniques. NRS can handle mixed types of data without destroying the neighborhood and sequential structure of the data [35]. Moreover, feature selection based on NRS does not require any prior knowledge of the feature space structure, and hence it is an ideal tool for online stream feature selection.

This paper proposes a dynamic online label distribution feature selection model based on label correlation and NRS, and proposes a dynamic label distribution feature selection algorithm for processing stream features. Mutual information is widely used to measure the degree of dependence between random variables and is an effective indicator for evaluating the discriminative ability of candidate features in the feature selection process. The proposed method uses the mutual information method to process the label space and obtain the correlation between labels through mutual information and graph characteristics. In addition, a new label space neighborhood relationship is proposed, in which the neighborhood class of the instance is constructed in the label space, replacing the calculation of the traditional logical label equivalence class. Simultaneously, the nearest neighbor distribution of the surrounding instances is used to calculate the neighborhood of the instance in the feature space, avoiding the problem of neighborhood granularity selection faced by the traditional NRS model. On this basis, the NRS model is extended to fit the label distribution data, and the corresponding feature dependency is redefined. Combining label correlation and NRS, a dynamic feature selection algorithm framework is proposed. The main contributions of this paper are as follows:

(1)

The average nearest neighbor method is used to calculate a new form of neighborhood granularity, and the mutual information method is used to calculate the label weight to obtain the label correlation. By combining the neighborhood granularity and label correlation, a new NRS relationship and feature importance model is constructed.

(2)

The traditional NRS model is generalized to adapt it to LDL.

(3)

Using the above model, a new label distribution stream feature selection algorithm is proposed that combines the new NRS model with the stream feature online importance update framework. As a result, it can better handle the label distribution stream feature problem.

The rest of this paper is organized as follows. In Section 2, we introduce the related concepts, including LDL, feature correlation, and NRS theory. Section 3 introduces label correlation and the NRS models. In Section 4, we propose a dynamic feature selection algorithm based on label correlation and NRS. We report our experimental results in Section 5. Finally, Section 6 summarizes this paper and discusses future work.

2. Related Work

2.1. LDL

At present, the mainstream LDL algorithms are mainly divided into three categories: The first is the problem transformation method, which converts the label distribution data into weighted single-label data or multi-label data and processes the data with traditional single-label or multi-label algorithms. For example, Geng proposed two representative problem transformation algorithms based on problem transformation, PT-SVM [36], which respectively transform the LDL problem into a SLL problem based on a support vector machine and Bayes’ theorem. The second category is the algorithm adaptation method, which naturally extends the existing learning model to directly process the label distribution. AA-kNN [37], AA-BP [37], Duo-LDL [38], and LALOTP [39] are four representative methods that use k-nearest neighbors and the BP neural network to solve the LDL problem. By contrast, the third category consists of algorithms specifically proposed to solve the LDL problem. Representative algorithms include the IS-LLD and CPNN algorithms [39], which were applied to the problem of facial age estimation using a label distribution.

2.2. Label Correlation

In the past few years, researchers have proposed various methods to improve the recognition rate of annotation labels by classifiers. By incorporating label correlation, a large number of studies have convincingly demonstrated that exploiting label correlation can significantly improve classifier performance. For example, in the Movie dataset, we often observe that “science fiction” movies tend to be associated with the “adventure” label, while some movies also have the “humor” label. This pattern of co-occurrence between “science fiction” and “adventure” signals the global label correlation between them. In addition, the frequent co-occurrence of “adventure” and “humor” in some movies also indicates the local correlation between them. A deeper understanding of the relationships among labels can provide guidance for building more accurate multi-label classifiers. The method proposed by Zhu et al. [40] simultaneously handles complete and missing labels, exploiting global and local label correlations by learning latent label representations and optimizing label manifolds. However, these methods may still have a limited ability to handle missing labels.

2.3. NRS

Rough set-based methods are another popular class of feature selection algorithm that can maintain the consistency of the original features using the smallest feature subset size. Few methods extend rough sets to NRS to deal with label distribution. The primary difficulty is the selection of the neighborhood granularity. Wang et al. [23] combined information theory with fuzzy rough sets to construct a label distribution feature selection method with maximum relevance and minimum redundancy. Qian et al. [41] constructed a label enhancement algorithm using probability and rough set models, and designed the MFS algorithm based on the degree of feature dependency.

3. Preliminaries

In Section 3, we will discuss some basic definitions of LDL and NRS. Based on the theory of NRS, we will propose a neighborhood rough set decision system by dividing the sample set’s neighborhood and constructing a lower approximation to achieve this system.

3.1. LDL

3.1.1. LDL Framework

We define $X=R^N$ to represent the N-dimensional sample space, that is, $U=\left\{X_1,X_2,X_i\dots ,X_n\right\}$, $L=\left\{l_1,l_2,\dots ,l_m\right\}$ represents the M-dimensional label space. For an instance $x_i\in U$ in the sample space, $x_i=\left\{F_{i1},F_{i2},\dots ,F_{id}\right\}$ represents a d-dimensional feature vector, and we denote label vector $Y=\left\{y_1,y_2,y_j\dots ,y_m\right\}$, where $y_j=\left\{d_{y_j}^{\left(x_1\right)},d_{y_j}^{\left(x_2\right)},d_{y_j}^{\left(x_3\right)},\dots ,d_{y_j}^{\left(x_n\right)}\right\}$ for each instance $x_i$ corresponding to $y_i=d_{y_i}^{\left(x\right)}=1$. According to the definition of LDL, both SLL and MLL can be regarded as more general cases of LDL. The task of LDL is to find a mapping $f:X\to Y$. When $x_i$ contains the label $l_i$, the corresponding value of $y_i$ is 1; otherwise, it is $-1$ [42].

3.1.2. NRS

Assume a decision system $NDT=\langle U,C,D\rangle$, where $U=\left\{x_1,X_2,\dots ,X_n\right\}$ represents a non-empty set of instances (that is, the set composed of all samples), $C=\left\{a_1,\dots ,a_N\right\}$ represents the attribute set corresponding to the sample, and D represents the set of decision attributes.

For a given parameter $\delta$ and feature set C, the $\delta$-domain relationship on X can be determined. We call the decision system a neighborhood decision system, denoted as NDS $=<$$U,C\cup D,\delta >$.

Definition 1. 

Given an N-dimensional real space $\Omega ,\Delta :R^N\times R^N\to R$, we say that Δ is a metric on $R^N$ if Δ satisfies the following constraints [43]:

(1) 

$\Delta \left(x_1,x_2\right)\ge 0$, if and only when $x_1=x_2,\forall x_1,x_2\in R^N$;

(2) 

$\Delta \left(x_1,x_2\right)=\Delta \left(x_2,x_1\right),\forall x_1,x_2\in R^N$;

(3) 

$\Delta \left(x_1,x_3\right)\le \Delta \left(x_1,x_2\right)+\Delta \left(x_2,x_3\right),\forall x_1,x_2,x_3\in R^N$.

Definition 2. 

For $\forall x_i\in U$ and a feature subset $B\subseteq C$, we define the δ-neighborhood of $x_i$ based on parameter C as follows [43]:

$${\delta }_B\left(x_i\right)=\left\{x_j\mid x_j\in U,{\Delta }_B\left(x_i,x_j\right)\le \delta \right\}$$

(1)

$x_i$ and $x_j$ are two different elements in set U, where $\delta >0$, and the set of instances of values is granulated by ${\Delta }_B\left(x_i,x_j\right)$. We call $〈\Omega ,{\Delta }_B〉$ the metric space, and ${\delta }_B\left(x_i\right)$ the δ-neighborhood information particle generated by $x_i$. In terms of the two-dimensional real space, the neighborhoods based on the 1-norm, 2-norm, and infinite norm are shown in Figure 3, which are diamond, circle, and square regions, respectively.

In this manner, we granulate the neighborhood of all objects in the universal space.

From the neighborhood information particle clusters, $\left\{\delta \left(x_i\right)\mid i=1,2,\dots ,n\right\}$ yields a neighborhood relation N on the universal space U. This relation can be represented by a matrix system $M\left(N\right)={\left(r_{ij}\right)}_{n\times m}$: if $x_j\in \delta \left(x_i\right)$, then $r_{ij}=1$; otherwise, $r_{ij}=0$. For neighborhood relations, we have

(1)

$\forall x_i\in U:{\delta }_1\left(x_i\right)\subseteq {\delta }_2\left(x_i\right)$

(2)

$N_1\subseteq N_2$

The neighborhood information particle clusters defined in this manner constitute the basic concept system in the universal space.

Definition 3. 

Given a non-empty finite set $U=\left\{x_1,x_2,\dots ,x_n\right\}$ on the actual space and a neighborhood relation N on U, we call the two-tuple $NAS=\langle U,N\rangle$ a neighborhood approximation space [43].

For example, $U=\left\{x_1,x_2,\dots ,x_n\right\}$, a is an attribute of U, $f=(x,a)$ represents the value of sample x on attribute a.$f\left(x_1,a\right)=1.1,f\left(x_2,a\right)=1.2,f\left(x_3,a\right)=1.6,f\left(x_4,a\right)=1.8,f\left(x_5,a\right)=1.9$. When the specified neighborhood size is 0.2, since $\left|f\left(x_1,a\right)-f\left(x_2,a\right)\right|\le 0.2$, then $x_2\in \delta \left(x_1\right)$, $x_1\in \delta \left(x_2\right)$.

Definition 4.  

For a given decision system $NDT=\langle U,C,D\rangle$ and $X\subseteq N$, the lower and upper approximations of X in the neighborhood approximation space $NAS=<U,N>$ are respectively defined as follows [44]:

(1) 

$\underline{N}X=\left\{x_i\mid \delta \left(x_i\right)\subseteq X,x_i\in U\right\}$

(2) 

$\overline{N}X=\left\{x_i\mid \delta \left(x_i\right)\cap X\ne \varnothing ,x_i\in U\right\}$

where $\underline{N}X$ is also referred to as the positive domain of X in the approximation space $NAS=$$\langle U,N\rangle$ and is the largest union of neighborhood information particles that can be completely contained in X.

Definition 5. 

For a neighborhood decision system $NDT=\langle U,A,D,\delta \rangle$, D partitions U into N equivalence classes: $X_1,X_2,\dots ,X_N.\forall B\subseteq A$, we define the upper and lower approximations of the decision attribute D with respect to B as [45]

$$\underline{N_{B}D}={\mathrm{U}}_{i=1}^N\underline{N_B}{X}_i$$

(2)

$$\overline{N_B}D=\bigcup _{i=1}^N\overline{N_B}{X}_i$$

(3)

respectively [45], where ${\delta }_B\left(x_i\right)$ is the informative neighborhood particle generated by attribute B and metric Δ.

Figure 4 provides a geometric explanation of the classical rough set model, where the equivalence classes marked in red belong entirely to X and are lower approximations of X. Compared with the neighborhood rough set model, these two models are consistent.

The lower approximation of decision attribute D, also called the decision-positive region, is denoted by $POS\left(D\right)$.

The size of the positive region reflects the degree to which the classification problem is separable in a given attribute space, with larger positive regions indicating areas of overlap (i.e., fewer boundaries) for each category. We can describe such classification problems in more detail using the following set of attributes:

$$POS\left(D\right)=\left\{x_i\mid {\delta }_B\left(x_i\right)\subseteq D,x_i\in U\right\}$$

(4)

Definition 6. 

Suppose that $A,B$ are two sets. We define the degree to which A is contained in $B,I(A,B)$, as follows [46]:

$$I(A,B)={\displaystyle \frac{Card(A\cap B)}{Card\left(A\right)}}$$

(5)

When $A=\varnothing$ or $B=\varnothing$, we define $I(A,B)=0$. Here, $I(A,B)$ reflects the importance of B to A.

The dependency of decision attribute D on condition attribute B is defined as follows [47]:

$${\gamma }_B\left(D\right)=Card\left(\underline{N_{B}D}\right)/Card\left(U\right)$$

(6)

where ${\gamma }_B\left(D\right)$ denotes the proportion of samples in the sample set that can be included by a decision according to the description of condition attribute B.

The positive region of the decision is larger if decision attribute D is more dependent on condition attribute B.

4. Proposed Method

4.1. Improvement of NRS

Given a decision system $NDT=\langle U,C,D\rangle$, $U=\left\{x_1,x_2,\dots ,X_n\right\}$ represents a non-empty set of instances, C represents the feature set corresponding to the instance set, and D represents the decision attribute set $D^L=\left\{D^{L_1},D^{L_2},\dots ,D^{L_m}.\right\}$ Traditional single-label and multi-label neighborhood information particle partitioning methods are not suitable for labeled distribution data. For general data, a group of instances with the same attribute value or label value is called an equivalence class. Similarly, for mixed data, a group of instances with similar attribute values or label values is called a neighborhood class. In the method proposed in this paper, the margin of particles in the sample is used for granulating the neighborhood size.

Definition 7. 

Given a sample x, the margin of x relative to a set of samples U is defined as follows [12]:

$$m\left(x\right)=\Delta (x,NS(x\left)\right)-\Delta (x,NT(x\left)\right)$$

(7)

where $NS\left(x\right)$ denotes the instance from U that has the shortest distance from x and whose label class is different from that of x. In addition, $NT\left(x\right)$ denotes the instance from U that has the shortest distance from x and has the same label class as x. We call these instances the nearest miss and the nearest hit, respectively. Moreover, $\Delta (x,NS(x\left)\right)$ denotes the distance between x and $NS\left(x\right)$, and $\Delta (x,NT(x\left)\right)$ denotes the distance between x and $NT\left(x\right)$. We call $\delta \left(x\right)=$$\{y\mid \Delta (x,y)\le m(x\left)\right\}$ the neighborhood particle about x. To facilitate the setting of neighborhood information particles, we set $m\left(x\right)=0$ when $m\left(x\right)<0$.

A sample may have a positive or negative effect on different labels. Thus, for a given sample, the degree of granularity depends on the label used.

Definition 8. 

For a sample x and label $l_k\in L$, the margin of x with respect to $l_k$ is $m_{l_k}\left(x\right)={\Delta }_{l_k}\left(x,N{S}_{l_k}\left(x\right)\right)-{\Delta }_{l_k}\left(x,N{T}_{l_k}\left(x\right)\right),l_k\in L$.

As noted above, each sample has a different label and, correspondingly, a different granularity. Depending on the different decision views, we need to combine all the single-label granularities of a given sample to form a label distribution granularity [44]. Therefore, in this paper, we choose the average granularity (i.e., the average nearest neighborhood, also known as the neutral view) to represent the label distribution granularity of a sample [12]:

$$m^{\mathrm{neu}}\left(x\right)={\displaystyle \frac{1}{L}}\sum _{i=1}^L{m}_{l_k}\left(x\right)$$

(8)

To solve the problem of the granularity selection of δ, combining Equations (1) and (9), the new neighborhood of the sample is defined as

$${\delta }_B\left(x_i\right)=\left\{x_j\mid x_j\in U,{\Delta }_B\left(x_i,x_j\right)\le m^{\mathrm{neu}}\left(x_i\right)\right\}$$

(9)

We define a new neighborhood information particle to solve the problem of selecting the neighborhood granularity, which is caused by label distribution data. In addition, the average nearest neighbor reflects the relationship between features in an instance. This new neighborhood model considers the relationships between features and is based on improved neighborhood information.

Definition 9. 

Given $\forall x_i\in U$, a label space $D^L=\left\{D^{L_1},D^{L_2},\dots ,D^{L_m}.\right\},D^{L_j}\in D^L$, the label space granularity on label $D^{L_j}$ is defined as follows:

$${\Theta }_{D^{L_j}}\left(x_i\right)=\left\{\begin{array}{c}\left\{x_j\mid {\Delta }_{D^{L_j}}\left(x_i,x_j\right)\le {\theta }_{D^{L_j}}\wedge {\Delta }_{D^{L_j}}\left(x_i,x_j\right)\ne 0,x_j\in U\right\},\mathrm{if}{d}_{x_i}^{L_j}\ne 0\\ \left\{x_j\mid {\Delta }_{D^{L_j}}\left(x_i,x_j\right)=0\right\},\mathrm{if}{d}_{x_i}^{L_j}=0\end{array}\right.$$

(10)

where ${\theta }_{D^{L_j}}={\displaystyle \frac{{\sum }_{j=1}^{j=m}{\sum }_{i=1}^{i=n}{d}_{x_i}^{L_j}}{m\times n}}$. Here, ${\Theta }_{D^{L_j}}\left(x_i\right)$ indicates whether the neighborhood is centered on $x_i$, and ${\theta }_{D^{L_j}}$ represents the radius. A larger ${\theta }_{D^{L_j}}$ value indicates the sample is closer to $x_i$. When $d_{x_i}^{L_j}=0$, the neighborhood granularity is equivalent to the equivalence class.

Example 1. 

Continue to

Table 1. Example of multi-label data.

No.	U	${\mathit{f}}_1$	${\mathit{f}}_1$	${\mathit{D}}^{{\mathit{L}}_1}$	${\mathit{D}}^{{\mathit{L}}_2}$	${\mathit{D}}^{{\mathit{L}}_3}$
1	$x_1$	0.89	0.08	0.33	0.67	0
2	$x_2$	0.21	0.48	0.64	0	0.36
3	$x_3$	0.11	0.06	0.79	0	0.21
4	$x_4$	0.43	0.20	1	0	0
5	$x_5$	0.15	0.14	0.82	0.15	0.03
6	$x_6$	0.82	0.31	0	0.48	0.52
7	$x_7$	0.03	0.15	1	0	0
8	$x_8$	0.73	0.09	0	0.6	0.4
9	$x_9$	0.86	0.50	0.41	0	0.59
10	$x_{10}$	0.66	0.78	0	1	0

. Assume 2-norm is applied to ${\Delta }_{D^{L_j}}\left(x_i,x_j\right)$, we take $D^{L_1}$ for example, and then we can compute ${\Theta }_{D^{L_1}}\left(x_i\right)$ as: ${\Theta }_{D^{L_1}}\left(x_1\right)=\left\{x_1,x_2,x_9\right\}$, ${\Theta }_{D^{L_1}}\left(x_2\right)=\left\{x_1,x_2,x_3,x_5,x_9\right\}$, ${\Theta }_{D^{L_1}}\left(x_3\right)=\left\{x_2,x_3,x_4,x_5,x_7\right\}$, ${\Theta }_{D^{L_1}}\left(x_4\right)=\left\{x_3,x_4,x_5,x_7,\right\}$, ${\Theta }_{D^{L_1}}\left(x_5\right)=\left\{x_2,x_3,x_4,x_5,x_7\right\}$, ${\Theta }_{D^{L_1}}\left(x_6\right)=\left\{x_6,x_8,x_{10},\right\}$, ${\Theta }_{D^{L_1}}\left(x_7\right)=\left\{x_3,x_4,x_5,x_7\right\}$,${\Theta }_{D^{L_1}}\left(x_8\right)=\left\{x_6,x_8,x_{10}\right\}$, ${\Theta }_{D^{L_1}}\left(x_9\right)=\left\{x_1,x_2,x_9\right\}$, ${\Theta }_{D^{L_1}}\left(x_{10}\right)=\left\{x_6,x_8,x_{10}\right\}$.

Based on the above definition, we can find that the neighborhood granularity of instance in the label space forms the granularity system that can cover the instance set. Then, we can summarize the following properties:

(1) ${\Theta }_{D^{L_j}}\left(x_i\right)\ne \varnothing ,for{x}_i\notin {\Theta }_{D^{L_j}}\left(x_i\right)$;

(2) ${\bigcup }_{i=1}^n{\Theta }_{D^{L_j}}\left(x_i\right)=U$.

Definition 10. 

In the neighborhood decision system $NDS=<U,C\cup D>$, $U=\left\{X_1,X_2,\dots ,X_n\right\}$ represents the sample space, and $D^L=\left\{D^{L_1},D^{L_2},\dots ,D^{L_m}.\right\}$ represents the label distribution’s feature space. By defining the multi-label decision space, we can expand the decision positive domain of the single-label decision using Equation (4). For a certain feature subset $B\subseteq C$, the lower approximation of the decision attribute $D^j$ with respect to B is as follows:

$$POS{\left(D^j\right)}^{\prime }=\underline{N_B}{D}^{L_j}=\bigcup _{i=1}^n\left\{x_i\mid {\delta }_B\left(x_i\right)\subseteq {\Theta }_{D^{L_j}}\left(x_i\right),x_i\in U\right\}$$

(11)

In this equation, the neighborhood particle ${\delta }_B\left(x_i\right)$ is obtained from Definition 9, and the label particle ${\Theta }_{D^{L_j}}\left(x_i\right)$ is obtained from Definition 10. Through this method, we extend the NRS and solve the problem of selecting the neighborhood granularity of rough sets in labeled distribution learning.

Example 2. 

Continue to Table 1. We take $L_1$ for example, for feature set $B=\left(f_1,f_2\right)$, and then we can compute ${\delta }_B\left(x_i\right)$ according to Definition 9: ${\delta }_B\left(x_1\right)=\left(x_1,x_4,x_8\right)$, ${\delta }_B\left(x_1\right)=\left(x_2,x_3,x_5\right)$; ${\delta }_B\left(x_3\right)=\left(x_3,x_5\right)$, ${\delta }_B\left(x_4\right)=\left(x_4,x_5,x_8\right)$,${\delta }_B\left(x_5\right)=\left(x_1,x_5,x_3\right)$,${\delta }_B\left(x_6\right)=\left(x_6,x_9\right)$; ${\delta }_B\left(x_7\right)=\left(x_7,x_3\right)$, ${\delta }_B\left(x_8\right)=\left(x_8,x_1\right)$, ${\delta }_B\left(x_9\right)=\left(x_9,x_6\right)$,${\delta }_B\left(x_{10}\right)=\left(x_{10},x_9\right)$.

According to Definition 11, using Examples 1 and 2, we can calculate $POS{\left(D^j\right)}^{\prime }=\left(x_2,x_3,x_7\right)$.

In LDL, because the labels of each instance are always related in some way, it is necessary to consider the importance of labels and the correlation between them.

Definition 11. 

For a sample $x_i$ and the corresponding feature vector $Y_i$, that is, $D=$$\left\{\left(x_i,Y_i\right)\mid 1\le i\le N,x_i\in U,Y_i\in L\right\}$, N is the number of instances in the training set, and $l_i,l_j\in$ and $L(1\le i,j\le k)$ are any two labels in label space L. The correlation between $l_i$ and $l_j$ is calculated using the mutual information as follows:

$$MI\left(l_i,l_j\right)=\sum _{k=1}^M\sum _{q=1}^{M}P\left(l_{ik},l_{jq}\right)log{\displaystyle \frac{P\left(l_{ik}l{l}_{jq}\right)}{P\left(l_{jq}\right)}}$$

(12)

An undirected label graph (WUG) $=(\mathrm{V},\mathrm{E},\mathrm{W})$ can be constructed by applying Equation (12). Here, $V=L=\left\{l_1,l_2,\dots ,l_m\right\}$ represents the set of nodes of the undirected graph, $E=$$\left\{\left(l_i,l_j\right)\mid l_i,l_j\in L\right\}$ represents its set of edges, and $w\left(l_i,l_j\right)=MI\left(l_i,l_j\right)$ represents the weight of each edge [48]. The importance of each node in this undirected graph is defined as follows:

$$LW\left(l_i\right)=(1-d)+d\sum _{l_j\in SN\left(l_i\right)}{\displaystyle \frac{LW\left(l_j\right)w\left(l_i,l_j\right)}{SW\left(l_j\right)}}$$

(13)

$$SW\left(l_j\right)=\sum _{l_j}w\left(l_i,l_j\right)$$

(14)

In the above equations, $LW\left(l_i\right)$ and $LW\left(l_j\right)$ represent the weight divisions of nodes $l_i$ and $l_j$, respectively. $SN\left(l_i\right)$ is the set of nodes with edges to label $l_i$, and $w\left(l_i,l_j\right)=MI\left(l_i,l_j\right)$ represents the correlation between nodes. Equation (10) is used to calculate $SN\left(l_i\right)$, which denotes the sum for the correlation for all edges starting from $l_j$. In addition, d is the damping coefficient, where $d=0.85$ is the recommended setting according to the method in Ref. [49]. For ease of calculation, an initial weight value can be set for all nodes; this is usually $1/L$, where L is the total number of nodes, that is, the total number of labels. Using this algorithm, we can calculate the correlation between node $l_i$ (i.e., label $l_i$ ) and other nodes $l_j$ related to it, as well as the structure of the graph (WUG). Through label correlation, we obtain the weight of each label in the label space, and this completes the exploration of label correlation.

4.2. Feature Selection Based on NRSs

The processing method of the labeled distribution neighborhood decision system is similar to that of the multi-label decision system. By extending the rough set importance theory of labeled distribution data (Equation (6)), combining the positive domain theory of labeled distribution decision (Equation (12)) and label correlation (Equation (14)), we obtain the importance of the decision attribute $D=L=\left\{l_1,l_2,\dots ,l_m\right\}$ on the set feature subset $B(B\subseteq C)$:

$${\gamma }_B\left(D^L\right)=\sum _{L_j\in L}^m{\displaystyle \frac{Card\left({PoS}_B\left(D^{L_j}\right)\right)LW\left(l_j\right)}{Card\left(U\right)}}$$

(15)

The above equation reflects the importance of the decision attributes corresponding to the decision positive domain and feature subset. It solves the problem of granularity selection and feature association of label distribution NRS.

In the neighborhood decision system $NDS=<U,C\cup D,\delta >$, for feature subset $B\subseteq C,a\in C-$B, the importance of a to B is defined as follows:

$$SIG(a,B,D)={\gamma }_{B\cup a}\left(D\right)-{\gamma }_B\left(D\right)$$

(16)

In the new importance model, we add label importance and label relevance to the NRS model. The new NRS model reflects the fusion of feature information and label relevance. For the above NRS model, we construct a method to compute the most reducible feature sets based on a greedy forward search.

In this method, steps 1 and 2 perform the preprocessing when the labeled data arrive. Our reduced set starts from the empty set and calculates the label weights $LW\left(L\right)$ for the entire label space. This step requires traversing the entire label space and constructing an undirected graph. Assuming that the number of labels in the label space is L, the time complexity of calculating the correlation between each pair of labels is $O\left({\left|L\right|}^2\right)$, and the time complexity of calculating the weight of each label is $O\left(1\right)$. Therefore, the time complexity of steps 1 and 2 is $\mathrm{O}\left({\left|L\right|}^2+1\right)=\mathrm{O}\left({\left|L\right|}^2\right)$. Steps $3-6$ are divided into two parts: calculating the neighborhood of the instance, and analyzing whether the instance and the neighborhood are important. First, the nearest hit or miss is determined for each instance by selecting the average approximate neighborhood as the domain granularity criterion. Assuming the instance space is U, the time complexity of this step is $\mathrm{O}\left({\left|U\right|}^2\right)$. Next, the neighborhood corresponding to the instance and the labeling granularity of the labeling space are determined, and then the decision positive domain and attribute importance are calculated. The time complexity of determining the neighborhood of each instance is $\mathrm{O}(nlogn)$; the time complexity of determining the marking granularity of the marking space is $\mathrm{O}\left({\left|U\right|}^2\right)$; and the time complexity of determining the decision positive domain and importance calculation corresponding to each sample is $\mathrm{O}\left(1\right)$. Therefore, the overall time complexity of instance domain calculation is $O\left({\left|U\right|}^2+{\left|U\right|}^2+\left|U\right|log\left|U\right|+1+1\right)=O\left({\left|U\right|}^2\right)$. Therefore, the time complexity of Algorithm 1 is $\mathrm{O}\left({\left|L\right|}^2+{\left|U\right|}^2\right)$.

Algorithm 1 Calculate $SIG(a,red,D).$

Input:  neighborhood decision system $\langle U,C\cup D\rangle$. Output:  $SIG(a,red,D)$  1: use Equation (5) to calculate the weight matrix $LW\left(L\right)$;  2: Initialize reduct $\leftarrow \varnothing$;  3: if reduct $=\varnothing$ then  4:     $POS\left(D\right)=0$;  5: end if  6: if reduct $\ne ⌀$ then  7:     $\forall x_i\in U$, the average approximate neighbor $d_{red}\left(x_i\right)$ under a is calculated by Equation (2);  8:     For $\forall x_i\in U$, calculate the neighborhood ${\delta }_{\mathrm{red}}\left(x_i\right)$ using Equation (3);  9:     For $\forall x_i\in U$, the label space granularity ${\Theta }_{D^{L_j}}\left(x_i\right)$ on label $D^{L_j}$ is calculated by Equation (7);  10:    For the label $l_j$ of $\forall x_i,{POS}_B\left(D^{L_j}\right)$ is calculated by Equation (7);  11:    Calculate ${\gamma }_B\left(D^L\right)$ by Equation (8);  12: end if  13: For $\forall a\in C-reduct$, repeat steps 3.1–3.5 to calculate ${\gamma }_{\mathrm{red}\cup a}\left(D\right)$;  14: Calculate $SIG(a$, red,$D)={\gamma }_{\mathrm{redua}}\left(D\right)-{\gamma }_a\left(D\right)$;  15: Output $SIG(a$, red,$D)$

4.3. Dynamic Algorithm for Online Label Distribution Feature Selection

Most feature selection algorithms assume that all candidate features are available to the algorithm before feature selection. In contrast, for streaming features, all the candidates cannot be collected before learning begins because they arrive dynamically and incrementally over time. Therefore, based on Method 1, we incorporate the online streaming feature selection framework [50] and propose an online labeled distributed streaming feature selection algorithm to solve the labeled distributed stream feature selection problem.

In the labeled distributed stream feature decision system $LFDS=<U,C\cup L,t>$, $U=\left\{x_1,x_2,\dots ,X_n\right\}$ represents the set of all non-empty instances, C represents the candidate feature subset, L represents the label space, t represents the time when the feature arrives, $F_t$ represents the new feature of t, and $S_{t-1}$ represents the current candidate feature subset at time t.

4.3.1. Importance Analysis

Letting $F_i$ be the new feature arriving at the i-th moment, $S_{t_{i-1}}{S}_{t_{i-1}}$ is the reduced set reduct at the i-th moment. For the new features that arrive, the first step is to perform redundancy analysis on $F_i$. The purpose of redundancy analysis is to determine whether the newly arrived feature $F_i$ is beneficial to the label set L, that is, to determine the importance of $F_i$ to the entire label set L.

Using Equation (17), $SIG\left(F_i,S_{t_{i-1}},D^L\right)={\gamma }_{F_i\cup S_{t_{i-1}}}\left(D^L\right)-{\gamma }_{S_{t_{i-1}}}\left(D^L\right)$ is set to measure the importance of $F_i$. If $SIG\left(F_i,S_{t_{i-1}},D^L\right)<0$, we consider $F_i$ to be redundant and unimportant for the current label set L, and hence $F_i$ can be omitted.

4.3.2. Significance Analysis

The purpose of correlation analysis is to evaluate the correlation between the new currently arriving features relative to the currently arriving features.

Define the significance of label L as follows:

$$\varsigma \left(L_j\right)={\displaystyle \frac{{\sum }_{i=1}^n{d}_{x_i}^{L_j}}{{\sum }_{j=i}^m{\sum }_{i=1}^n{d}_{x_i}^{L_j}}}$$

(17)

For a reduced ${red}_j$ of label $L_j$, we design a mapping ${\Phi }_j:{red}_j\to F$ that maps ${red}_j$ to the following d-dimensional vector:

$${\Phi }_j=\left[{\Phi }_j\left(f_1\right),{\Phi }_j\left(f_2\right),\dots ,{\Phi }_j\left(f_d\right)\right]$$

(18)

where ${\Phi }_j\left(f_i\right)=1$, if $f_i\in {red}_j$; otherwise, ${\Phi }_j\left(f_i\right)=0$.

We then define the significance of feature $f_i$ relative to $D^L$ on A as follows:

$${SIG}^{*}\left(f_i,A,D^L\right)=\sum _{\mathrm{j}=1}^{\mathrm{m}}\varsigma \left(L_j\right)·{\Phi }_j\left(f_i\right)$$

(19)

For a new feature $F_i$, we calculate ${SIG}^{*}\left(F_i,S_{t_{i-1}},D^L\right)$; when ${SIG}^{*}\left(F_i,S_{t_{i-1}},D^L\right)>$ 0, we believe that the significance $S_{t_{i-1}}$ of the new feature $F_i$ for label set $D^L$ is greater than or equal to the average significance of the features that have been processed in label set $D^L$. Therefore, we believe that $F_i$ is a significant feature and should be retained.

4.3.3. Redundancy Analysis

The purpose of redundancy analysis is to determine whether there is a feature $F_k$ in the current reduction set $S_{t_{i-1}}$ and whether the contribution of $F_k$ to label set L is the same as that of $F_i\left(F_i\subseteq S_{t_{i-1}}\right)$. When their contributions are the same, it is necessary to choose between the two features $F_k$ and $F_i$.

For the two features $F_i$ and $F_k$, if $SIG\left(F_i,F_k,D\right)=0$, then $F_i$ and $F_k$ have the same contribution as L. We thus compare ${SIG}^{*}\left(F_i,S_{t_{i-1}},D^L\right)$ and ${SIG}^{*}\left(F_k,S_{t_{i-1}},D^L\right)$. If ${SIG}^{*}\left(F_i,S_{t_{i-1}},D^L\right)\ge$${SIG}^{*}\left(F_k,S_{t_{i-1}},D^L\right)$, we keep $F_k$ and discard $F_i$; if ${SIG}^{*}\left(F_i,S_{t_{i-1}},D^L\right)<{SIG}^{*}\left(F_k,S_{t_{i-1}},D^L\right)$, we keep $F_i$ and discard $F_k$.

The streaming feature selection framework is based on online importance analysis, saliency analysis, and redundancy analysis. In this framework, a training set of known feature size is used to simulate the streaming features, and each streaming feature is generated from the candidate feature set. In the framework shown in Figure 2, we propose the dynamic tag distribution feature selection algorithm (Algorithm 1), which considers tag importance and tag correlation, and combines the above three analyses.

The flow feature selection framework, illustrated in Figure 5, is based on online importance analysis, significance analysis, and redundancy analysis. In this framework, a training set with known feature sizes is used to simulate flow features, and each flow feature is generated from the candidate feature set. In the framework shown in Figure 2, we propose a dynamic online label distribution feature selection algorithm that considers label importance and label correlation (Algorithm 2), which incorporates the above three types of analysis.

The main part of Algorithm 2 is the calculation of the dependency relationship between features. Here, $S_{t-1}$ is the number of features in the currently selected feature set at time t. Algorithm 1 evaluates whether new features at their time of arrival need to be retained and decides how to retain them. The whole process is an online selection problem, and includes three main parts: importance analysis, significance analysis, and redundancy analysis. We have made our code publicly available.

Algorithm 2 Dynamic online label distribution feature selection (DLILC-LDL).

Input:  New feature $F_i$ that arrives at the i-th moment, the simplified set $reduct\left(S_{t_{i-1}}\right)$ that arrives the i - 1-th moment, label set L, and redundancy weight $\delta$. Output:  Reduced tset reduct.  1: Use Equation (5) to calculate weight matrix $W\left(L\right)$;  2: Initialize reduct $\leftarrow \varnothing$;  3: If new features $F_i$ arrive, calculate ${\gamma }_{F_i}\left(D\right)$ according to Equation (16);  4: /significance analysis/  5: if red $=\varnothing$ then  6:     Calculate $SIG\left(F_i,S_{t_{i-1}},D\right)={\gamma }_{F_i\cup S_{t_{i-1}}}\left(D\right)-{\gamma }_{S_{t_{i-1}}}\left(D\right)$;  7:     if $SIG\left(F_i,S_{t_{i-1}},D\right)<0$ then  8:           discard $F_i$, to $\mathbf{5}$  9:     else  10:          reduct = reduct $\cup F_i$, to $\mathbf{5}$;  11:     end if  12:     /importance Analysis/  13: else  14:     According to Equation (11), calculate ${SIG}^{*}\left(F_i,S_{t_{i-1}},D^L\right)$;  15: end if  16: if${SIG}^{*}\left(F_i,S_{t_{i-1}},D^L\right)>0$then  17:     reduct = reduct $\cup F_i$, to $\mathbf{5}$;  18: else  19:     /Redundancy Analysis/  20:     while $\exists F_k\in$ reduct do  21:         if $SIG\left(F_i,F_k,D\right)=0$ and ${SIG}^{*}\left(F_i,S_{t_{i-1}},D^L\right)\ge {SIG}^{*}\left(F_k,S_{t_{i-1}},D^L\right)$ then  22:               discard $F_i$, to 5;  23:         else if ${SIG}^{*}\left(F_i,S_{t_{i-1}},D^L\right)<{SIG}^{*}\left(F_k,S_{t_{i-1}},D^L\right)$ then  24:               reduct = reduct $-F_k$  25:               reduct = reduct $\cup F_i$, to $\mathbf{5}$  26:         end if  27:     end while  28: end if  29: No new features $F_i$ arrive;  30: Output reduced set reduct.

The feature calculation performed by Algorithm 2 is taken from Method 1. Hence, the time complexity of a single feature selection is $O\left({\left|L\right|}^2+\left|U\right|log\left|U\right|\right)$. However, in most cases, Algorithm 2 is neither simple nor optimistic, and it requires online updates. Because the time complexity of the update depends on the calculation of feature dependencies, in the worst case, it is necessary to traverse all selected features for processing, and hence the time complexity in the worst case is $\mathrm{O}\left({\left|L\right|}^2+\left|S_{t-1}\right|\left|U\right|log\left|U\right|\right)$.

5. Experimental Data

5.1. Datasets

To verify the efficiency and accuracy of our algorithm, we conducted experiments on 10 labeled distribution datasets, including 1 facial expression dataset SJAFFE [51], 7 biological experiment datasets (Yeast [24]), 1 natural scene dataset (Natural_Scene [52]), and 1 large-scale biomedical research dataset (Human_Gene [53]) in the

Table 2. Main features of the 12 datasets.

No.	Dataset	Examples	Features	Labels
1	S-JAFFE	213	243	6
2	Natural_Scene	2000	294	9
3	Human_Gene	17,892	36	68
4	Yeast-alpha (alpha)	2465	24	18
5	Yeast-diau (diau)	2465	24	7
6	Yeast-dtt (dtt)	2465	24	4
7	Yeast-elu (elu)	2465	24	14
8	Yeast-heat (heat)	2465	24	6
9	Yeast-spo (spo)	2465	24	6
10	Yeast-spo5 (spo5)	2465	24	3

.

5.2. Evaluation Indices

In the experiment, one indicator of the performance of the algorithm is the average distance or similarity between the predicted and true label distributions. To verify the efficiency of our algorithm, we selected six evaluation indicators widely used in the labeled distribution community: the Chebyshev distance, Clark distance, Canberra distance, Kullback–Leibler (KL) divergence, cosine similarity, and intersection similarity. The specific calculation method for six evaluation indicators is shown in

Table 3. Algorithm evaluation indicators.

Name	Formula
Chebyshev distance (Chebyshev) ↓	$Dis(\overline{D},D)={max}_i\left|{\overline{d}}_i-d_i\right|$
Clark distance (Clark) ↓	$Dis(\overline{D},D)=\sqrt{{\sum }_{j=1}^L{\displaystyle \frac{{\left({\overline{d}}_i-d_i\right)}^2}{{\left({\overline{d}}_i+d_i\right)}^2}}}$
Canberra distance (Canberra) ↓	$Dis(\overline{D},D)={\sum }_{j=1}^L{\displaystyle \frac{\left|{\overline{d}}_i-d_i\right|}{{\overline{d}}_i+d_i}}$
KL divergence (KL) ↑	$Dis(\overline{D},D)={\sum }_{j=1}^L{\overline{d}}_{i}ln{\displaystyle \frac{{\overline{d}}_i}{d_i}}$
Cosine similarity (Cosine) ↑	$Sim(\overline{D},D)={\displaystyle \frac{{\sum }_{j=1}^L{\overline{d}}_i{d}_i}{\Gamma }}$
Intersection similarity (Intersection) ↑	$Sim(\overline{D},D)={\sum }_{j=1}^{L}min\left({\overline{d}}_i,d_i\right)$

. Table 2 summarizes the mathematical equations of these six indicators, where $D=\left\{d_1,d_2,\dots ,d_L\right\}$ represents the predicted label distribution and $\overline{D}=\left\{\overline{d_1},\overline{d_2},\dots ,\overline{d_L}\right\}$ represents the actual label distribution. In the table, for each evaluation indicator, the downward arrow $(\downarrow )$ indicates that smaller values signify better performance, whereas the upward arrow (↑) indicates that larger values signify better performance.

5.3. Experimental Setup

To evaluate the effectiveness and efficiency of our proposed algorithm, we compared it with MDFS [54], GRRO [55,56], MDDM [57], FSFL [23], LDFS [58], six advanced feature selection algorithms.

When reducing the dimensions of the feature space, the experimental dataset needs to be adapted for different feature selection methods. For the label distribution feature selection method, a multi-label dataset must be converted into a label distribution dataset through label enhancement. Specifically, the experiments in this study used the label enhancement method to convert the data. Multi-label feature selection methods cannot directly use the real-world label distribution dataset. Therefore, we used the equal frequency strategy to discretize the label distribution dataset before dimensionality reduction.

The experiment was conducted using a 10 -fold cross-validation strategy. To obtain the output, the learner was trained on the above 15 datasets using SA-BFGS after feature selection, and the prediction performance was obtained in the test. Following [59], the number of training iterations of SA-BFGS was set to 10, and the regularization factor was 1.

5.4. Experimental Results

To demonstrate the effectiveness of our algorithm, we compared the algorithm proposed in this paper, the DLILC-LDL algorithms, with MDDMproj, MDDMspc, MDFS, LDFS, GRRO, and FSF in terms of their predictive classification performance. To ensure the results were comparable, all features obtained by all comparison algorithms were sorted, the number of features selected by the dynamic feature selection method proposed in this paper was used as the final feature subset, and the features with the same number of features in the comparison algorithm results were used as their feature subsets.

Because all the comparison algorithms use the result of feature selection as the result of feature sorting, we give the detailed experimental results of all algorithms on the various classification datasets in

Table 4. Comparison of the Chebyshev distance results of different algorithms ↓.

	MDDMproj	MDDMspc	MDFS	LDFS	GRRO	FSFL	DLILC-LDL
S-JAFFE	0.1029	0.1013	0.1087	0.1099	0.1054	0.0860	0.0832
Natural_Scene	0.3431	0.3395	0.3336	0.3266	0.3323	0.3497	0.3199
Human_Gene	0.06303	0.06310	0.06304	0.06312	0.06320	0.06313	0.06291
Yeast-alpha (alpha)	0.009805	0.009863	0.009910	0.009883	0.009835	0.009878	0.009775
Yeast-diau (diau)	0.03600	0.03604	0.03558	0.03564	0.03558	0.03576	0.03456
Yeast-dtt (dtt)	0.04020	0.03998	0.03974	0.03815	0.03990	0.03973	0.03939
Yeast-elu (elu)	0.01690	0.01700	0.01688	0.01680	0.01688	0.01690	0.01668
Yeast-heat (heat)	0.04363	0.04331	0.04343	0.04338	0.04335	0.04298	0.04282
Yeast-spo (spo)	0.05936	0.05948	0.05940	0.05946	0.05942	0.05918	0.05852
Yeast-spo5 (spo5)	0.09360	0.09243	0.09338	0.09265	0.09245	0.09343	0.09168
Avg	0.08085	0.080020	0.08036	0.07955	0.07983	0.07967	0.07595

,

Table 5. Comparison of the Clark distance results of different algorithms ↓.

	MDDMproj	MDDMspc	MDFS	LDFS	GRRO	FSFL	DLILC-LDL
S-JAFFE	0.3776	0.3546	0.3827	0.3979	0.3795	0.3398	0.3369
Natural_Scene	2.0478	2.0607	2.0347	2.0114	2.0491	2.0463	2.0216
Human_Gene	1.4321	1.4602	1.4548	1.4397	1.4425	1.4423	1.4090
Yeast-alpha (alpha)	0.2234	0.2213	0.2086	0.2087	0.2083	0.2094	0.2073
Yeast-diau(diau)	0.1900	0.1908	0.1886	0.1892	0.1890	0.1899	0.1850
Yeast-dtt (dtt)	0.1126	0.1125	0.1123	0.1416	0.1122	0.1118	0.1106
Yeast-elu (elu)	0.2064	0.2060	0.2059	0.2066	0.2069	0.2082	0.2060
Yeast-heat (heat)	0.1453	0.1454	0.1454	0.1458	0.1462	0.1440	0.1456
Yeast-spo (spo)	0.1986	0.2101	0.2102	0.2104	0.2106	0.2104	0.2075
Yeast-spo5 (spo5)	0.1809	0.1796	0.1798	0.1803	0.1804	0.1804	0.1775
Avg	0.5115	0.5141	0.5123	0.5131	0.5125	0.5082	0.5007

,

Table 6. Comparison of the Canberra distance results of different algorithms ↓.

	MDDMproj	MDDMspc	MDFS	LDFS	GRRO	FSFL	DLILC-LDL
S-JAFFE	0.8108	0.7718	0.8413	0.8359	0.8163	0.7368	0.7564
Natural_Scene	5.7448	5.8124	5.7120	5.6357	5.7524	5.7997	5.6653
Human_Gene	16.8239	16.3366	16.3188	16.3464	16.3496	16.3495	16.2999
Yeast-alpha (alpha)	0.6781	0.6837	0.6836	0.6860	0.6847	0.6872	0.6841
Yeast-diau (diau)	0.4135	0.4138	0.4087	0.4103	0.4103	0.4109	0.4014
Yeast-dtt (dtt)	0.1887	0.1885	0.1885	0.1890	0.1888	0.1880	0.1866
Yeast-elu (elu)	0.6104	0.6072	0.6106	0.6091	0.6115	0.6163	0.6055
Yeast-heat (heat)	0.3742	0.3751	0.3758	0.3753	0.3770	0.3740	0.3720
Yeast-spo (spo)	0.5251	0.5258	0.5250	0.5246	0.5259	0.5233	0.5176
Yeast-spo5 (spo5)	0.2763	0.2757	0.2773	0.2766	0.2765	0.2738	0.2751
avg	2.6446	2.5991	2.5942	2.5889	2.5993	2.5959	2.5764

,

Table 7. Comparison of the KL divergence results of different algorithms ↓.

	MDDMproj	MDDMspc	MDFS	LDFS	GRRO	FSFL	DLILC-LDL
S-JAFFE	0.06453	0.06130	0.06767	0.06957	0.06657	0.04857	0.04340
Natural_Scene	0.7070	0.7387	0.6850	0.6489	0.7028	0.7058	0.6468
Human_Gene	0.6394	0.6063	0.6071	0.6138	0.6014	0.6510	0.5862
Yeast-alpha (alpha)	0.005350	0.005417	0.005413	0.005447	0.005413	0.005510	0.005403
Yeast-diau (diau)	0.01168	0.01166	0.01166	0.01174	0.01176	0.01154	0.01168
Yeast-dtt (dtt)	0.006878	0.006900	0.006882	0.006960	0.006900	0.006838	0.006716
Yeast-elu (elu)	0.006867	0.006833	0.006700	0.006750	0.006775	0.006875	0.006625
Yeast-heat (heat)	0.01323	0.01325	0.01320	0.01330	0.01335	0.01345	0.01305
Yeast-spo (spo)	0.02470	0.02478	0.02456	0.02458	0.02472	0.02456	0.02396
Yeast-spo5 (spo5)	0.02965	0.02960	0.02968	0.02988	0.02998	0.02993	0.02923
avg	0.1509	0.1505	0.1458	0.1431	0.1470	0.1504	0.1373

,

Table 8. Comparison of the cosine similarity results of different algorithms ↑.

	MDDMproj	MDDMspc	MDFS	LDFS	GRRO	FSFL	DLILC-LDL
S-JAFFE	0.9421	0.9442	0.9372	0.9361	0.9389	0.9540	0.9614
Natural_Scene	0.7586	0.7519	0.7667	0.7745	0.7627	0.7604	0.7753
Human_Gene	0.7437	0.7412	0.7399	0.7347	0.7421	0.7440	0.7458
Yeast-alpha (alpha)	0.9948	0.9948	0.9946	0.9947	0.9947	0.9946	0.9947
Yeast-diau (diau)	0.9887	0.9887	0.9889	0.9888	0.9888	0.9887	0.9892
Yeast-dtt (dtt)	0.9926	0.9926	0.9927	0.9926	0.9925	0.9928	0.9928
Yeast-elu (elu)	0.9936	0.9936	0.9936	0.9935	0.9935	0.9937	0.9936
Yeast-heat (heat)	0.9799	0.9873	0.9873	0.9872	0.9872	0.9874	0.9875
Yeast-spo (spo)	0.9763	0.9766	0.9762	0.9762	0.9762	0.9764	0.9769
Yeast-spo5 (spo5)	0.9742	0.9743	0.9744	0.9748	0.9739	0.9741	0.9746
avg	0.9344	0.9346	0.9351	0.9352	0.9350	0.9366	0.9392

and

Table 9. Comparison of the intersection similarity results of different algorithms ↑.

	MDDMproj	MDDMspc	MDFS	LDFS	GRRO	FSFL	DLILC-LDL
S-JAFFE	0.8661	0.8680	0.8677	0.8626	0.8670	0.8804	0.8825
Natural_Scene	0.5297	0.5195	0.5472	0.5679	0.5403	0.5391	0.5633
Human_Gene	0.7323	0.7329	0.7329	0.7323	0.7321	0.7322	0.7330
Yeast-alpha (alpha)	0.9621	0.9621	0.9621	0.9620	0.9621	0.9624	0.9622
Yeast-diau (diau)	0.9423	0.9422	0.9427	0.9426	0.9426	0.9425	0.9438
Yeast-dtt (dtt)	0.9542	0.9542	0.9544	0.9547	0.9544	0.9546	0.9550
Yeast-elu (elu)	0.9568	0.9570	0.9568	0.9569	0.9567	0.9564	0.9572
Yeast-heat (heat)	0.9383	0.9382	0.9371	0.9383	0.9379	0.9378	0.9388
Yeast-spo (spo)	0.9135	0.9134	0.9135	0.9134	0.9132	0.9171	0.9146
Yeast-spo5 (spo5)	0.9107	0.9109	0.9108	0.9106	0.9106	0.9104	0.9119
avg	0.8706	0.8699	0.8725	0.8741	0.87179	0.8732	0.8762

. For each evaluation criterion, “↓” indicates that smaller values are better and “↑” indicates that larger values are better. In addition, the best predictive classification performance of the evaluation indicator is shown in bold and the average predictive classification performance of each algorithm is shown in italics.

In addition, in our experiments, to study the performance of the streaming feature selection method when the entire feature set is unknown in advance, we used the 12 benchmark datasets listed in Table 2 as our testbed and simulated the streaming feature situation by observing the features on the training data that arrived each time.

From the experimental results in Table 4, Table 5, Table 6, Table 7, Table 8 and Table 9, we can infer the following:

(1)

For all evaluation indicators, DLILC-LDL is generally better than the comparison algorithms. The advantages of Algorithm 1 are basically suitable for all the labeled distribution datasets in the experiment.

(2)

In terms of the Chebyshev distance and cosine similarity, DLILC-LDL achieved the best performance on nine of the labeled distribution datasets. Note that DLILC-LDL achieved suboptimal or near-suboptimal performance on the remaining dataset. The prediction and classification performance of DLILC-LDL is also very close to the best performance of another comparison algorithm on the labeled distribution dataset. In particular, in terms of the Chebyshev distance, the DLILC-LDL algorithm generally achieved the best performance.

(3)

In terms of the Clark and Canberra distances, the prediction and classification performance results of DLILC-LDL are significantly better than those of the compared algorithms on seven label distribution datasets. Note that even on the datasets where the best performance was not achieved, the results of DLILC-LDL are still quite good, and it ranked second on the remaining datasets.

(4)

In terms of different evaluation indicators, the average classification performance of DLILC-LDL is significantly better than that of all the compared algorithms. These experimental results reveal that the DLILC-LDL has better performance than the other compared algorithms.

Prediction performance is expected to vary. To clearly evaluate the differences among the algorithms, the prediction performance was normalized to [0.1, 0.5] following [23]. Figure 6 shows the normalized stability indices for the Chebyshev distance, Clark distance, Canberra distance, KL divergence, cosine similarity, and intersection similarity. Each corner of the spider-web chart in Figure 6 represents a different dataset, and each colored line represents a different algorithm.

If the area of the graph enclosed by a specific line is large and its shape is similar to that of a regular dodecagon, the performance and stability of the corresponding algorithm are good. A stability value of approximately 0.5 is considered to be a good value. Figure 6 reveals the following:

(1)

For all the evaluation indices, DLILC-LDL has the best stability because its shape is very close to that of a regular dodecagon, and it has the largest enclosed area.

(2)

For the Chebyshev distance, intersection similarity, cosine similarity, and intersection similarity, DLILC-LILC maintains stability on at least ten datasets.

(3)

For the Clark distance, KL divergence, and Canberra distance, the values of SLILC-LDL are similar to those of some comparison algorithms. Therefore, its performance advantage over the existing algorithms is not as obvious as in other evaluation criteria.

(4)

For all evaluation criteria, the area of DLILC-LILC is larger than or similar to those of the existing algorithms and is closer in shape to a regular dodecagon. In fact, comprehensive Bonferroni–Dunn tests on the DLILC-LILC algorithm comparing it with existing algorithms reveal that the performance and the stability of the DIILC-LILC algorithm is the best.

5.5. Statistical Tests

Because some of the experimental results were similar, statistical tests were used to evaluate whether there are significant differences in the experimental results. To systematically explore the statistical significance of the comparison algorithms, we used the Friedman test to further analyze the differences between the performance results of the comparison algorithms. This is a highly accepted method of statistically comparing multiple algorithms for significant differences across many datasets [60]. The specific method is as follows: given k comparison algorithms and N multi-label datasets, $R_j={\displaystyle \frac{1}{N}}{\sum }_{i=1}^N{r}_i^{j}v$ represents the average rank of the j-th algorithm on all datasets, where $r_i^j$ is the rank of algorithm j on the i-th dataset. Under the null hypothesis (it is assumed that the classification performance of each algorithm under each evaluation index is equal, that is, the ranks of all algorithms are equal), the Friedman test can be defined as follows:

$$F_F={\displaystyle \frac{(N-1){\chi }_F^2}{N(k-1)-{\chi }_F^2}},\mathrm{where}{\chi }_F^2={\displaystyle \frac{12N}{k(k+1)}}\left(\sum _{i=1}^k{R}_i^2-{\displaystyle \frac{k{(k+1)}^2}{4}}\right)$$

(20)

where $F_F$ follows an F-distribution with $(k-1)$ and $(k-1)(N-1)$ degrees of freedom. Table 9 summarizes the $F_F$ values and the corresponding critical values of each evaluation index after the Friedman test statistics were calculated [61].

As

Table 10. Summary of the Friedman test ( $\mathrm{k}=8,\mathrm{N}=10$ ), giving the $F_F$ values and critical values of each evaluation index on $\alpha =0.10$.

Evaluation Metric	${\mathit{F}}_{\mathit{F}}$	Critical Value $(\mathit{\alpha }=0.10)$
Chebyshev	6.357	1.770
Clark	6.294
Canberra	5.286
KL	8.450
Cosine	5.269
Intersection	6.683

reveals, for all evaluation indicators with a significance level of $a=0.10$, the null hypothesis is clearly disproved. Next, we used post hoc tests to further determine the differences in performance of these comparison algorithms. Because our focus was on comparing the proposed method with other algorithms, the Bonferroni–Dunn test was used [62]. The Bonferroni–Dunn test is a method used in multiple comparison problems to control the probability of a Type I error (that is, incorrectly rejecting a hypothesis that is truly null). When conducting multiple statistical tests, the probability of rejecting one or more null hypotheses (i.e., the probability of making a Type I error) increases with the number of tests, even if all of the null hypotheses are true. The Bonferroni–Dunn test reduces the overall risk of this error by reducing the significance level of a single test. The disadvantage of the Bonferroni–Dunn test is that it is relatively conservative and may increase the risk of Type II errors (i.e., incorrectly accepting a false null hypothesis) when the amount of data is large. Therefore, we set the reported corrected P value (the probability of observing the current result or a more extreme result under the premise that the null hypothesis is true) greater than 1 as 1. This was performed to maintain the accuracy of statistical analysis and the legitimacy of probability, and reduce the probability of Type II errors. When the distance between the average ranks exceeds the following critical difference (cd), we consider DLILC-LDL to have the same performance as that of a comparison algorithm.

From Figure 7, we can observe the following:

(1)

DLILC-LDL shows significant improvement in several evaluation indicators.

DLILC-LDL significantly outperforms other comparison algorithms. The relative advantage is obvious.

(2)

Although the classification performance results of FSFL and GRRO are comparable, the average classification performance of DLILC-LDL in Table 3, Table 4, Table 5, Table 6, Table 7 and Table 8 is significantly better than that of the comparison algorithms. In summary, DLILC-LDL is better than other advanced multi-label feature selection algorithms, and this method has stronger statistical performance.

6. Summary

This paper proposed a label distribution feature selection model based on label correlation and NRS, and it further proposed a label distribution feature selection algorithm based on this model, namely, a dynamic algorithm for processing streaming features. In the proposed approach, we first defined a new neighborhood particle by averaging the nearest neighbor method to better connect the information between the features of the label distribution data. Then, by calculating the mutual information between the features, we obtained the feature correlation weight and combined the lower approximation of the new neighborhood with the feature weight to obtain a new feature subset importance model. On this basis, we generalized the traditional NRS model, defined a new label space granularity, and granulated the label space of the label distribution data to adapt it to LDL. Incorporating rough set theory, we defined the label distribution neighborhood decision system and proposed a new importance model. Moreover, based on this algorithm, we proposed a dynamic feature selection algorithm that can solve the label distribution streaming feature problem by testing the arriving features in the time series using importance, significance, and redundancy analyses. The experimental results show that our algorithm is highly competitive compared with other commonly used algorithms. However, in a large number of dynamic data environments, the practical application of this method is somewhat challenging due to the high time complexity of the algorithm. The time complexity of the algorithm mainly comes from the calculation of the sample neighborhood and the feature selection of the sample based on the neighborhood. In the labeled distribution environment, the label granularity of the sample also needs to be calculated. Therefore, in order to improve the time complexity, the samples can be pre-sorted according to the rules of the label space, and the correctness of the feature selection can be further improved. This will be the direction for research and optimization in the next step. Therefore, in future work, we hope to further optimize the algorithm and reduce the running time of the algorithm. At the same time, we hope to further improve the label space particles and propose a better label distribution neighborhood decision system. In addition, the next step is to combine this method with practical applications (such as sentiment computing and multimodal sentiment recognition) to practice this method.