Updating Correlation-Enhanced Feature Learning for Multi-Label Classification

Abstract

In the domain of multi-label classification, label correlations play a crucial role in enhancing prediction precision. However, traditional methods heavily depend on ground-truth label sets, which can be incompletely tagged due to the diverse backgrounds of annotators and the significant cost associated with procuring extensive labeled datasets. To address these challenges, this paper introduces a novel multi-label classification method called updating Correlation-enhanced Feature Learning (uCeFL), which extracts label correlations directly from the data instances, circumventing the dependency on potentially incomplete label sets. uCeFL initially computes a revised label matrix by multiplying the incomplete label matrix with the label correlations extracted from the data matrix. This revised matrix is then utilized to enrich the original data features, enabling a neural network to learn correlation-enhanced representations that capture intricate relationships between data features, labels, and their interactions. Notably, label correlations are not static; they are dynamically updated during the neural network’s training process. Extensive experiments carried out on various datasets emphasize the effectiveness of the proposed approach. By leveraging label correlations within data instances, along with the hierarchical learning capabilities of neural networks, it offers a significant improvement in multi-label classification, even in scenarios with incomplete labels.

1. Introduction

Given the widespread occurrence of multi-label issues, researchers have introduced various multi-label classification techniques across diverse domains [1,2,3,4]. Unlike traditional single-label classification tasks [5,6], the semantic polysemy associated with multi-label data contributes to the complexity of these challenges. Specifically, as the number of categorical labels proliferates exponentially, the challenge of selecting an apt label set for novel data intensifies significantly [7].

Multi-label sample data consists of a wide array of category labels, many of which exhibit either direct or indirect relationships with each other. For instance, consider two illustrative images: one featuring a pyramid alongside a camel and the other displaying a pyramid situated in a desert. Initially, the camel and the desert in these two images may appear unrelated at a cursory glance. However, upon deeper examination of the contextual relationships inherent within these images, it becomes evident that there is an indirect correlation between the labels “camel” and “desert”. This observation is noteworthy as the terms “camel” and “desert” typically evoke a direct association in real-world scenarios. Research indicates that leveraging these label correlations, especially in scenarios with numerous category labels, can significantly reduce the complexity of classification problems and enhance classifier performance. The investigation of label correlations thus fuels the progress of multi-label classification, constituting a crucial research direction in this field [8,9].

Multi-label classification methods primarily categorize into four key strategies that capitalize on label correlations:

(1) Implicit exploitation of label dependencies, which involves partitioning a multi-label dataset into subsets based on labels and individually learning the data features within each subset. The representative method is label powerset (LP) [10], a simple method that treats each label subset in a training set as a single new label to construct a single-label classification problem. Although LP considers label correlations, it may produce a large number of classes, each with only a few examples. To solve the problem of insufficient training samples of new labels generated by LP, Read proposed pruned problem transformation (PPT) [11], which clips label subsets whose frequencies are much less than a threshold to smaller label subsets with slightly higher frequencies. Random k-Labelsets (RAkEL) [12] is an ensemble algorithm based on LP, which randomly selects m non-repetitive label subset combinations composed of k labels from an original set of $K$ category labels, and trains an LP classifier on the data of each label subset. Multiple LP classifiers vote for the prediction results. Parameters k and m are set artificially. RAkEL degenerates to a BR algorithm when $K$ = 1 and $m$ = $K$ and LP when $k$ = $K$ and $m$ = 1. RAkEL considers correlations between labels while avoiding the drawbacks of LP. However, for the best results, it requires a large number of datasets, as well as cross-testing of input parameters such as subset size and threshold. It is difficult to find the optimal parameters without sufficient training samples.

(2) Incorporating label correlations as prior knowledge during the training of classification models, thereby leveraging pre-existing relationships between labels. Wang et al. [13] proposed a subspace learning algorithm based on label sparse coding for feature extraction. This implies the exploitation of label correlations to annotate different images with the same label and appropriate weights. Zhang and Zhou [7] used label correlations to find a projection matrix that maximizes correlations between data features and labels in the dimensionally reduced feature space. Wang et al. [14] calculated the cosine similarity between category labels and constrained the learning of the objective function with the similarity matrix to obtain a Green’s function enhanced by the label correlations.

(3) Concurrent discovery of label correlations during the training process of the multi-label model, allowing for dynamic exploitation of inter-label relationships. Huang et al. [15] assumed that label correlations are local because sample sets share different label correlations, few of which apply globally. To encode the local effects of label correlations, they derived an LOC code to enhance the feature representation of each sample and combined the global discriminant fit and local correlation sensitivity in a unified framework. Xu et al. [9] captured label correlations by a low-rank structure and introduced a low-rank constraint of the label correlation matrix in the model learning process. Gu et al. [16] used a normal distribution of matrix variables on the vector of classifiers to model label correlations and minimized the label correlation regularized loss of label ranking based on a subset of features. Ji et al. [17] proposed a general framework to extract shared structures, within which a binary classifier was constructed for each label. Unlike traditional binary classifiers, all binary classifiers were defined on both the original data space and low-dimensional shared structures to capture label correlations.

(4) Augmenting the original feature space with a label space, thereby enriching the data with additional information stemming from label correlations. Read et al. introduced classifier chains (CCs) [18], constructing a binary classifier for each label, whose result is used to expand the feature space of the next classifier. However, the order of construction affects classification performance. Montañes et al. proposed dependent binary relevance (DBR) [19], which learns a binary classifier for each label, each employing $k$ − 1 features made up of other labels to expand the original feature space. Learning the label-expanded feature space, its binary classifiers can discover label correlations by themselves. However, the number of classifiers is proportional to the label length.

The aforementioned approaches primarily derive label correlations from the label sets available in the training data. However, these label sets are often incomplete, which is mainly attributed to the diverse backgrounds of grassroots users annotating data features [20] and the high cost associated with extracting a significant number of labeled samples for large-scale tasks [21]. Consequently, the accuracy of label correlations can suffer when labels are missing.

In the context of multi-label learning, a single instance is simultaneously associated with multiple class labels, and each label is determined by unique features specific to that instance. Since the amount of data feature information varies across objects corresponding to different category labels, the importance of labels within the same instance also differs. Nevertheless, the binary value (0 or 1) assigned to a label $y_{ij}$ in a tagged label set merely signifies the presence or absence of the feature represented by label $j$ on data $x_i$. This approach fails to capture the significance of the specific feature corresponding to label $j$. As a result, relying solely on the given label matrix to derive label correlations may lead to distorted classification results.

Taking into account the aforementioned challenges, a novel multi-label learning architecture known as updating Correlation-enhanced Feature Learning (uCeFL) is introduced in this paper, specifically tailored for multi-label classification tasks. This approach harnesses label information to enrich the features of the original data, echoing the principles outlined in point 4 of the previous discussion. Notably, rather than merely incorporating the given labels directly into the expanded feature space, uCeFL enriches the data feature space by utilizing labels completed through label correlation. This label correlation is extracted from the data matrix, ensuring resilience against missing labels. This innovative approach effectively circumvents the challenges posed by incomplete label matrices, resulting in more precise and robust multi-label classification.

uCeFL employs a covariance matrix to efficiently capture the distinct features of each subset of data affiliated with individual category labels. Through the comparison of covariance matrices within an $n\times n$ dimensional space, it accurately quantifies the correlation among these labels. It is universally acknowledged that the covariance between two variables increases as their interaction intensifies, and a covariance matrix comprehensively characterizes the interactions among all variables. The similarity between two covariance matrices suggests that their respective characteristic variables exhibit comparable distributions. Since category labels serve as abstract representations of interrelated data features, it is logically justifiable to express the correlation among labels through the similarity of covariance matrices pertaining to categorical data features. By deriving label relationships directly from the data, uCeFL effectively mitigates any potential interference caused by incomplete labels.

Regarding the intricate new data structure that is expanded with complete labels, this paper refrains from making any assumptions. However, since real-world data often reside in a low-dimensional manifold embedded within a high-dimensional space, the new data structure can be characterized as a nonlinear manifold. Multi-layer neural networks excel at simulating such nonlinear manifold structures without relying on hypotheses. They can acquire high-level data features by learning the hierarchical structure. Therefore, this paper utilizes a neural network to efficiently learn the multi-label data with increased dimensions, obtaining the fundamental variables that explain the enhanced data features. Label correlations are not merely considered as prior knowledge; rather, they are dynamically updated throughout the neural network training process, which ensures that our neural network continuously optimizes the label correlation, ultimately leading to more accurate and refined predictions.

Moreover, to emphasize the effectiveness of uCeFL, this paper introduces a variant named Correlation-enhanced Feature Learning (CeFL). This approach solely treats label correlations as prior knowledge, serving as a static input to the neural network throughout the entire training process. This input remains constant, ensuring consistency and stability.

Distinct from numerous existing methods, uCeFL distinguishes itself as being significantly more intuitive and comprehensible. Rigorous experiments spanning seven diverse databases have unequivocally established its superiority. Moreover, the label correlations inherently extracted from the data effectively capture the varying degrees of importance ascribed to each label, thereby reflecting the intricate relationships and interactions among the distinct features represented by these labels.

In essence, the key contributions of this article are outlined as follows:

• Our method distinguishes itself from the preponderance of multi-label correlation learning algorithms by directly extracting label correlations from the data itself, thereby enhancing their precision beyond those derived solely from pre-assigned labels.
• We introduce a revised label matrix, obtained by multiplying the incomplete label matrix with the captured label correlation, which serves to enrich the original data’s feature representation. A multi-layer neural network is then employed to learn the correlation-enhanced features that encapsulate intricate relationships among data features, labels, and their interconnections.
• Leveraging the high-level semantic features extracted by the multi-layer neural network, our uCeFL approach re-evaluates the label correlation, facilitating continuous updates of these correlations throughout the neural network’s learning trajectory. This approach effectively captures the label propagation effects, ensuring a comprehensive and accurate representation of the label space.

The remainder of this work is structured as follows. Section 2 provides the related work, exploring the existing literature and prior studies pertinent to our investigation. Section 3 delves into the uCeFL approach, offering a comprehensive explanation of the methodologies utilized and the architecture adopted in our research. Additionally, an introduction to CeFL is also provided. Section 4 presents the experimental results and discusses the outcomes obtained through rigorous testing and validation. Finally, Section 5 outlines our conclusions and future work, summarizing the key findings of our study and highlighting potential avenues for further exploration.

2. The Related Work

Among the various multi-label classification algorithms, classifier chains (CC) [18] occupy a significant position as a noteworthy approach. However, a major limitation of CC lies in its sequential nature, where each classifier’s prediction heavily depends on the output of its predecessor. This sequential dependency often leads to error accumulation and propagation, adversely affecting the overall prediction accuracy. To mitigate this limitation, dependent binary relevance (DBR) [19] methods have emerged as a promising alternative. DBR constructs independent binary classifiers for each label, incorporating features relevant to other labels to enrich the input data space. This allows classifiers to independently capture and utilize label dependencies.

However, traditional binary classifiers may not fully realize the potential of DBR when dealing with specific multi-label loss functions. To enhance DBR’s performance, various optimization algorithms have been proposed. Rauber et al. [22] introduced a recursive approach that leverages DBR’s initial predictions as a starting point. This method iteratively improves predictions through feedback, aiming to boost accuracy by continuously modeling label dependencies. Alternatively, Zhang et al. [23] proposed a DBR pruning technique based on Phi coefficients, which evaluates label-task relevance, optimizes the model structure, reduces costs, and demonstrates practical effectiveness. Furthermore, the RDBR algorithm [24], a novel approach within decision tree methods, generates crucial intermediate values for constructing DP matrices. While it offers a unique perspective, ensemble methods still provide superior diversity and robustness.

With the progress of multi-label classification research, researchers have increasingly focused on two key directions: label-specific feature learning and label dependency modeling. These approaches aim to learn the most suitable feature representation for each category and utilize label correlations to improve classification performance.

Label-specific feature learning aims to extract features closely related to the label from the data to improve classification performance. Zhang et al. [25] employed clustering techniques to delve into the inherent patterns of the feature space. However, this clustering approach was executed in a label-specific manner, overlooking the potential interplay between different class labels. ML-DFL, introduced by Zhang et al. [26], relies on a novel spectral clustering algorithm known as Spectral Instance Alignment (SIA). This method excels in capturing the latent patterns between positive and negative instances for each label. In another pursuit, RFS-LIFT [27] endeavors to select a refined set of label-specific features generated by LIFT, leveraging the fuzzy rough set framework. Concurrently, [28] introduced an efficient multi-label feature selection technique grounded in an information-theoretic label selection approach. This method identifies a subset of labels that significantly influence feature importance, enabling an effective output of a tailored feature subset. Nonetheless, a significant oversight among these methodologies is the lack of consideration for label correlations, which could potentially bolster their performance even further.

Label dependency modeling is another crucial research direction in multi-label classification, focusing on capturing and utilizing dependencies between labels. Loza et al. [29] tackle label dependencies in multi-label classification. They propose using rules to represent dependencies and introduce two rule-learning approaches: a bootstrapped stacking method that learns separate rulesets for each label and an algorithm that reincorporates predicted labels for learning subsequent rules. Both methods make dependencies explicit in rules and employ standard techniques to ensure rule usefulness. Li et al. [30] proposed the A-GCN, which leverages popular Graph Convolutional Networks with an adaptive label correlation graph to model label dependencies. Wang et al. [31] leverage Bayesian networks (BNs) to model the intricate correlations among labels and introduce a novel BN-based classifier chain method (BNCC). They utilize conditional entropy to capture the dependencies between labels, transforming the labels into nodes in a Bayesian Network (BN). The strength of these dependencies is represented by the weights of the edges. Furthermore, they devise a novel scoring function to evaluate the quality of the BN structure and employ a heuristic algorithm to refine and optimize the BN. Ultimately, through the application of topological sorting on the optimized BN’s nodes, a label order is derived, which serves as the foundation for constructing the classifier chain model.

Rather than focusing solely on label-specific features or label correlation modeling, some studies have also attempted to combine these two techniques to achieve better classification results. Zhan et al. [32] present LIFTACE, a multi-label learning method that leverages clustering ensemble techniques to generate label-specific features while accounting for label correlations, thus enhancing LIFT’s learning prowess. Zhang et al. [33] integrate partial least squares discriminant analysis (PLS-DA) with an $l_1$-norm penalty, effectively capturing the intricate relationships between variables and labels while simultaneously reducing feature dimensionality. In another study, Liu et al. [34] employ partial least squares (PLS) to identify a latent shared space that bridges both data and label variable spaces, laying the foundation for an efficacious learning model. Ma et al. [35] introduce a label-specific dual graph neural network (LDGN) that harnesses categorical insights to derive label-tailored representations from documents. Central to its approach is a dual Graph Convolution Network (GCN) that comprehensively models the intricate and adaptive interactions among these representations, leveraging both statistical label co-occurrences and dynamic reconstruction graphs in a synergistic fashion.

Although numerous methods have been investigated to incorporate label-specific features and label correlations, many algorithms extract these features within the original feature space, which may not be the most effective approach. Therefore, this paper utilizes a neural network framework to map the correlation-augmented feature space into an embedded feature space. This approach captures the intricate interplay between label-specific features, labels, and their interactions while continuously optimizing label correlations. Ultimately, this leads to more accurate and refined predictions.

3. Algorithm Description

A multi-label classification problem can be defined as follows: $\mathcal{D}=\{\left(x_i,y_i\right){\}}_{i=1}^N$ be a training dataset consisting of N instances, where each instance is associated with $K$ possible category labels ($l_1,...,l_K$). Each instance $x_i$ is represented by a feature vector $x_i={\left[x_{i1},x_{i2},...,x_{im}\right]}^T$ containing $m$ attribute values. The corresponding label set for $x_i$ is denoted by a binary vector $y_i={\left[y_{i1},y_{i2},...,y_{iK}\right]}^T$, where $y_{ij}=1$ indicates that the instance $x_i$ belongs to label $j$, and $y_{ij}=0$ otherwise.

The training dataset can be further represented as data matrix $X=\left[x_1,x_2,\dots ,x_N\right]$ and label matrix $Y=\left[y_1,y_2,\dots ,y_N\right]$. The goal of this paper is to leverage the information contained in $X$ and $Y$ to predict the label sets for unlabeled data points.

3.1. Motivation

Category labels serve as concise representations of intrinsic features, and the coexistence of various features within a sample indicates underlying interdependencies among those labels. Therefore, to attain accurate predictions for data samples’ labels, it is crucial to consider label correlations that possess substantial informational worth. Akin to numerous multi-label classification methodologies, such as CC and DBR, our research explicitly utilizes labels as data attributes. However, the distinction lies in our introduction of innovative approaches that harness label correlations inherent in the data itself rather than merely relying solely on ground-truth labels.

This paper first constructs the label-expanded feature vector $z_i={\left[x_i;y_i\right]}^T$, which is utilized instead of the data vector $x_i$ to train the model. Obviously, $z_i$ contains more distinguishable information than $x_i$, and therefore, training with $z_i$ can achieve better classification performance. However, in multi-label problems, sample data with different category labels may share the same labels, suggesting that indirect relationships may exist between labels. This means that solely studying the combination $z_i$ may not be sufficient, especially when there are missing labels in $y_i$, as $y_i$, the components of ${\mathrm{z}}_{\mathrm{i}}$, contains only the tagged labels of the corresponding instance; that is, $y_i$ can only indicate direct correlations between labels.

To fully leverage labels (including both direct and indirect relationships), this paper introduces a label correlation matrix $\mathcal{K}$. This matrix is used to transform the original labels $y_i$ into $\widehat{{\mathrm{y}}_{\mathrm{i}}}=\mathcal{K}\times y_i$, effectively completing label sets with missing labels by utilizing a matrix [36]. Furthermore, by capturing varying degrees of importance among labels, the transformed labels $\mathcal{K}\times y_i$ enable a stronger association between data points with similar characteristics. Conventionally, the label matrix $\mathcal{K}$ is defined using the normalized cosine similarity, as described in Equation (1):

$${\mathcal{K}}_{pq}=cos\left(y_{\left(p\right)},y_{\left(q\right)}\right)=\frac{<y_{\left(p\right)},y_{\left(q\right)}>}{\parallel y_{\left(p\right)}\parallel \parallel y_{\left(q\right)}\parallel }\hspace{0.17em},$$

(1)

where $p$ and $q$ are category labels, with corresponding row vectors $y_{\left(p\right)}$ and $y_{\left(q\right)}$ in label matrix $Y$. There are many approaches to improve the performance of multi-label classification using a label correlation matrix computed by cosine similarity [37,38]. However, as the amount of data increases and the data types become increasingly diverse, it is difficult to obtain the complete label information of each instance, which results in unreliable label correlations.

Assume that the sample data points $x_1$ and $x_2$ are accompanied by their corresponding label vectors $y_1$ and $y_2$, respectively. These label vectors are defined as

$$y_1=\left[\begin{array}{c}1\\ 0\\ 1\end{array}\right]\hspace{0.17em}\hspace{0.17em}\mathrm{a}\mathrm{n}\mathrm{d}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{y}_2=\left[\begin{array}{c}0\\ 1\\ 0\end{array}\right], \mathrm{respectively},$$

which implies that $x_1$ comprises labels $l_1$ and $l_3$, while $x_2$ solely contains $l_2$. When assessing label correlations via cosine similarity with the original label vectors, it can be deduced that label $l_1$ is related to label $l_3$, whereas label $l_2$ seems to be completely separate. However, the accuracy of label correlations derived from cosine similarity is compromised in scenarios where, despite $x_2$ possessing features indicative of label $l_1$, this label is absent from the label vector ${\mathrm{y}}_2$. Consequently, it becomes imperative to rely on the data matrix, rather than the label matrix, for calculating label correlations whenever missing labels are encountered.

3.2. The Proposed CeFL

Motivated by the desire to enrich the data with supplementary information derived from label correlations and to minimize the influence of missing labels by relying on the data matrix for the computation of these correlations, this paper introduces a novel multi-label learning architecture termed updating Correlation-enhanced Feature Learning (uCeFL). This innovative approach is schematically represented in Figure 1. Subsequently, a detailed explanation of the architecture is provided as follows.

Figure 1. The proposed architecture of uCeFL.

Let $X^p=\left\{x_i\right|\left(x_i,y_i\right)\in D,y_{i,p}=1,i=1,\dots ,N^p\}$ denotes a sample subset with label $p$ in the original dataset D, where $N^p$ is the number of samples in the subset. The covariance matrix similarity coefficient ${\mathcal{K}}_{corr}\in {\mathbb{R}}^{K\times K}$ in Equation (2) is used to obtain a more accurate relationship between labels p and q from the data:

$${\mathcal{K}}_{corr}\left(C_p,C_q\right)=\frac{tr\left(C_p{C}_q\right)}{\parallel C_p{\parallel }_F\parallel C_q{\parallel }_F}\hspace{0.17em},$$

(2)

$tr\{.\}$ is the matrix trace, and $\parallel .{\parallel }_F$ is the Frobenius norm, ${\mathcal{K}}_{corr}\left(C_p,C_q\right)=1-d_{corr}\left(C_p,C_q\right)$, where $d_{corr}$ is the correlation matrix distance [39], which becomes zero if the covariance matrices are equal up to a scaling factor and 1 if they differ to a maximum extent. $C_p$ and $C_q$ are the covariance matrices of sets $X^p$ and $X^q$, respectively. The covariance matrix is actually a point on the Riemannian manifold, so ${\mathcal{K}}_{corr}\left(C_p,C_q\right)$ can be calculated by the log-Euclidean metric [40], i.e., logarithmically mapping the points from the Riemannian manifold to the Euclidean space and calculating their correlation.

Without prior assumptions on the data distribution, the covariance matrix provides a natural representation for a dataset with any type of features and with an arbitrary number of samples, which can effectively distinguish data subsets with different category labels by encoding the feature-related information specific to each object class. The higher the degree of feature overlap between two subsets, the closer the relationship between the category labels and the greater the value of Equation (2). If the label matrix is used directly to calculate the label correlations, the feature overlap between sample sets cannot be truly represented, while the label correlation may be distorted to some extent when labels are missing. In short, by utilizing a covariance matrix to model the data subset, the issue of label correlation calculation can be resolved, even in datasets with incomplete labeling. Consequently, the label-expanded feature vector $z_i$ is updated to $z_i={\left[x_i;\widehat{y_i}\right]}^T$, where $\widehat{y_i}={\mathcal{K}}_{corr}\times y_i$ represents the revised label information.

As the feature space becomes larger, the data structure becomes more complicated, and this paper uses the multi-layer neural network to study the label-enhanced dataset ${\mathcal{D}}^{\prime }=\{\left(z_i,y_i\right){\}}_{i=1}^N$ because the neural network can simulate the nonlinear manifold structure of the data with no assumptions. If there are L + 1 layers in the neural network, for feature $z_i$, the output $h_i^l$ in the $l$-th layer is calculated as Equation (3):

$$\left\{\begin{array}{ll}{h}_i^1=z_i,& \\ h_i^l=f\left(W^{\left(l-1\right)}{h}_i^{l-1}+b^{\left(l-1\right)}\right),& 2\le l\le L,\\ h_i^{L+1}=\left[\begin{array}{c}P\left(y_{i,1}=1|x_i;\theta \right)\\ ⋮\\ P\left(y_{i,K}=1|x_i;\theta \right)\end{array}\right],& \end{array}\right.$$

(3)

where the first layer is the input layer, and the input is also the output of this layer. The symbol $W^{\left(l\right)}$ denotes the connection parameter between layers $l$ and $l$ + 1, while $b^{\left(l\right)}$ represents the bias term; both of these parameters are learned at the subsequent layer (i.e., layer $l$ + 1). Additionally, $f$ signifies the activation function employed in the hidden layer, which can be a sigmoid function or any other nonlinear function. To address the multi-label problem, softmax regression, trained via batch gradient descent, is utilized. The objective function of this model is described by Equation (4):

$$J\left(\theta \right)=-\frac{1}{n}\left[{\sum }_{i=1}^n{\sum }_{k=1}^{K}1\{y_{i,k}=1\}logP\left(y_{i,k}=1|x_i;\theta \right)\right]+\frac{\lambda }{2}{\sum }_{k=1}^K{\left({\theta }^{\left(k\right)}\right)}^2\hspace{0.17em},$$

(4)

wherer each output item $P\left(y_{i,k}=1|x_i;\theta \right)=\frac{exp\left({\theta }^{\left(k\right)T}{x}_i\right)}{\sum _{j=1}^{K}exp\left({\theta }^{\left(j\right)T}{x}_i\right)}$ in layer L + 1 is the probability that label $k\in \left(1,K\right)$ is assigned to data $x_i$, $\lambda$ is a tradeoff parameter, and ${\theta }^{\left(k\right)}$ is the row vector of matrix $W^{\left(L\right)}$ (i.e., $W^{\left(L\right)}=\left[{\theta }^{\left(1\right)T};{\theta }^{\left(2\right)T};...;{\theta }^{\left(K\right)T}\right]$).

By utilizing a neural network with concealed layers, the process of acquiring label-expanded data features becomes layered and structured. This hierarchical learning approach ensures that the high-level semantic features of the data are extracted efficiently and accurately. These high-level features represent a fusion of three distinct types of relationships: relationships among data features, relationships connecting labels with data features, and pure relationships between labels. This fusion of relationships results in features that are highly informative and beneficial for subsequent studies of classification models.

As the learning process continues, the augmented features fed into the network are continuously updated with the aid of learned parameters $W$ and $b$. This updating process ensures that the network adapts to the evolving data distribution and captures the most relevant features for classification. Specifically, after each iteration, a feedforward pass is performed. This pass computes the output $h_{X^k}^L$ for each subset $X^k$ (where k ranges from 1 to K) using the updated parameters $W$ and $b$ learned during that iteration. This output represents the network’s understanding of the data based on the current set of parameters. Subsequently, the covariance matrix similarity coefficient ${\mathcal{K}}_{corr}$ is recalculated. This coefficient measures the similarity between different subsets of data based on their covariance matrices. It captures the underlying relationships among the data (or labels) and is crucial for the next iteration, where it will be used to further refine the learning process. By iteratively updating the network parameters, recalculating the covariance matrix similarity coefficient, and performing feedforward passes, the neural network gradually learns to extract meaningful high-level semantic features from the label-expanded data. This process enables the network to adapt to complex datasets and produce accurate classification results.

Based on the preceding analysis, the optimization of the uCeFL can be achieved through gradient descent, and the corresponding pseudo code is outlined in Algorithm 1. Additionally, a variation called CeFL is presented, which is described thoroughly in Algorithm 2. CeFL relies solely on Equation (2) to compute the correlation matrix ${\mathcal{K}}_{corr}$ of the original dataset, thus negating the need for frequent updates of the extended features. However, it must be emphasized that if ${\mathcal{K}}_{corr}$ remains static, the procedural steps outlined in Algorithm 2 may not fully capture the label propagation effects and the inherent neural network benefits exhibited by uCeFL.

Algorithm 1: uCeFL

Input: $\mathrm{Training} \mathrm{data} \mathrm{matrix} X=\left[x_1,\dots ,x_n\right]$$, \mathrm{label} \mathrm{matrix} Y=\left[y_1,\dots ,y_n\right]$,             $\mathrm{network} \mathrm{layer} \mathrm{number} L+1$$, \mathrm{iterative} \mathrm{number} T$$, \mathrm{convergence} \mathrm{error} \epsilon$; Output: $\mathrm{Parameters} W_{\left(1\le l\le L\right)}^{\left(l\right)}$$\mathrm{and} b_{\left(1\le l<L\right)}^{\left(l\right)}$. 1. $\mathrm{Initialize} W_{\left(1\le l\le L\right)}^{\left(l\right)}$$\mathrm{and} b_{\left(1\le l<L\right)}^{\left(l\right)}$ with appropriate values; 2. FirstFlag = true; 3. for each label k do 4.       Construct the subset of training data associated with label k as $X^k=\left\{x_i\right|x_i\in X,y_{i,k}=1\}$; 5. end for 6. for t = 1, 2, … T do 7.       for each label k do 8.             if FirstFlag = false then 9.                   $Z^k=\left[X^k;{\mathcal{K}}_{corr}\times Y^k\right]$; 10.                 $h_{X^k}^1=Z^k$; 11.                 $h_{X^k}^l=f\left(W^{\left(l-1\right)}{h}_{X^k}^{l-1}+b^{\left(l-1\right)}\right)\hspace{0.17em},1\le l\le L$; 12.           else 13.                 $h_{X^k}^L=X^k$; 14.                 FirstFlag = false; 15.           end if 16.           Compute covariance matrix $C_{h_{X^k}^L}$ of each $h_{X^k}^L$; 17.       end for 18.       Calculate the Covariance Matrix Similarity Coefficient ${\mathcal{K}}_{corr}$ by Equation (2); 19.       Train the neural network using $Z=\left[X;{\mathcal{K}}_{corr}\times Y\right]$ as input, then update $W_{\left(1\le l\le L\right)}^{\left(l\right)}$ and $b_{\left(1\le l<L\right)}^{\left(l\right)}$; 20.        if $t>T$ or $\left|{J\left(W^L\right)}_t-{J\left(W^L\right)}_{t-1}\right|<\epsilon$ then 21.             Turn to return; 22.       end if 23.   end for 24. return Parameters $W_{\left(1\le l\le L\right)}^{\left(l\right)}$ and $b_{\left(1\le l<L\right)}^{\left(l\right)}$.

Algorithm 2: CeFL

Input: Training data matrix $X=\left[x_1,\dots ,x_n\right]$, label matrix $Y=\left[y_1,\dots ,y_n\right]$,             network layer number $L+1$, iterative number $T$, convergence error $\epsilon$; Output: Parameters $W_{\left(1\le l\le L\right)}^{\left(l\right)}$ and $b_{\left(1\le l<L\right)}^{\left(l\right)}$. 1. Initialize $W_{\left(1\le l\le L\right)}^{\left(l\right)}$ and $b_{\left(1\le l<L\right)}^{\left(l\right)}$ with appropriate values; 2. for each label $k$ do 3.       Construct the subset of training data associated with label $\mathrm{k}$ as $X^k=\left\{x_i\right|x_i\in X,y_{i,k}=1\}$; 4.       Compute the covariance matrix $C_{X^k}$ of matrix $X^k$; 5. end for 6. Calculate the Covariance Matrix Similarity Coefficient ${\mathcal{K}}_{corr}$ by Equation (2); 7. for $t=\mathrm{1,2},\dots T$ do 8.       Train the neural network using $Z=\left[X;{\mathcal{K}}_{corr}\times Y\right]$ as input, then update $W_{\left(1\le l\le L\right)}^{\left(l\right)}$ and $b_{\left(1\le l<L\right)}^{\left(l\right)}$; 9.       if $t>T$ or $\left|{J\left(W^L\right)}_t-{J\left(W^L\right)}_{t-1}\right|<\epsilon$ then 10.             Turn to return; 11.     end if 12. end for 13. return Parameters $W_{\left(1\le l\le L\right)}^{\left(l\right)}$ and $b_{\left(1\le l<L\right)}^{\left(l\right)}$.

4. Experiments

To validate the efficacy of CeFL and uCeFL, seven diverse multi-label datasets spanning domains such as music, biology, images, and text were carefully selected. Table 1 concisely outlines the essential characteristics of these datasets. Most of these datasets can be conveniently downloaded from Mulan (http://mulan.sourceforge.net/datasets.html (accessed on 15 February 2024)), with the exception of the education dataset, which is obtainable from Yahoo [7] (https://www.lamda.nju.edu.cn/code_MDDM.ashx (accessed on 20 February 2024)). Cardinality indicates the average number of labels per instance, while attributes refer to the fact that all features belong to one of two categories: nominal or numeric. To assess the performance of our methods, they were benchmarked against established approaches such as ML-KNN [41], TRAM [42], LLFS-DL [43], and LCFM [44]. Notably, the use of a label-expanded feature space as the basis for training inputs is a commonality shared by our approaches and LLFS-DL. This common ground facilitated meaningful comparisons and the evaluation of the relative strengths of each method.

Table 1. Characteristics of the datasets used in the experiments.

		Attributes
Dataset	Instances	Nominal	Numeric	Labels	Cardinality
emotions	593	0	72	6	1.869
scene	2407	0	294	6	1.074
corel5k	5000	499	0	374	3.522
enron	1702	1001	0	53	3.378
education	5000	550	0	33	1.461
genbase	662	1186	0	27	1.252
medical	978	1449	0	45	1.245

ML-KNN was developed from traditional k-nearest neighbor (KNN) problem transformation methods. For each test instance, its k-nearest neighbors are found in the training set. Based on statistical information from these neighbors, such as the number belonging to the same label category, the maximum a posteriori (MAP) principle is employed to determine the label set of the test instance. Although ML-KNN does not consider label correlation, it often outperforms many multi-label approaches.

TRAM examines the intricate composition of label concepts, considering the varying degrees of typicality that individual examples exhibit with respect to specific labels. Initially, TRAM builds a similarity matrix by determining the $k$-nearest neighbor relationships for all samples, labeled and unlabeled, within the dataset. This step aims to capture the inherent connections and similarities among the data points. Subsequently, TRAM transforms the transductive multi-label learning challenge into an optimization problem, where the focus is on refining the definition and representation of label concept combinations for each multi-label instance through the optimization process. Leveraging the estimated label concept combination details, TRAM then utilizes the label concept combinations of labeled samples, coupled with their corresponding ground-truth labels, to solve an optimization function that admits a closed-form solution. The objective of this optimization is to derive a transformation matrix $P$, which will serve as a crucial tool in assigning labels to unlabeled samples in subsequent stages.

LLFS-DL utilizes the label matrix $Y$ as additional features to augment the feature space $X$ and supposes that each class label is only determined by a subset of relevant features from a given dataset. Unlike DBR, LLFS-DL does not directly increase the dimension of data features to learn a model. Instead, it solves a variation of the $l_1$-norm regularization, incorporating both second-order and high-order label correlations in a sparse stacking manner to improve multi-label classification.

LCFM aims to address label prediction when labels are missing, a goal similar to ours. It combines global and local label features to learn correlations and restore missing labels. The approach utilizes sparse Lasso regression and $l_2$ regularization for featureselection, optimizing the model through binary cross-entropy loss and the APG algorithm.

4.1. Experimental Criteria

In the experiments, six performance metrics [45] were chosen to evaluate the performance of several multi-label classification approaches. Let $\mathcal{T}=\{\left(x_i,y_i\right){\}}_{i=1}^P$ be the test dataset, $f\left(.\right)$ be the trained model, and $K$ be the number of labels. The indicators are defined as follows.

• HammingLoss is an example-based multi-label classification evaluation criterion that indicates the proportion of misclassified instances in two scenarios: when predicted labels do not belong to the instance, and when labels belonging to the instance are not predicted,

  $$\mathrm{HammingLoss}=\frac{1}{P}{\sum }_{i=1}^P\frac{y_i\oplus f\left(x_i\right)}{K},$$

  where $\oplus$ indicates the symmetric difference between the predicted and actual label vectors for each instance.
• SubsetAccuracy is a measure of classification accuracy. It considers a classification to be correct when the set of predicted labels exactly matches the set of true labels, and incorrect otherwise. Specifically, SubsetAccuracy calculates the proportion of instances in the test set that are fully and accurately classified,

  $$\mathrm{SubsetAccuracy}=\frac{1}{P}{\sum }_{i=1}^{P}I\left(f\left(x_i\right)=y_i\right),$$

  where $I\left(.\right)$ is the indicator function. This metric provides a holistic view of the performance of a classification model with respect to predicting entire sets of labels, rather than just individual labels.
• Accuracy is the ratio of correctly predicted labels to the total number of predicted labels:

  $$\mathrm{Accuracy}=\frac{1}{P}{\sum }_{i=1}^P\frac{\left|y_i\cap f\left(x_i\right)\right|}{\left|y_i\cup f\left(x_i\right)\right|}.$$
• F1Measure is a comprehensive evaluation index that combines precision and recall, and is also referred to as the comprehensive classification rate. The precision rate measures how many of the predicted labels are actually correct,
  $$\mathrm{Precision}=\frac{1}{P}{\sum }_{i=1}^P\frac{\left|y_i\cap f\left(x_i\right)\right|}{\left|f\left(x_i\right)\right|}.$$

  The recall rate is the ratio of correctly predicted positive labels to the total number of actual positive labels,

  $$\mathrm{Recall}=\frac{1}{P}{\sum }_{i=1}^P\frac{\left|y_i\cap f\left(x_i\right)\right|}{\left|y_i\right|}.$$
  The F1Measure is

  $$\mathrm{F}1\mathrm{Measure}=\frac{2\times \mathrm{P}\mathrm{r}\mathrm{e}\mathrm{s}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}\times \mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}}{\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{s}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}+\mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}}.$$

  These evaluation metrics are example-based. We also consider label-based evaluation metrics, which are widely used in the literature. Let $T{P}_j$, $T{N}_j$, $F{P}_j$, and $F{N}_j$ represent the number of true positives, true negatives, false positives, and false negatives, respectively, for label $l_{j,j\in \left(1,K\right)}$.

  $${\mathrm{Precision}}_j=\frac{T{P}_j}{T{P}_j+F{P}_j},$$

  $${\mathrm{Recall}}_j=\frac{T{P}_j}{T{P}_j+F{N}_j}$$
• MacroF1Measure is the arithmetic average value of the F1Measure of all labels.

  $$\mathrm{MacroF}1\mathrm{Measure}=\frac{1}{K}\sum _{j=1}^K\frac{2\times {\mathrm{Precision}}_j\times {\mathrm{Recall}}_j}{{\mathrm{Precision}}_j+{\mathrm{Recall}}_j}.$$
• MicroF1Measure sums the Precision_j and Recall_j of all labels before calculating the F1measure.

  $$\mathrm{MicroF}1\mathrm{Measure}=\frac{2\times \sum _{j=1}^K{\mathrm{Precision}}_j\times \sum _{j=1}^K{\mathrm{Recall}}_j}{\sum _{j=1}^K{\mathrm{Precision}}_j+\sum _{j=1}^K{\mathrm{Recall}}_j}.$$

  For all these metrics except HammingLoss, a larger value indicates better classifier performance.

4.2. Implementation Details

The experiment was conducted on a computer that was equipped with an Intel(R) Core(TM) i7-10510U CPU running at 1.80GHz (base clock speed) up to 2.30 GHz (turbo boost speed), and had 16.0 GB of RAM. The MATLAB software version used for the experiment was R2018a, and the experiment was performed in a Microsoft Windows 10 operating system environment.

For the four comparison methods, default settings were applied. The CeFL and uCeFL approaches encounter a similar challenge to the LLSF-DL algorithm, which is the absence of initial label assignments for the test samples, and this obstacle hinders the utilization of a neural network. To align with the input space of $z_i$, certain existing methods were employed to provide initial predictions for the test data $x_t$. Specifically, KNN was adopted to annotate each test instance with the label set identical to that of the nearest training instance, with $k$ set to 1.

When calculating $\widehat{y_i}={\mathcal{K}}_{corr}\times y_i$, both our algorithms need to input it into a logistic function first before expanding the attribute space. For the five nominal datasets of corel5k, enron, education, genbase, and medical, each item of $\widehat{y_i}$ must continue to be reset to 0 or 1. The strategy is to set it to 1 when the item is greater than the average and otherwise to 0.

The neural network in this work is comprised of two hidden layers. The first layer had 300 neurons, while the second layer had 120 neurons. Both hidden layers employed a sigmoid activation function. The network underwent training via the L$\_$BFGS method, which was implemented with the minFunc (gradient descent toolbox). The regularization parameter $\lambda$ in Equation (4) was assigned a value of 0.0002.

4.3. Performance Comparison and Discussion

In the experiment, 10-fold cross-validation was performed on each dataset to evaluate the performance of each algorithm. For each experiment, 1/4 of the dataset was randomly selected as test data, and the remaining data was used for training. To assess the effectiveness of uCeFL in scenarios with incomplete labels, the observed labels were randomly removed from the training data at a preset missing rate that ranged from 0 to 0.6, incremented in steps of 0.2. Also, to prevent empty classes or instances without positive labels, at least one instance was retained for each class label, and each instance was ensured to have at least one positive class label.

The average performance of 10 iterations was used to evaluate the performance of each algorithm. Table 2, Table 3, Table 4, Table 5, Table 6, Table 7 and Table 8 display the experimental results (including the standard deviation) of the six evaluation indicators for each algorithm on the seven datasets. The symbol ↓ signifies that the smaller the value of an indicator, the better the performance, whereas ↑ indicates that a larger value signifies better performance. The optimal results within each row are highlighted in bold.

Table 2. Comparison results (mean ± std) of six approaches over the emotions dataset.

Emotions
Missing rate: 0	ML-KNN	TRAM	LLFS-DL	LCFM	CeFL	uCeFL
HammingLoss (↓)	0.2723 ± 0.0049	0.2986 ± 0.0035	0.3139 ± 0.0033	0.4083 ± 0.0101	0.2617 ± 0.0056	0.2584 ± 0.0045
SubsetAccuracy (↑)	0.1105 ± 0.0198	0.1467 ± 0.0736	0.0000 ± 0.0000	0.0051 ± 0.0058	0.1847 ± 0.0335	0.1912 ± 0.0369
Accuracy (↑)	0.3016 ± 0.0119	0.4152 ± 0.0132	0.0000 ± 0.0000	0.2735 ± 0.0083	0.4378 ± 0.0116	0.4398 ± 0.0178
F1measure (↑)	0.3667 ± 0.0112	0.5112 ± 0.0091	0.0000 ± 0.0000	0.3970 ± 0.0119	0.5231 ± 0.0105	0.5227 ± 0.0138
MacroF1measure (↑)	0.3511 ± 0.0010	0.4963 ± 0.0058	0.0000 ± 0.0000	0.2161 ± 0.0089	0.5403 ± 0.0078	0.5599 ± 0.0186
MicroF1Measure (↑)	0.4343 ± 0.0039	0.5265 ± 0.0099	0.0000 ± 0.0000	0.3975 ± 0.0082	0.5575 ± 0.0013	0.5667 ± 0.0089
Missing rate: 0.2	ML-KNN	TRAM	LLFS-DL	LCFM	CeFL	uCeFL
HammingLoss (↓)	0.2835 ± 0.0077	0.4048 ± 0.0077	0.3116 ± 0.0011	0.4101 ± 0.0311	0.2607 ± 0.0068	0.2555 ± 0.0107
SubsetAccuracy (↑)	0.0838 ± 0.0239	0.0909 ± 0.0028	0.0000 ± 0.0000	0.0051 ± 0.0043	0.1878 ± 0.0136	0.1947 ± 0.0037
Accuracy (↑)	0.1986 ± 0.0085	0.2345 ± 0.0088	0.0000 ± 0.0000	0.2741 ± 0.0151	0.3996 ± 0.0068	0.4116 ± 0.0213
F1measure (↑)	0.2415 ± 0.0032	0.2901 ± 0.0095	0.0000 ± 0.0000	0.3966 ± 0.0195	0.4739 ± 0.0101	0.4875 ± 0.0318
MacroF1measure (↑)	0.2088 ± 0.0067	0.1646 ± 0.0081	0.0000 ± 0.0000	0.2005 ± 0.0241	0.5218 ± 0.0191	0.5363 ± 0.0259
MicroF1Measure (↑)	0.2910 ± 0.0011	0.2887 ± 0.0143	0.0000 ± 0.0000	0.3952 ± 0.0151	0.5301 ± 0.0166	0.5443 ± 0.0211
Missing rate: 0.4	ML-KNN	TRAM	LLFS-DL	LCFM	CeFL	uCeFL
HammingLoss (↓)	0.2946 ± 0.0038	0.3897 ± 0.0126	0.3053 ± 0.0077	0.3958 ± 0.0195	0.2684 ± 0.0014	0.2613 ± 0.0075
SubsetAccuracy (↑)	0.0471 ± 0.0131	0.0502 ± 0.0151	0.0000 ± 0.0000	0.0051 ± 0.0052	0.1308 ± 0.0054	0.1443 ± 0.0111
Accuracy (↑)	0.0944 ± 0.0183	0.1561 ± 0.0211	0.0000 ± 0.0000	0.2798 ± 0.0136	0.3426 ± 0.0089	0.3486 ± 0.0304
F1measure (↑)	0.1123 ± 0.0187	0.1932 ± 0.0235	0.0000 ± 0.0000	0.4031 ± 0.0212	0.4158 ± 0.0122	0.4186 ± 0.0241
MacroF1measure (↑)	0.1335 ± 0.0190	0.0923 ± 0.0177	0.0000 ± 0.0000	0.2015 ± 0.0262	0.4433 ± 0.0212	0.4575 ± 0.0011
MicroF1Measure (↑)	0.1512 ± 0.0121	0.1943 ± 0.0222	0.0000 ± 0.0000	0.4018 ± 0.0204	0.4687 ± 0.0138	0.4819 ± 0.0112
Missing rate: 0.6	ML-KNN	TRAM	LLFS-DL	LCFM	CeFL	uCeFL
HammingLoss (↓)	0.3093 ± 0.0028	0.3836 ± 0.0018	0.3105 ± 0.0009	0.4055 ± 0.0054	0.3045 ± 0.0214	0.2986 ± 0.0178
SubsetAccuracy (↑)	0.0067 ± 0.0067	0.0536 ± 0.0071	0.0000 ± 0.0000	0.0153 ± 0.0152	0.0804 ± 0.0221	0.0873 ± 0.0204
Accuracy (↑)	0.0229 ± 0.0229	0.1425 ± 0.0069	0.0000 ± 0.0000	0.2933 ± 0.0201	0.2458 ± 0.0428	0.2621 ± 0.0379
F1measure (↑)	0.0285 ± 0.0285	0.1738 ± 0.0066	0.0000 ± 0.0000	0.4151 ± 0.0200	0.3050 ± 0.0480	0.3254 ± 0.0424
MacroF1measure (↑)	0.0406 ± 0.0406	0.0685 ± 0.0009	0.0000 ± 0.0000	0.2585 ± 0.0336	0.3330 ± 0.0312	0.3457 ± 0.0355
MicroF1Measure (↑)	0.0403 ± 0.0403	0.1734 ± 0.0074	0.0000 ± 0.0000	0.4187 ± 0.0178	0.3539 ± 0.04521	0.3667 ± 0.0454

Table 3. Comparison results (mean ± std) of six approaches over the scene dataset.

Scene
Missing rate: 0	ML-KNN	TRAM	LLFS-DL	LCFM	CeFL	uCeFL
HammingLoss (↓)	0.0868 ± 0.0021	0.0946 ± 0.0010	0.1774 ± 0.0032	0.4016 ± 0.02574	0.1013 ± 0.0023	0.1012 ± 0.0018
SubsetAccuracy (↑)	0.6295 ± 0.0076	0.6918 ± 0.0059	0.0000 ± 0.0000	0.0102 ± 0.0131	0.6685 ± 0.0081	0.6654 ± 0.0057
Accuracy (↑)	0.6743 ± 0.0055	0.7201 ± 0.0015	0.0000 ± 0.0000	0.2773 ± 0.0144	0.7003 ± 0.0085	0.7003 ± 0.0077
F1measure (↑)	0.6896 ± 0.0047	0.7295 ± 0.0003	0.0000 ± 0.0000	0.3985 ± 0.0122	0.7113 ± 0.0081	0.7123 ± 0.0081
MacroF1measure (↑)	0.7408 ± 0.0041	0.7325 ± 0.0021	0.0000 ± 0.0000	0.2116 ± 0.0077	0.7121 ± 0.0088	0.7158 ± 0.0078
MicroF1Measure (↑)	0.7358 ± 0.0048	0.7256 ± 0.0021	0.0000 ± 0.0000	0.3976 ± 0.0138	0.7071 ± 0.0071	0.7088 ± 0.0068
Missing rate: 0.2	ML-KNN	TRAM	LLFS-DL	LCFM	CeFL	uCeFL
HammingLoss (↓)	0.0891 ± 0.0049	0.2835 ± 0.0068	0.1803 ± 0.0018	0.4215 ± 0.0347	0.1207 ± 0.0075	0.1194 ± 0.0031
SubsetAccuracy (↑)	0.6195 ± 0.0315	0.1596 ± 0.0248	0.0000 ± 0.0000	0.0016 ± 0.0069	0.5634 ± 0.0267	0.5665 ± 0.0123
Accuracy (↑)	0.6718 ± 0.0334	0.1741 ± 0.0222	0.0000 ± 0.0000	0.2768 ± 0.0064	0.5945 ± 0.0265	0.5977 ± 0.0121
F1measure (↑)	0.6897 ± 0.0319	0.1785 ± 0.0224	0.0000 ± 0.0000	0.4002 ± 0.0121	0.6046 ± 0.0261	0.6081 ± 0.0123
MacroF1measure (↑)	0.7410 ± 0.0117	0.0591 ± 0.0013	0.0000 ± 0.0000	0.2151 ± 0.0275	0.6437 ± 0.0222	0.6483 ± 0.0075
MicroF1Measure (↑)	0.7314 ± 0.0203	0.1813 ± 0.0189	0.0000 ± 0.0000	0.3992 ± 0.0135	0.6334 ± 0.0238	0.6373 ± 0.0124
Missing rate: 0.4	ML-KNN	TRAM	LLFS-DL	LCFM	CeFL	uCeFL
HammingLoss (↓)	0.1199 ± 0.0017	0.2712 ± 0.0021	0.1787 ± 0.0034	0.4042 ± 0.0229	0.1390 ± 0.0064	0.1338 ± 0.0054
SubsetAccuracy (↑)	0.3928 ± 0.0173	0.1237 ± 0.0015	0.0000 ± 0.0000	0.0052 ± 0.0012	0.4552 ± 0.0205	0.4658 ± 0.0015
Accuracy (↑)	0.4186 ± 0.0157	0.1317 ± 0.0066	0.0000 ± 0.0000	0.2774 ± 0.0042	0.4823 ± 0.0267	0.4948 ± 0.0049
F1measure (↑)	0.4273 ± 0.0187	0.1344 ± 0.0023	0.0000 ± 0.0000	0.4011 ± 0.0111	0.4918 ± 0.0226	0.5043 ± 0.0101
MacroF1measure (↑)	0.5433 ± 0.0064	0.0458 ± 0.0028	0.0000 ± 0.0000	0.2047 ± 0.0233	0.5567 ± 0.0235	0.5743 ± 0.0088
MicroF1Measure (↑)	0.5542 ± 0.0101	0.1463 ± 0.0059	0.0000 ± 0.0000	0.4015 ± 0.0065	0.5502 ± 0.0259	0.5662 ± 0.0056
Missing rate: 0.6	ML-KNN	TRAM	LLFS-DL	LCFM	CeFL	uCeFL
HammingLoss (↓)	0.1692 ± 0.0028	0.2137 ± 0.0017	0.1797 ± 0.0014	0.4002 ± 0.0201	0.1511 ± 0.0058	0.1493 ± 0.0066
SubsetAccuracy (↑)	0.0682 ± 0.0106	0.0425 ± 0.0055	0.0000 ± 0.0000	0.0102 ± 0.0065	0.4193 ± 0.0175	0.4229 ± 0.0126
Accuracy (↑)	0.0715 ± 0.0101	0.0458 ± 0.0082	0.0000 ± 0.0000	0.2846 ± 0.0229	0.4498 ± 0.0119	0.4528 ± 0.0101
F1measure (↑)	0.0726 ± 0.0078	0.0468 ± 0.0087	0.0000 ± 0.0000	0.4048 ± 0.0351	0.4602 ± 0.0101	0.4632 ± 0.0081
MacroF1measure (↑)	0.1263 ± 0.0149	0.0337 ± 0.0038	0.0000 ± 0.0000	0.2552 ± 0.0216	0.5187 ± 0.0155	0.5233 ± 0.0141
MicroF1Measure (↑)	0.1283 ± 0.0167	0.0710 ± 0.0132	0.0000 ± 0.0000	0.4125 ± 0.0310	0.5158 ± 0.0136	0.5203 ± 0.0146

Table 4. Comparison results (mean ± std) of six approaches over the corel5k dataset.

Corel5k
Missing rate: 0	ML-KNN	TRAM	LLFS-DL	LCFM	CeFL	uCeFL
HammingLoss (↓)	0.0093 ± 0.0021	0.0153 ± 0.0001	0.0097 ± 0.0011	0.3380 ± 0.0013	0.0149 ± 0.0001	0.0149 ± 0.0011
SubsetAccuracy (↑)	0.0041 ± 0.0015	0.0071 ± 0.0005	0.0061 ± 0.0025	0.0000 ± 0.0000	0.0105 ± 0.0044	0.0113 ± 0.0001
Accuracy (↑)	0.0134 ± 0.0014	0.1137 ± 0.0019	0.0161 ± 0.0015	0.0224 ± 0.0001	0.1284 ± 0.0023	0.1313 ± 0.0049
F1measure (↑)	0.0172 ± 0.0009	0.1701 ± 0.0028	0.0191 ± 0.0005	0.0436 ± 0.0003	0.1885 ± 0.0043	0.1926 ± 0.0027
MacroF1measure (↑)	0.0101 ± 0.0053	0.0281 ± 0.0022	0.0066 ± 0.0011	0.0219 ± 0.0001	0.0476 ± 0.0029	0.0522 ± 0.0017
MicroF1Measure (↑)	0.0292 ± 0.0017	0.1706 ± 0.0015	0.0334 ± 0.0016	0.0434 ± 0.0003	0.1906 ± 0.0033	0.1954 ± 0.0014
Missing rate: 0.2	ML-KNN	TRAM	LLFS-DL	LCFM	CeFL	uCeFL
HammingLoss (↓)	0.0093 ± 0.0011	0.0154 ± 0.0012	0.0096 ± 0.0011	0.3357 ± 0.0009	0.0133 ± 0.0021	0.0134 ± 0.0004
SubsetAccuracy (↑)	0.0021 ± 0.0002	0.0004 ± 0.0034	0.0027 ± 0.0052	0.0000 ± 0.0000	0.0055 ± 0.0087	0.0063 ± 0.0056
Accuracy (↑)	0.0128 ± 0.0033	0.0405 ± 0.0055	0.0098 ± 0.0039	0.0220 ± 0.0003	0.0963 ± 0.0089	0.0933 ± 0.0075
F1measure (↑)	0.0174 ± 0.0055	0.0656 ± 0.0088	0.0119 ± 0.0011	0.0428 ± 0.0006	0.1427 ± 0.0055	0.1384 ± 0.0062
MacroF1measure (↑)	0.0087 ± 0.0028	0.0026 ± 0.0023	0.0042 ± 0.0054	0.0212 ± 0.0002	0.0387 ± 0.0000	0.0372 ± 0.0022
MicroF1Measure (↑)	0.0283 ± 0.0031	0.0696 ± 0.0082	0.0227 ± 0.0073	0.0426 ± 0.0006	0.1558 ± 0.0099	0.1515 ± 0.0054
Missing rate: 0.4	ML-KNN	TRAM	LLFS-DL	LCFM	CeFL	uCeFL
HammingLoss (↓)	0.0096 ± 0.0012	0.0143 ± 0.0011	0.0092 ± 0.0001	0.3247 ± 0.0021	0.0143 ± 0.0019	0.0142 ± 0.0019
SubsetAccuracy (↑)	0.0004 ± 0.0011	0.0000 ± 0.0000	0.0005 ± 0.0012	0.0000 ± 0.0000	0.0043 ± 0.0084	0.0033 ± 0.0021
Accuracy (↑)	0.0112 ± 0.0045	0.0242 ± 0.0015	0.0013 ± 0.0011	0.0226 ± 0.0002	0.0824 ± 0.0021	0.0801 ± 0.0056
F1measure (↑)	0.0168 ± 0.0065	0.0389 ± 0.0031	0.0012 ± 0.0088	0.0441 ± 0.0004	0.1226 ± 0.0012	0.1197 ± 0.0063
MacroF1measure (↑)	0.0013 ± 0.0001	0.0013 ± 0.0012	0.0003 ± 0.0094	0.0220 ± 0.0004	0.0331 ± 0.0021	0.0327 ± 0.0011
MicroF1Measure (↑)	0.0202 ± 0.0089	0.0436 ± 0.0021	0.0023 ± 0.0012	0.0437 ± 0.0004	0.1296 ± 0.0081	0.1283 ± 0.0067
Missing rate: 0.6	ML-KNN	TRAM	LLFS-DL	LCFM	CeFL	uCeFL
HammingLoss (↓)	0.0097 ± 0.0052	0.0127 ± 0.0012	0.0091 ± 0.0091	0.2996 ± 0.0025	0.0143 ± 0.0004	0.0143 ± 0.0015
SubsetAccuracy (↑)	0.0005 ± 0.0001	0.0000 ± 0.0000	0.0000 ± 0.0000	0.0000 ± 0.0000	0.0012 ± 0.0014	0.0025 ± 0.0007
Accuracy (↑)	0.0087 ± 0.0059	0.0072 ± 0.0052	0.0000 ± 0.0000	0.0239 ± 0.0003	0.0634 ± 0.0011	0.0646 ± 0.0027
F1measure (↑)	0.0132 ± 0.0117	0.0118 ± 0.0087	0.0000 ± 0.0000	0.0466 ± 0.0006	0.0968 ± 0.0055	0.0981 ± 0.0067
MacroF1measure (↑)	0.0012 ± 0.0003	0.0006 ± 0.0013	0.0000 ± 0.0000	0.0227 ± 0.0003	0.0235 ± 0.0022	0.0252 ± 0.0001
MicroF1Measure (↑)	0.0155 ± 0.0135	0.0138 ± 0.0021	0.0000 ± 0.0000	0.0461 ± 0.0005	0.1022 ± 0.0017	0.1035 ± 0.0012

Table 5. Comparison results (mean ± std) of six approaches over the enron dataset.

Enron
Missing rate: 0	ML-KNN	TRAM	LLFS-DL	LCFM	CeFL	uCeFL
HammingLoss (↓)	0.0523 ± 0.0006	0.0509 ± 0.0005	0.0583 ± 0.0006	0.2778 ± 0.0021	0.0573 ± 0.0016	0.0563 ± 0.0014
SubsetAccuracy (↑)	0.0353 ± 0.0095	0.1213 ± 0.0059	0.0163 ± 0.0046	0.0000 ± 0.0000	0.0903 ± 0.0200	0.1059 ± 0.0071
Accuracy (↑)	0.2883 ± 0.0340	0.4723 ± 0.0005	0.3470 ± 0.0030	0.1743 ± 0.0015	0.3623 ± 0.0235	0.3819 ± 0.0077
F1measure (↑)	0.3813 ± 0.0356	0.5923 ± 0.0008	0.4723 ± 0.0069	0.2880 ± 0.0023	0.4674 ± 0.0226	0.4903 ± 0.0100
MacroF1measure (↑)	0.0850 ± 0.0070	0.1873 ± 0.0150	0.0638 ± 0.0006	0.1418 ± 0.0050	0.1754 ± 0.0021	0.1878 ± 0.0107
MicroF1Measure (↑)	0.4548 ± 0.0356	0.5890 ± 0.0002	0.5038 ± 0.0029	0.2883 ± 0.0025	0.4800 ± 0.0153	0.4898 ± 0.0101
Missing rate: 0.2	ML-KNN	TRAM	LLFS-DL	LCFM	CeFL	uCeFL
HammingLoss (↓)	0.0568 ± 0.0006	0.1083 ± 0.0013	0.0628 ± 0.0021	0.2754 ± 0.0046	0.0553 ± 0.0013	0.0553 ± 0.0011
SubsetAccuracy (↑)	0.0213 ± 0.0070	0.0073 ± 0.0025	0.0273 ± 0.0035	0.0000 ± 0.0000	0.0973 ± 0.0129	0.1024 ± 0.0107
Accuracy (↑)	0.1953 ± 0.0315	0.0220 ± 0.0003	0.0673 ± 0.0106	0.1743 ± 0.0065	0.3409 ± 0.0060	0.3410 ± 0.0030
F1measure (↑)	0.2703 ± 0.0369	0.0298 ± 0.0015	0.0843 ± 0.0165	0.2878 ± 0.0090	0.4384 ± 0.0053	0.4394 ± 0.0017
MacroF1measure (↑)	0.0583 ± 0.0086	0.0179 ± 0.0025	0.0203 ± 0.0049	0.1380 ± 0.0040	0.1839 ± 0.0270	0.1724 ± 0.0307
MicroF1Measure (↑)	0.3263 ± 0.0395	0.0347± 0.0021	0.0903 ± 0.0253	0.2878 ± 0.0085	0.4512 ± 0.0050	0.4513 ± 0.0034
Missing rate:0.4	ML-KNN	TRAM	LLFS-DL	LCFM	CeFL	uCeFL
HammingLoss (↓)	0.0623 ± 0.0006	0.0973 ± 0.0003	0.0633 ± 0.0011	0.2743 ± 0.0079	0.0603 ± 0.0015	0.0603 ± 0.0014
SubsetAccuracy (↑)	0.0163 ± 0.0095	0.0024 ± 0.0025	0.0294 ± 0.0035	0.0000 ± 0.0000	0.0729 ± 0.0095	0.0813 ± 0.0177
Accuracy (↑)	0.0633 ± 0.0136	0.0123 ± 0.0013	0.0303 ± 0.0033	0.1754 ± 0.0075	0.2579 ± 0.0185	0.2593 ± 0.0240
F1measure (↑)	0.0864 ± 0.0160	0.0193 ± 0.0005	0.0313 ± 0.0030	0.2888 ± 0.0105	0.3383 ± 0.0233	0.3384 ± 0.0267
MacroF1measure (↑)	0.0174 ± 0.0003	0.0030 ± 0.0000	0.0039 ± 0.0000	0.1360 ± 0.0046	0.1628 ± 0.0000	0.1528 ± 0.0095
MicroF1Measure (↑)	0.1070 ± 0.0103	0.0210 ± 0.0030	0.0193 ± 0.0021	0.2893 ± 0.0100	0.3473 ± 0.0123	0.3498 ± 0.0157
Missing rate: 0.6	ML-KNN	TRAM	LLFS-DL	LCFM	CeFL	uCeFL
HammingLoss (↓)	0.0628 ± 0.0011	0.0848 ± 0.0005	0.0633 ± 0.0006	0.2773 ± 0.0069	0.0604 ± 0.0007	0.0603 ± 0.0011
SubsetAccuracy (↑)	0.0033 ± 0.0035	0.0033 ± 0.0013	0.0200 ± 0.0083	0.0000 ± 0.0000	0.0648 ± 0.0059	0.0673 ± 0.0059
Accuracy (↑)	0.0189 ± 0.0005	0.0099 ± 0.0011	0.0223 ± 0.0090	0.1728 ± 0.0075	0.2213 ± 0.0005	0.2202 ± 0.0048
F1measure (↑)	0.0273 ± 0.0015	0.0133 ± 0.0006	0.0233 ± 0.0093	0.2853 ± 0.0105	0.2918 ± 0.0021	0.2924 ± 0.0039
MacroF1measure (↑)	0.0084 ± 0.0026	0.0030 ± 0.0011	0.0033 ± 0.0006	0.1309 ± 0.0079	0.0998 ± 0.0005	0.1411 ± 0.0018
MicroF1Measure (↑)	0.0348 ± 0.0065	0.0129 ± 0.0005	0.0153 ± 0.0055	0.2860 ± 0.0100	0.2919 ± 0.0045	0.2923 ± 0.0015

Table 6. Comparison results (mean ± std) of six approaches over the education dataset.

Education
Missing rate: 0	ML-KNN	TRAM	LLFS-DL	LCFM	CeFL	uCeFL
HammingLoss (↓)	0.0378 ± 0.0005	0.0463 ± 0.0005	0.0398 ± 0.0000	0.2813 ± 0.0019	0.0563 ± 0.0043	0.0563 ± 0.0049
SubsetAccuracy (↑)	0.1968 ± 0.0233	0.2944 ± 0.0032	0.1420 ± 0.0045	0.0000 ± 0.0000	0.2053 ± 0.0385	0.2073 ± 0.0497
Accuracy (↑)	0.2318 ± 0.0250	0.4134 ± 0.0018	0.1704 ± 0.0035	0.1323 ± 0.0003	0.2878 ± 0.0439	0.2873 ± 0.0577
F1measure (↑)	0.2443 ± 0.0255	0.4573 ± 0.0031	0.1793 ± 0.0030	0.2273 ± 0.0003	0.3193 ± 0.0456	0.3174 ± 0.0601
MacroF1measure (↑)	0.1133 ± 0.0129	0.2173 ± 0.0398	0.0753 ± 0.0005	0.1129 ± 0.0065	0.1433 ± 0.0246	0.1333 ± 0.0241
MicroF1Measure (↑)	0.3364 ± 0.0273	0.4498 ± 0.0069	0.2804 ± 0.0011	0.2304 ± 0.0003	0.3198 ± 0.0453	0.3200 ± 0.0544
Missing rate: 0.2	ML-KNN	TRAM	LLFS-DL	LCFM	CeFL	uCeFL
HammingLoss (↓)	0.0399 ± 0.0021	0.0683 ± 0.0011	0.0413 ± 0.0011	0.2779 ± 0.0006	0.0508 ± 0.0006	0.0500 ± 0.0002
SubsetAccuracy (↑)	0.1673 ± 0.0066	0.0780 ± 0.0035	0.0973 ± 0.0095	0.0000 ± 0.0000	0.2353 ± 0.0165	0.2473 ± 0.0024
Accuracy (↑)	0.2069 ± 0.0025	0.0999 ± 0.0016	0.1163 ± 0.0060	0.1328 ± 0.0015	0.3104 ± 0.0119	0.3219 ± 0.0014
F1measure (↑)	0.2213 ± 0.0013	0.1083 ± 0.0006	0.1233 ± 0.0050	0.2283 ± 0.0025	0.3390 ± 0.0096	0.3493 ± 0.0030
MacroF1measure (↑)	0.0993 ± 0.0013	0.0289 ± 0.0155	0.0350 ± 0.0021	0.1083 ± 0.0056	0.1483 ± 0.0048	0.1404 ± 0.0021
MicroF1Measure (↑)	0.2903 ± 0.0056	0.1048 ± 0.0003	0.1713 ± 0.0025	0.2313 ± 0.0025	0.3353 ± 0.0066	0.3443 ± 0.0045
Missing rate: 0.4	ML-KNN	TRAM	LLFS-DL	LCFM	CeFL	uCeFL
HammingLoss (↓)	0.0403 ± 0.0005	0.0668 ± 0.0005	0.0424 ± 0.0000	0.2613 ± 0.0025	0.0484 ± 0.0006	0.0483 ± 0.0007
SubsetAccuracy (↑)	0.1493 ± 0.0275	0.0753 ± 0.0066	0.0744 ± 0.0033	0.0000 ± 0.0000	0.2493 ± 0.0120	0.2484 ± 0.0077
Accuracy (↑)	0.1809 ± 0.0305	0.0973 ± 0.0090	0.0840 ± 0.0059	0.1370 ± 0.0016	0.3133 ± 0.0072	0.3123 ± 0.0057
F1measure (↑)	0.1928 ± 0.0316	0.1054 ± 0.0096	0.0873 ± 0.0070	0.2348 ± 0.0025	0.3363 ± 0.0056	0.3373 ± 0.0050
MacroF1measure (↑)	0.0883 ± 0.0030	0.0258 ± 0.0160	0.0219 ± 0.0016	0.1090 ± 0.0065	0.1373 ± 0.0023	0.1433 ± 0.0087
MicroF1Measure (↑)	0.2494 ± 0.0319	0.1028 ± 0.0086	0.1213 ± 0.0121	0.2368 ± 0.0023	0.3338 ± 0.0033	0.3343 ± 0.0041
Missing rate: 0.6	ML-KNN	TRAM	LLFS-DL	LCFM	CeFL	uCeFL
HammingLoss (↓)	0.0423 ± 0.0002	0.0608 ± 0.0005	0.0433 ± 0.0005	0.2663 ± 0.0036	0.0514 ± 0.0021	0.0513 ± 0.0003
SubsetAccuracy (↑)	0.0744 ± 0.0086	0.0553 ± 0.0013	0.0524 ± 0.0060	0.0000 ± 0.0000	0.1868 ± 0.0020	0.1933 ± 0.0037
Accuracy (↑)	0.0933 ± 0.0076	0.0708 ± 0.0011	0.0558 ± 0.0066	0.1333 ± 0.0013	0.2528 ± 0.0023	0.2553 ± 0.0001
F1measure (↑)	0.1008 ± 0.0073	0.0763 ± 0.0005	0.0573 ± 0.0073	0.2294 ± 0.0016	0.2783 ± 0.0040	0.2794 ± 0.0014
MacroF1measure (↑)	0.0388 ± 0.0083	0.0063 ± 0.0000	0.0159 ± 0.0019	0.0989 ± 0.0015	0.1019 ± 0.0106	0.1094 ± 0.0050
MicroF1Measure (↑)	0.1453 ± 0.0070	0.0834 ± 0.0000	0.0780 ± 0.0106	0.2309 ± 0.0023	0.2938 ± 0.0125	0.2929 ± 0.0057

Table 7. Comparison results (mean ± std) of six approaches over the genbase dataset.

Genbase
Missing rate: 0	ML-KNN	TRAM	LLFS-DL	LCFM	CeFL	uCeFL
HammingLoss (↓)	0.0043 ± 0.0021	0.0008 ± 0.0006	0.0110 ± 0.0023	0.1723 ± 0.0100	0.0003 ± 0.0007	0.0008 ± 0.0008
SubsetAccuracy (↑)	0.9158 ± 0.0300	0.9833 ± 0.0150	0.8103 ± 0.0453	0.0453 ± 0.0783	0.9847± 0.0000	0.9847± 0.0151
Accuracy (↑)	0.9478 ± 0.0195	0.9820 ± 0.0080	0.8839 ± 0.0355	0.2613 ± 0.0593	0.9913 ± 0.0065	0.9920 ± 0.0080
F1measure (↑)	0.9569 ± 0.0170	0.9938 ± 0.0063	0.9093 ± 0.0336	0.3838 ± 0.0483	0.9943 ± 0.0043	0.9950 ± 0.0062
MacroF1measure (↑)	0.5610 ± 0.0065	0.7720 ± 0.0056	0.4388 ± 0.0015	0.4593 ± 0.0220	0.7723 ± 0.0057	0.7673 ± 0.0107
MicroF1Measure (↑)	0.9498 ± 0.0115	0.9929 ± 0.0070	0.8809 ± 0.0216	0.3483 ± 0.0221	0.9929 ± 0.0059	0.9938 ± 0.0084
Missing rate: 0.2	ML-KNN	TRAM	LLFS-DL	LCFM	CeFL	uCeFL
HammingLoss (↓)	0.0133 ± 0.0025	0.0408 ± 0.0086	0.0163 ± 0.0013	0.1483 ± 0.0075	0.0103 ± 0.0025	0.0103 ± 0.0021
SubsetAccuracy (↑)	0.8013 ± 0.0300	0.5663 ± 0.1205	0.7530 ± 0.0060	0.0000 ± 0.0000	0.8464 ± 0.0150	0.8494 ± 0.0120
Accuracy (↑)	0.8463 ± 0.0343	0.5733 ± 0.1215	0.7789 ± 0.0026	0.2658 ± 0.0139	0.9003 ± 0.0205	0.9029 ± 0.0178
F1measure (↑)	0.8653 ± 0.0359	0.5758 ± 0.1220	0.7908 ± 0.0011	0.4043 ± 0.0133	0.9193 ± 0.0223	0.9218 ± 0.0201
MacroF1measure (↑)	0.3273 ± 0.0143	0.1838 ± 0.0245	0.2593 ± 0.0003	0.4578 ± 0.0565	0.4463 ± 0.0316	0.4660 ± 0.0167
MicroF1Measure (↑)	0.8363 ± 0.0323	0.5128 ± 0.1063	0.7873 ± 0.0146	0.3803 ± 0.0145	0.8754 ± 0.0286	0.8793 ± 0.0247
Missing rate: 0.4	ML-KNN	TRAM	LLFS-DL	LCFM	CeFL	uCeFL
HammingLoss (↓)	0.0443 ± 0.0003	0.0708 ± 0.0036	0.0173 ± 0.0039	0.1179 ± 0.0135	0.0148 ± 0.0013	0.0138 ± 0.0021
SubsetAccuracy (↑)	0.0453 ± 0.0453	0.1054 ± 0.0270	0.7319 ± 0.0573	0.0468 ± 0.0811	0.7713 ± 0.0602	0.7683 ± 0.0572
Accuracy (↑)	0.0473 ± 0.0473	0.1099 ± 0.0255	0.7470 ± 0.0563	0.3373 ± 0.0495	0.8060 ± 0.0469	0.8053 ± 0.0444
F1measure (↑)	0.0483 ± 0.0483	0.1114 ± 0.0250	0.7523 ± 0.0556	0.4773 ± 0.0416	0.8193 ± 0.0420	0.8189 ± 0.0400
MacroF1measure (↑)	0.0183 ± 0.0185	0.0163 ± 0.0095	0.2393 ± 0.0003	0.4003 ± 0.0306	0.3623 ± 0.0226	0.3688 ± 0.0187
MicroF1Measure (↑)	0.0739 ± 0.0739	0.1073 ± 0.0250	0.7663 ± 0.0530	0.4298 ± 0.0366	0.8143 ± 0.0179	0.8253 ± 0.0254
Missing rate: 0.6	ML-KNN	TRAM	LLFS-DL	LCFM	CeFL	uCeFL
HammingLoss (↓)	0.0453 ± 0.0005	0.0600 ± 0.0016	0.0389 ± 0.0065	0.0503 ± 0.0056	0.0123 ± 0.0023	0.0120 ± 0.0020
SubsetAccuracy (↑)	0.0030 ± 0.0030	0.0783 ± 0.0240	0.1658 ± 0.1656	0.5303 ± 0.0195	0.7923 ± 0.0513	0.7953 ± 0.0482
Accuracy (↑)	0.0043 ± 0.0045	0.0813 ± 0.0241	0.1658 ± 0.1656	0.6973 ± 0.0120	0.8313 ± 0.0456	0.8333 ± 0.0440
F1measure (↑)	0.0050 ± 0.0050	0.0823 ± 0.0241	0.1658 ± 0.1656	0.7673 ± 0.0133	0.8464 ± 0.0433	0.8488 ± 0.0415
MacroF1measure (↑)	0.0043 ± 0.0040	0.0089 ± 0.0015	0.0363 ± 0.0365	0.4113 ± 0.0421	0.4270 ± 0.0235	0.4338 ± 0.0174
MicroF1Measure (↑)	0.0098 ± 0.0096	0.0933 ± 0.0219	0.2148 ± 0.2146	0.6378 ± 0.0296	0.8413 ± 0.0321	0.8483 ± 0.0267

Table 8. Comparison results (mean ± std) of six approaches over the medical dataset.

Medical
Missing rate: 0	ML-KNN	TRAM	LLFS-DL	LCFM	CeFL	uCeFL
HammingLoss (↓)	0.0163 ± 0.0005	0.0128 ± 0.0021	0.0213 ± 0.0000	0.2533 ± 0.0040	0.0113 ± 0.0021	0.0113 ± 0.0017
SubsetAccuracy (↑)	0.4633 ± 0.0346	0.6613 ± 0.0285	0.2673 ± 0.0143	0.0000 ± 0.0000	0.6773 ± 0.0366	0.6858 ± 0.0490
Accuracy (↑)	0.5303 ± 0.0225	0.7378 ± 0.0195	0.3143 ± 0.0075	0.1013 ± 0.0030	0.7623 ± 0.0230	0.7668 ± 0.0347
F1measure (↑)	0.5529 ± 0.0185	0.7638 ± 0.0163	0.3293 ± 0.0050	0.1803 ± 0.0049	0.7914 ± 0.0186	0.7943 ± 0.0300
MacroF1measure (↑)	0.2033 ± 0.0163	0.2783 ± 0.0069	0.0678 ± 0.0089	0.1043 ± 0.0063	0.3263 ± 0.0013	0.3293 ± 0.0254
MicroF1Measure (↑)	0.6343 ± 0.0120	0.7538 ± 0.0173	0.4773 ± 0.0070	0.1758 ± 0.0046	0.7769 ± 0.0203	0.7803 ± 0.0315
Missing rate: 0.2	ML-KNN	TRAM	LLFS-DL	LCFM	CeFL	uCeFL
HammingLoss (↓)	0.0169 ± 0.0003	0.0450 ± 0.0006	0.0218 ± 0.0001	0.2503 ± 0.0025	0.0140 ± 0.0013	0.0138 ± 0.0011
SubsetAccuracy (↑)	0.4184 ± 0.0265	0.0838 ± 0.0103	0.2388 ± 0.0060	0.0000 ± 0.0000	0.6103 ± 0.0103	0.6184 ± 0.0020
Accuracy (↑)	0.4910 ± 0.0121	0.0959 ± 0.0123	0.2713 ± 0.0045	0.1013 ± 0.0016	0.6940 ± 0.0183	0.7023 ± 0.0080
F1measure (↑)	0.5153 ± 0.0055	0.1000 ± 0.0129	0.2828 ± 0.0036	0.1803 ± 0.0030	0.7229 ± 0.0206	0.7308 ± 0.0101
MacroF1measure (↑)	0.1793 ± 0.0100	0.0043 ± 0.0005	0.0514 ± 0.0063	0.0947± 0.0063	0.3343 ± 0.0032	0.3253 ± 0.0152
MicroF1Measure (↑)	0.6143 ± 0.0020	0.0963 ± 0.0126	0.4153 ± 0.0029	0.1763 ± 0.0033	0.7170 ± 0.0225	0.7243 ± 0.0151
Missing rate: 0.4	ML-KNN	TRAM	LLFS-DL	LCFM	CeFL	uCeFL
HammingLoss (↓)	0.0210 ± 0.0003	0.0443 ± 0.0021	0.0223 ± 0.0021	0.2300 ± 0.0035	0.0160 ± 0.0003	0.0159 ± 0.0005
SubsetAccuracy (↑)	0.2878 ± 0.0060	0.0714 ± 0.0103	0.2043 ± 0.0083	0.0000 ± 0.0000	0.5043 ± 0.0020	0.5063 ± 0.0041
Accuracy (↑)	0.3373 ± 0.0146	0.0793 ± 0.0065	0.2429 ± 0.0123	0.1099 ± 0.0030	0.5913 ± 0.0083	0.5933 ± 0.0078
F1measure (↑)	0.3543 ± 0.0183	0.0820 ± 0.0050	0.2558 ± 0.0135	0.1940 ± 0.0046	0.6213 ± 0.0111	0.6234 ± 0.0098
MacroF1measure (↑)	0.1299 ± 0.0203	0.0038 ± 0.0000	0.0323 ± 0.0021	0.0934 ± 0.0043	0.2573 ± 0.0135	0.2583 ± 0.0092
MicroF1Measure (↑)	0.4608 ± 0.0086	0.0810 ± 0.0033	0.3610 ± 0.0275	0.1873 ± 0.0043	0.6553 ± 0.0023	0.6573 ± 0.0015
Missing rate: 0.6	ML-KNN	TRAM	LLFS-DL	LCFM	CeFL	uCeFL
HammingLoss (↓)	0.0268 ± 0.0021	0.0350 ± 0.0003	0.0273 ± 0.0013	0.2200 ± 0.0025	0.0193 ± 0.0006	0.0193 ± 0.0002
SubsetAccuracy (↑)	0.0573 ± 0.0040	0.0510 ± 0.0060	0.0020 ± 0.0020	0.0000 ± 0.0000	0.4510 ± 0.0060	0.4533 ± 0.0287
Accuracy (↑)	0.0663 ± 0.0030	0.0553 ± 0.0060	0.0020 ± 0.018	0.1133 ± 0.0033	0.5463 ± 0.0235	0.5463 ± 0.0361
F1measure (↑)	0.0694 ± 0.0055	0.0563 ± 0.0060	0.0020 ± 0.0021	0.1999 ± 0.0045	0.5809 ± 0.0290	0.5804 ± 0.0389
MacroF1measure (↑)	0.0494 ± 0.0113	0.0048 ± 0.0011	0.0003 ± 0.0003	0.0893 ± 0.0023	0.2608 ± 0.0243	0.2853 ± 0.0222
MicroF1Measure (↑)	0.1103 ± 0.0096	0.0699 ± 0.0073	0.0034 ± 0.0035	0.1929 ± 0.0043	0.6088 ± 0.0235	0.6108 ± 0.0225

The experimental results indicate that the following conclusions can be drawn:

• Overall, uCeFL was obviously competitive with the comparison approaches, especially on the emotions, genbase, medical, and corel5k datasets, where uCeFL performed best, regardless of the loss rate.
• When the label is not missing, TRAM performs relatively well on the three scene, enron, and education databases, and its performance on the other four datasets is also quite competitive. However, it is crucial to note that in the step of deriving the transformation matrix P, there is a significant reliance on the sample ground-truth labels. If labels are missing in the dataset, the accuracy of the obtained transformation matrix P may be compromised, potentially leading to inaccuracies in the labeling of unlabeled samples. Consequently, as the rate of missing labels increases, the overall performance of the system declines accordingly.
• ML-KNN only performed well on the scene dataset, and its performance was optimal when the missing rate was 0.2. ML-KNN utilizes the label information of the k-nearest neighbors of the test data to estimate the label set, but it does not take into account label correlation. Therefore, poor label annotation performance is possible when some class labels of the training data are missing.
• Similar to our approaches, the label space $Y$ is augmented in LLSF-DL with the feature space $X$ as additional features. LLSF-DL removes unnecessary dependency relationships by identifying the sparsity coefficient of the label space $Y$. However, it may disregard correct indirect relationships resulting from labels that are not subject to sparsity coefficient consideration, which could be a reason why its performance is not ideal.
• Compared to LCFM, which tackles missing labels by integrating both global and local label-specific features, our method leverages neural networks with hidden layers. This facilitates hierarchical learning of label-expanded data features and adeptly extracts high-level semantic features from the data. These features capture three distinct relationships: between data features, between labels and data features, and among labels. Experimental results confirm that these features offer significant benefits for the development of subsequent classification models.
• At the end of each round of training, uCeFL performs a feedforward process to extract high-level features from the input data. Label correlations are then recalculated to update the input for the next round of training. Therefore, uCeFL not only takes the label correlations as prior knowledge but also continuously improves them during the learning process of the classification model. The experimental results show that uCeFL works better than CeFL, which demonstrates the effectiveness of using neural networks with hidden layers for feature learning.

To demonstrate the efficacy of extracting label correlations directly from the data, a random division of the data into testing and training sets was made, with 1/3 designated for testing and the remaining portion for training. Subsequently, two techniques were employed to compute label correlations: the cosine similarity (CeFL_CS) and the covariance matrix similarity coefficient (uCeFL). Figure 2, Figure 3, Figure 4 and Figure 5 offer a comparative analysis of these techniques across diverse datasets. Notably, the results underscore the potential of extracting label correlations from the data, particularly in scenarios where a substantial proportion of class labels for the training data are missing, resulting in impressive performance.

Figure 2. Performance comparison between two similarity methods on the emotions and scene datasets.

Figure 3. Performance comparison between two similarity methods on the corel5k and enron datasets.

Figure 4. Performance comparison between two similarity methods on the education and genbase datasets.

Figure 5. Performance comparison between two similarity methods on the medical dataset.

5. Conclusions and Future Works

In summary, the uCeFL methodology proposed in this paper has incontrovertibly established its superiority in tackling multi-label classification tasks, especially when dealing with missing labels. By directly extracting label correlations from the data instances, uCeFL avoids reliance on potentially incomplete label sets. The method first computes a refined label matrix by multiplying the incomplete label matrix with the label correlations obtained from the data matrix. This refined matrix then augments the original data features, enabling a neural network to learn correlation-enhanced representations that capture the intricate interplay between data features, labels, and their interactions. Importantly, uCeFL does not treat label correlations as static, predefined knowledge. Instead, it iteratively updates the label correlations during the training process, further optimizing its performance. This is achieved through a feedforward mechanism implemented at the end of each training iteration, where high-level features of the input data are extracted, and the label correlations are recalculated to refine the input for the next iteration. This iterative refinement allows uCeFL to adapt to the changing nature of the data and label correlations, ultimately resulting in superior classification outcomes. The experimental results across a range of datasets highlight the competitiveness of uCeFL. Notably, it outperforms other comparative approaches, especially on datasets related to emotions, enron, genbase, and medical, where it achieves optimal performance regardless of the label missing rate.

Future research will consider the following aspects:

• This paper employed a covariance matrix to represent the dataset comprising category labels and derived label correlations through the calculation of a covariance matrix similarity coefficient. However, computing these correlations becomes challenging when dealing with a substantial data volume. The issue of efficiently acquiring label correlations in a big-data environment demands further exploration.
• Moreover, the paper only concentrated on one-to-one label correlations. To accurately capture the actual label correlations, further research is needed, encompassing investigations into both local and global correlations.
• The presence of multi-label data introduces the curse of dimensionality. Future aims can explore the relationship between labels and features to identify a more discriminative subspace.