Establishing Two-Dimensional Dependencies for Multi-Label Image Classification

Abstract

As a fundamental upstream task, multi-label image classification (MLIC) work has made a great deal of progress in recent years. Establishing dependencies between targets is crucial for MLIC as targets in the real world always co-occur simultaneously. However, due to the complex spatial relationships and semantic relationships among targets, existing methods fail to effectively establish the dependencies between targets. In this paper, we propose a Two-Dimensional Dependency Model (TDDM) for MLIC. The network consists of an Spatial Feature Dependency Module (SFDM) and a Label Semantic Dependency Module (LSDM), which establish effective dependencies in the dimensions of image spatial features and label semantics, respectively. Our method was tested on three publicly available multi-label image datasets, PASCAL VOC 2007, PASCAL VOC 2012, and MS-COCO, and it produced superior results compared to existing state-of-the-art methods, as demonstrated in our experiments.

1. Introduction

Multi-label image classification (MLIC) is a fundamental yet crucial task in the domain of computer vision, with profound implications in diverse fields such as target detection [1,2,3], medical image recognition [4,5,6], and human attribute recognition [7,8,9]. Since targets typically coexist in the real world, a key aspect of MLIC lies in establishing dependencies between targets.

In the literature, previous methods have primarily focused on modeling spatial dependencies or label semantic dependencies. (1) For modeling spatial dependencies, convolution-based architectures [10,11] are the most commonly utilized approach. However, due to the constraint of the kernel size, the receptive field of CNN networks is limited, which would lead to a problem, i.e., the establishment of spatial dependency can be significantly affected by the pixel distance between targets. Therefore, it is challenging to establish effective long-range spatial dependencies when the targets in an image are distantly located. (2) For modeling label semantic dependencies, architectures [12,13] based on Graph Convolutional Networks (GCNs) are the most commonly utilized approach. However, the utilization of first-order neighborhood aggregation in the standard GCN formulation leads to a limited expressive power to the GCN and a lack of global information integration. For example, based on human experience, the probability of ‘dog’ and ‘fork’ appearing together is typically low, but if there is a ‘person’ present, the probability of ‘dog’ and ‘fork’ appearing together significantly increases. However, existing methods fail to consider the impact of the presence of ‘person’ on the coexistence probability of ‘dog’ and ‘fork’.

In multi-label image classification, existing methods face limitations in modeling long-range spatial dependencies and global semantic dependencies. Traditional CNNs are constrained by convolutional kernel sizes, making it difficult to capture long-range spatial relationships, while traditional GCNs rely only on first-order neighborhood aggregation, thus lacking the ability to integrate global information. To address these challenges, we propose the TDDM, which incorporates the SFDM and the LSDM to simultaneously model spatial and semantic dependencies. SFDM enhances cross-layer feature interactions through the FFM and FEM, improving feature representation and effectively capturing long-range spatial dependencies. LSDM introduces the GREM, which employs a multi-head max-attention mechanism to overcome the local aggregation limitation of traditional GCNs, strengthening global semantic modeling. Compared to conventional CNNs [14,15,16], GCNs [17,18], and existing multi-label classification methods [12,19], TDDM jointly models spatial and semantic dependencies, achieving excellent results on multiple datasets. These results validate TDDM’s effectiveness in modeling spatial and semantic dependencies, providing a more comprehensive solution for multi-label image classification.

In short, there are two major challenges in the field of MLIC currently. Firstly, it is difficult to establish effective long-range spatial feature dependencies. Secondly, it is challenging to establish global label semantic dependencies. In this paper, inspired by the work of [12,19], we propose a Two-Dimensional Dependency Model (TDDM) to simultaneously address the aforementioned issues. The TDDM consists of an Spatial Feature Dependency Module (SFDM), which establishes the dependency in the image spatial feature dimension, and a Label Semantic Dependency Module (LSDM), which establishes the dependency in the label semantic dimension. In the following, we will elaborate on our proposed method.

A. 

Establishing effective long-range spatial feature dependencies.

The Spatial Feature Dependency Module (SFDM) is used to establish effective long-range spatial feature dependencies. SFDM utilizes the ResNet network and extract feature maps from different layers, which preserves the information of targets at different spatial positions. These feature maps are then fed into our designed Feature Fusion Module (FFM) to fuse, establishing the spatial dependency of target features. For the highly abstract feature maps output by the last layer of ResNet, we designed a Feature Enhancement Module (FEM) to enhance and enrich the high-dimensional information. Through cross-layer fusion and feature enhancement, the spatial dependencies between targets are modeled, long-range dependencies are established, and the limitations of fixed convolutional kernel receptive fields are alleviated. By fusing features across different layers, we alleviated the limitation of local feature extraction and preserved detailed information as much as possible. In doing so, we effectively addressed the issues associated with using multiple layers of CNN for establishing spatial dependencies. In addition, our method adds only a few convolutional and fully connected layers in addition to the backbone network, which does not over-increase the network complexity and does not lead to gradient vanishing or explosion problems. This is also a strength compared to stacking multiple layers of CNNs.

B. 

Establishing global label semantic dependencies.

The Label Semantic Dependency Module (LSDM) uses a GCN to establish global label semantic dependencies. We also designed the Global Relationship Enhancement Module (GREM) based on a multi-head max-attention mechanism, which incorporates a max pooling operation on the $Query$ tensor in addition to the self-attention mechanism, resulting in improved global enhancement effects. The information extracted from GCN is enhanced by GREM. By implementing the aforementioned operations, we can address the issue of insufficient extraction of label relationships and establish global semantic dependencies among labels.

In summary, the main contributions of this article are as follows.

• We propose a Two-Dimensional Dependency Model (TDDM) for MLIC, which can simultaneously establish effective long-range spatial feature dependencies and global label semantic dependencies. This approach addresses the challenges of capturing the feature dependencies between distant targets in image feature extraction and incomplete understanding of the semantic dependencies between labels. To the best of our knowledge, this is the first multi-label image classification network that considers both problems simultaneously.
• We propose a Feature Fusion Module (FFM) and a Feature Enhancement Module (FEM), which effectively integrate image feature information from different spatial positions while enhancing and enriching high-dimensional abstract information. In terms of semantic extraction, we design a Global Relationship Extraction Module (GREM) to enhance the fusion of global relationships.
• We conducted experiments comparing our method with state-of-the-art methods on commonly used benchmark datasets. The results show that our method has superior computational performance. Specifically, our model achieves mAPs of 96.5% on PASCAL VOC 2007, 96.0% on PASCAL VOC 2012, and 85.2% on MS-COCO. The datasets and source code can be accessed from https://github.com/12pid/TDDM, accessed on 2 March 2025.

2. Related Work

The problem of MLIC has attracted increasing attention from researchers and was comprehensively reviewed in [20,21]. The most straightforward approach is to transform the multi-label classification problem into multiple single-label classification problems [22]. However, these methods overlook inter-label correlations, leading to low accuracy, and they also fail to address large-scale target recognition tasks. With the advancement of deep learning, neural networks have been able to learn rich information from datasets such as MS-COCO [23], PASCAL VOC [24], and ImageNet [25]. As a result, scholars have proposed various MLIC methods based on image features [26,27]. In recent years, there has been a growing body of research on exploring the dependencies among labels [16], with scholars simultaneously considering both image features and label semantics. In this paper, we will primarily review the relevant works from the following three perspectives.

2.1. Traditional Multi-Label Classification Methods

There are generally two directions for handling multi-label classification problems. The first direction involves problem transformation, while the second direction focuses on leveraging the characteristics of label relationships to propose adaptive algorithms. Early multi-label classification algorithms were based on the transformation approach, treating the problem as multiple independent single-label problems and ignoring the correlation between labels. Among them, the typical algorithms are One-vs-All [28] and Binary Relevance [29]. The Binary Relevance algorithm constructs multiple datasets for each label, where examples containing the label are labeled as positive and specific label classifiers are trained for prediction. Many methods apply association rules to address the correlation among labels in multi-label classification problems. However, these methods may encounter the issue of rule explosion, where a large number of rules are generated, leading to increased computational complexity, especially when dealing with a large number of labels. The representative algorithms in this category include Apriori [30] and FP-Growth [31]. Through using Apriori, one can obtain frequent item sets from the dataset, and they one can subsequently extract association rules from these frequent item sets based on their definition. Other classical machine learning algorithms, such as decision trees [32], boosting [33], and neural networks [34], have also been used.

2.2. Deep Learning Methods for MLIC

With the advancement of CNNs [26,27,35,36,37], numerous models [14,15,16,38] suitable for MLIC have been proposed. To leverage the relationships among labels, the study of [39] utilized Recurrent Neural Networks (RNNs) to transform labels into embedded label vectors. In [40], Zhu et al. improved the performance of a multi-label image classification task by introducing spatial regularization and image-level supervisory information. The focus was on constructing and classifying different labeled regions. However, because of the use of the spatial regularization method, it involved a large number of parameter tuning problems, such as, in the process of practical application, the image perhaps being affected by lighting, angle, scale, and other factors, which will affect the SRN spatial regularization; in addition, the method’s robustness is not good. The study of [41] incorporated label reordering and handling into the overall network architecture using context gating strategies. Guo et al. [42] considered visual attention consistency in multi-label image classification. Furthermore, label balancing techniques [43] have also been applied to enhance multi-label image classification performance.

2.3. Graph Structure Methods for MLIC

A significant body of work has demonstrated that complex label relationships are well suited for constructing graph structures. Li et al. [17] created a tree-like structure graph in the label space using the maximum spanning tree algorithm, and they also identified sets of informative label combinations to improve overall multi-label prediction performance. Li et al. [44] proposed a graphical lasso framework that models label correlations by jointly considering image features and labels. In state-of-the-art approaches [14,45], GCNs have been introduced for MLIC. The study of [12] utilized GCN to establish a dependency model among labels. The model constructs a directed graph on target labels, where each node is represented by the word embeddings of the label. GCN learns to map this label graph into a set of interdependent target classifiers. Additionally, the model employs a novel reweighting scheme to create an effective label correlation matrix, optimizing the propagation of information between GCN nodes. However, the method does not consider the global nature of image features and label semantics, and it only employs ResNet and standard GCN modules as the backbone network. In this paper, when we established the image spatial feature dependency, we also added FFM for feature fusion and FEM for high-level feature enhancement on top of ResNet. This makes up for the defect of not being able to establish long-range spatial dependencies due to the size limitation of the convolutional kernel. Meanwhile, when modeling inter-label semantic dependencies, we also added our designed GREM after the GCN module to enhance the global inter-label semantics. Zhao et al. [19] proposed a transformer-based dual relationship learning framework that constructs complementary relationships by exploring the correlation between a structural relationship graph and a semantic relationship graph. The structural relationship graph aims to capture long-term correlations by leveraging a scale-crossing transformer architecture within the target context. The semantic graph dynamically models the semantic significance of image objects using explicit semantic-aware constraints. However, on the one hand, this method introduces multiple Transformer modules and complex structures, leading to a model that is too large to be used in some scenarios with limited resources and, at the same time, also leading to more difficult model training. On the other hand, the approach uses only standard GCN modules and does not take into account the global nature of inter-label semantics. In this paper, we used FFM and FEM to complete the feature fusion and enhancement, which does not add too much computational burden and can achieve superior results. At the same time, GREM was designed to consider and solve the label semantic global problem.

Motivated by the aforementioned works, we propose Two-Dimensional Dependency Model (TDDM). Our model leverages a Spatial Feature Dependency Module (SFDM) to extract image information and establish dependencies in the image feature space while utilizing a Label Semantic Dependency Module (LSDM) to establish global label semantic dependencies. The key aspect of TDDM lies in simultaneously considering both the image spatial feature dependencies and global label semantic dependencies. We integrated the outputs of SFDM and LSDM to perform MLIC.

3. Method

In this section, we provide a detailed exposition of the Two-Dimensional Dependency Model (TDDM) framework for MLIC. We begin by presenting the preliminaries behind our approach, followed by a comprehensive description of the components of TDDM, i.e., SFDM and LSDM. Lastly, we provide an overview of the loss function used.

3.1. Preliminaries

We propose the TDDM framework, as shown in Figure 1. TDDM consists of two branches, SFDM and LSDM. SFDM is used to model the dependency of image features, where we employ ResNet as the backbone. During the downsampling process of ResNet, feature maps of different sizes are extracted from different layers and are then fused together. These feature maps contain information about targets at different spatial locations, thereby capturing the spatial dependency of targets in the fused feature map. Additionally, we enhanced the features from the last layer of ResNet to emphasize high-dimensional abstract information. LSDM was utilized to model the global label semantic dependency. We employed GCN to model the semantic dependency of the labels, and we utilized the Global Relationship Enhancement Module (GREM) to enhance the global semantic coherence among labels. Specifically, we devised a maximum attention mechanism to realize the functionality of GREM.

3.2. SFDM

In the SFDM, ResNet is used as the backbone. Given the input image $\mathit{I}\in {\mathbb{R}}^{448\times 448}$, ResNet generates feature maps $X_3\in {\mathbb{R}}^{1024\times 28\times 28}$ and $X_4\in {\mathbb{R}}^{2048\times 14\times 14}$ from ‘Layer3’ and ‘Layer4’, respectively. The feature maps are processed as follows:

$$\begin{array}{c}\hfill u1=\mathrm{Conv}\left(X_3\right),u2={\mathrm{Conv}}_1\left(X_4\right),u3={\mathrm{Conv}}_2\left(X_4\right),\end{array}$$

(1)

where ${\mathrm{Conv}}_1(·)$ and ${\mathrm{Conv}}_2(·)$ are two convolutional operations applied to $X_4$. The resulting tensors $u1$, $u2$, and $u3$ are passed into the Feature Fusion Module (FFM), while the output $X_4$ from the last layer of ResNet is input into the Feature Enhancement Module (FEM). The FFM integrates shallow and deep features along with spatial positional information, producing the following:

$$\begin{array}{c}\hfill {\mathit{u}}_{FFM}=f_{FFM}(u1,u2,u3).\end{array}$$

(2)

The FEM focuses on enhancing the abstract feature $X_4$ to improve its representation as follows:

$$\begin{array}{c}\hfill {\mathit{u}}_{FEM}=f_{FEM}\left(X_4\right).\end{array}$$

(3)

Finally, the SFDM output is obtained by fusing the outputs of both FFM and FEM:

$$\begin{array}{c}\hfill {\mathit{X}}_{SFDM}=f_{FFM}(u1,u2,u3)+f_{FEM}\left(X_4\right).\end{array}$$

(4)

This results in ${\mathit{X}}_{SFDM}$, which captures both individual target features and their spatial dependencies, effectively enhancing the model’s expressive power.

3.2.1. Feature Fusion Module

The Feature Fusion Module (FFM) performs cross-layer and cross-scale fusion, as shown in Figure 2. To enhance the model’s representation power, feature tensors $X_3\in {\mathbb{R}}^{1024\times 28\times 28}$ and $X_4\in {\mathbb{R}}^{2048\times 14\times 14}$ from ‘Layer3’ and ‘Layer4’ of ResNet are extracted and processed. Tensor $X_3$ passes through a convolutional layer to produce $u_1\in {\mathbb{R}}^{512\times 28\times 28}$, and $X_4$ undergoes two convolutional layers to generate $u_2\in {\mathbb{R}}^{512\times 14\times 14}$ and $u_3\in {\mathbb{R}}^{512\times 7\times 7}$. These tensors are unified using bilinear operations, resulting in $u_1^{\prime }$, $u_2^{\prime }$, and $u_3^{\prime }$. The feature fusion factor $B_f$ is then computed as the sum of these unified tensors:

$$\begin{array}{c}\hfill B_f=u_1^{\prime }+u_2^{\prime }+u_3^{\prime }.\end{array}$$

(5)

Subsequently, the fusion factor $B_f$ is added to $u_1^{\prime }$, $u_2^{\prime }$, and $u_3^{\prime }$, respectively, to achieve information fusion of the information among them:

$$\begin{array}{c}\hfill u_1^{″}=u_1^{\prime }+B_f,u_2^{″}=u_2^{\prime }+B_f,u_3^{″}=u_3^{\prime }+B_f.\end{array}$$

(6)

For the abstract tensor $u_3^{\prime }$, an information compensation step is applied to improve feature fusion. This involves element-wise multiplication of $u_1^{\prime }$, $u_2^{\prime }$, and $u_3^{\prime }$, followed by multiplication with a compensation coefficient $C_c$, resulting in the compensation factor $C_f$:

$$\begin{array}{c}\hfill C_f=C_c·(u_1^{\prime }\odot u_2^{\prime }\odot u_3^{\prime }),\end{array}$$

(7)

where ⊙ denotes element-wise multiplication. The tensor $C_f$ is then multiplied with $u_3^{″}$ to accomplish information compensation:

$$\begin{array}{c}\hfill u_3^{‴}=C_f\odot u_3^{″}.\end{array}$$

(8)

At this stage, cross-layer information fusion was completed. By interpolation, the tensors that had been fused with each other’s information were restored to their original dimensions, and they are denoted as $U_1$, $U_2$, and $U_3$, respectively:

$$\begin{array}{c}\hfill U_1=\mathrm{Interp}\left(u_1^{″}\right),U_2=\mathrm{Interp}\left(u_2^{″}\right),U_3=\mathrm{Interp}\left(u_3^{‴}\right),\end{array}$$

(9)

where Interp(·) represents the interpolation operation. Subsequently, a global maximum pooling operation is performed on each tensor to transform their dimensions to $[B,512]$, where B represents the batch size:

$$\begin{array}{c}\hfill U_1^{pool}=\mathrm{MaxPool}\left(U_1\right),U_2^{pool}=\mathrm{MaxPool}\left(U_2\right),U_3^{pool}=\mathrm{MaxPool}\left(U_3\right).\end{array}$$

(10)

The three feature tensors are then concatenated, resulting in a feature tensor of dimension $[B,3\times 512]$:

$$\begin{array}{c}\hfill U_f=\mathrm{Concat}(U_1^{pool},U_2^{pool},U_3^{pool}).\end{array}$$

(11)

Finally, the feature tensor undergoes a fully connected layer to obtain a feature tensor of dimension $[B,C]$, where C represents the desired number of output classes:

$$\begin{array}{c}\hfill f_{FFM}=f_c\left(U_f\right),\end{array}$$

(12)

where $f_c(·)$ represents the fully connected layer. The computation of the FFM module can be represented by the following equation:

$$\begin{array}{c}\hfill f_{FFM}={\xi }^{GCL}\left(\mathrm{\Psi }(X_3,X_4)\right),\end{array}$$

(13)

where $\mathrm{\Psi }(·,·)$ represents the utilization of the $X_3$ and $X_4$ convolutions, interpolation, and feature fusion; and the process ${\xi }^{GCL}(·)$ involves applying global max pooling, concatenating the fused tensors, and adjusting the length through a final fully connected layer.

3.2.2. Feature Enhancement Module

The output information from ‘$Layer4$’ in ResNet is highly abstract. To enhance these feature representations, we designed the Feature Enhancement Module (FEM). The output of ‘$Layer4$’, denoted as ${\mathit{X}}_{\mathbf{4}}\in {\mathbb{R}}^{2048\times 14\times 14}$, can be represented as ${\mathit{x}}^{\mathbf{1}},{\mathit{x}}^{\mathbf{2}},\dots ,{\mathit{x}}^{\mathbf{196}}({\mathit{x}}_{\mathit{i}}\in {\mathbb{R}}^{2048})$. Firstly, we defined the scoring formula, which represents the attention weight coefficient, for category i and position j as follows:

$$v_j^i={\displaystyle \frac{softmax\left(T{C}^i\left({\mathit{x}}_j\right)\right)}{{\sum }_{\begin{array}{c}k=1\end{array}}^{196}softmax\left(T{C}^i\left({\mathit{x}}_k\right)\right)}},$$

(14)

where $C^i$ is the classifier of class i, and T is the control proportion coefficient.

For general features, the global average attention is defined as follows:

$$\mathcal{G}={\displaystyle \frac{1}{196}}\sum _{n=1}^{196}{\mathit{x}}_n.$$

(15)

To enhance abstract features, we defined the computation of feature attention as follows (which can be regarded as a class-specific attention mechanism that focuses attention on the classification scores at different positions for different categories):

$${\mathcal{A}}^i=\sum _{n=1}^{196}{v}_n^i{\mathit{x}}_n.$$

(16)

To better integrate features, we combined feature attention with global average attention, referred to as fusion attention, and defined it as follows:

$${\mathcal{F}}^i=\mathcal{G}·{\mathcal{A}}^i.$$

(17)

At this point, we obtain the global enhanced attention for the i-th class, which is defined as follows:

$$f^i=\mathcal{G}+\alpha {\mathcal{A}}^i+\beta {\mathcal{F}}^i,$$

(18)

where $\alpha$ and $\beta$ represent weight proportion control coefficients.

The final output to the FEM is as follows:

$$f_{FEM}=(f^1,f^2,\dots ,f^C),$$

(19)

where C represents the number of required output classes.

3.3. LSDM

3.3.1. Overview of GCN

GCN is a rapidly developing research direction in recent years. Unlike CNN, which can only handle problems on Euclidean data, GCN is applicable to a wider range of general data structures. By considering nodes and the directed relationships between them, GCN can effectively capture complex data structures. Messages can be propagated between nodes, and the node representations are updated after message passing. The objective of GCN is to learn a function $f(·,·)$ on graph structures, which can update the representation of each node. The function $f(·,·)$ takes, as input, the feature descriptions ${\mathit{F}}^l\in {\mathbb{R}}^{n\times d}$ and the adjacency matrix $\mathit{A}\in {\mathbb{R}}^{n\times n}$, where n represents the number of nodes and d denotes the dimension of node features. Each layer of nodes can be represented as follows:

$${\mathit{F}}^{l+1}=f({\mathit{F}}^l,\mathit{A}).$$

(20)

After applying the convolution operation, it can be represented as follows:

$${\mathit{F}}^{l+1}=h\left(\widehat{\mathit{A}}{\mathit{F}}^l{\mathit{W}}^l\right),$$

(21)

where $\widehat{\mathit{A}}\in {\mathbb{R}}^{n\times n}$ is the normalized version of the adjacency matrix $\mathit{A}$, ${\mathit{F}}^l$ represents the node information state at layer l, ${\mathit{W}}^{\mathit{l}}\in {\mathbb{R}}^{d\times d^{\prime }}$ represents the learnable transformation parameters, and $h(·)$ denotes the activation function. By stacking multiple GCN layers using the above equation, we can learn the complex relationships between nodes.

3.3.2. Construction of a Correlation Matrix

The adjacency matrix describes the correlations between nodes and guides the propagation of information between nodes in GCN. Typically, the adjacency matrix is predefined. However, in this study, we adopted a data-driven approach, following the method proposed by [12], to construct the adjacency matrix. Specifically, we constructed the adjacency matrix in the form of conditional probabilities. Firstly, we calculated the occurrences of label pairs in the training set to obtain the matrix $\mathit{D}\in {\mathbb{R}}^{C\times C}$, where C represents the number of label categories and $D_{ij}$ denotes the number of occurrences of label i and label j appearing together. Then, based on the matrix D, we can derive the conditional probability matrix:

$${\mathit{P}}_{ij}={\mathit{D}}_{ij}/{\mathit{N}}_j,$$

(22)

where $N_j$ represents the total occurrences of label j in the training set, and $P_{ij}=P\left(L_i\right|L_j)$ represents the probability of label i occurring when label j is present.

The above method for obtaining the adjacency matrix may encounter two issues. Firstly, the data may exhibit a long-tail distribution. Secondly, the adjacency matrix computed from the training set may suffer from overfitting, affecting its generalization ability. To address these issues, we propose to binarize the correlation P and filter out noise using a threshold value $\tau$. Therefore, the adjacency matrix can be represented as follows:

$${\mathit{A}}_{ij}=\left\{\begin{array}{cc}0,\hfill & \mathrm{if}{\mathit{P}}_{ij}<\tau \hfill \\ 1,\hfill & \mathrm{if}{\mathit{P}}_{ij}\ge \tau \hfill \end{array}\right..$$

(23)

However, binarization can introduce the issue of over-smoothing. To mitigate this, we can employ a reweighting scheme, which is represented as follows:

$${\mathit{A}}_{ij}^{\prime }=\left\{\begin{array}{cc}p/\sum _{\begin{array}{c}j=1\\ i\ne j\end{array}}^C{\mathit{A}}_{ij},\hfill & \mathrm{if}i\ne j\hfill \\ 1-p,\hfill & \mathrm{if}i=j\hfill \end{array}\right.,$$

(24)

where $A^{\prime }$ represents the reweighted adjacency matrix, and p denotes the allocation weight to be set. By incorporating the weighting operation, the reweighted adjacency matrix retains both the information of the individual nodes and the information from the neighboring nodes, effectively addressing the issue of over-smoothing.

3.3.3. Global Relationship Enhancement Module

The information representing the graph structure can be propagated among all of the nodes in GCN. The utilization of the standard GCN formula, which aggregates information from first-order neighborhoods, leads to limited expressive capabilities of GCN and a lack of global information integration. Furthermore, increasing the depth of GCN layers does not necessarily guarantee improved performance. This may result in the inability of GCN alone to establish global label semantic dependencies. To address this issue, we propose a Global Relationship Enhancement Module (GREM), as illustrated in Figure 3. In the Global Relationship Enhancement Module (GREM), the input tensor $X_f$ is first linearly transformed to generate the Query (Q), Key (K), and Value (V) tensors. These transformations can be expressed as follows:

$$\begin{array}{c}\hfill Q=W_Q{X}_f+b_Q,K=W_K{X}_f+b_K,V=W_V{X}_f+b_V,\end{array}$$

(25)

where $W_Q$, $W_K$, and $W_V$ are the weight matrices for the Query, Key, and Value tensors, respectively; and $b_Q$, $b_K$, and $b_V$ are the corresponding bias terms. Next, the Query tensor Q undergoes a MaxPooling operation, which is followed by a repeat operation to match the original dimension of Q:

$$\begin{array}{c}\hfill Q^{\prime }=\mathrm{MaxPool}\left(Q\right),Q^{″}=\mathrm{Repeat}\left(Q^{\prime }\right),\end{array}$$

(26)

where $Repeat(·)$ represents the repeat operation and $MaxPool(·)$ is the maxpooling operation. The **attention mechanism** is applied by computing the dot product of the Query and Key tensors, which is followed by a Softmax operation to compute the attention weights:

$$\begin{array}{c}\hfill \text{Max-Atten}=\mathrm{softmax}\left({\displaystyle \frac{Q{K}^T+Q^{″}}{\sqrt{d_k}}}\right)V,\end{array}$$

(27)

where $d_k$ is the dimensionality of the Query tensor. The output is then multiplied by the Value tensor V. For each attention head, the output is denoted as $H_1,H_2,\dots ,H_h$, where h is the number of attention heads:

$$\begin{array}{c}\hfill H_i={\text{Max-Atten}}_i\mathrm{for}i\in \{1,2,\dots ,h\}.\end{array}$$

(28)

After computing the attention outputs for each head, they are concatenated to form the final tensor $X_f^{\prime }$:

$$\begin{array}{c}\hfill X_f^{\prime }=\mathrm{Concat}(H_1,H_2,\dots ,H_h).\end{array}$$

(29)

Finally, the output of GREM, $f_{GREM}$, is obtained by applying a fully connected layer:

$$\begin{array}{c}\hfill f_{GREM}=X_f^{\prime }{W}^o+b^o,\end{array}$$

(30)

where $W^o$ is the weight matrix and $b^o$ is the bias term of the fully connected layer. In summary, GREM utilizes multiple attention heads to enhance the global relationships among nodes in the graph, with each head focusing on different parts of the graph and their interactions. The operations are designed to strengthen the global features and improve the model’s representation of distant label dependencies.

4. Experiments

In this section, we begin by describing the evaluation metrics and experimental details, followed by introducing the two datasets, PASCAL VOC 2007 and MS-COCO, and then we report the experimental results on both datasets. Subsequently, we conducted ablation studies from four perspectives, and, finally, we now present the results of the visualization analysis.

4.1. Evaluation Metrics

Based on previous work [12,46], we adopted several evaluation metrics for assessing the performance of our model, including average per-class precision (CP), recall (CR), F1 (CF1), average overall precision (OP), recall (OR), and F1 (OF1). CF1 comprehensively assesses the model’s accuracy and recall, making it particularly suitable for evaluating the model’s performance on the positive class and aiding in finding a balance between precision and recall. OF1 comprehensively considers the system’s accuracy and recall, and it is typically used to evaluate the overall performance of the system in detection or recognition tasks. For each image, if the confidence score for a label is greater than 0.5, the label is predicted as positive. Additionally, we present the results for the top three predicted labels, and we also calculated the mean average precision (mAP).

4.2. Implementation Details

In SFDM, we employed ResNet-101 as the backbone network, which was pre-trained on the ImageNet [25] dataset to leverage its feature extraction capabilities. In FEM, the scaling control coefficients $\alpha$ and $\beta$ were set to 0.1 and 0.04, respectively, to control the relative weight of the feature enhancement mechanism. For LSDM, the Graph Convolutional Network (GCN) was constructed with two layers, where the output dimensions of the first and second layers were 1024 and 2048, respectively, to capture semantic dependencies. For the calculation of the correlation matrix, the parameters $\tau$ and p were set to 0.4 and 0.3, respectively. During the training process, the input images were randomly cropped and resized to a uniform resolution of 448 × 448 pixels, which was followed by random horizontal flipping to augment the data. For network optimization, we used Stochastic Gradient Descent (SGD) with a momentum of 0.9 and a weight decay of ${10}^{-4}$ to prevent overfitting. The initial learning rate was set to 0.01, and the network was trained for a total of 100 epochs to ensure convergence. Additionally, we incorporated a warm-up scheduler to gradually increase the learning rate during the initial two epochs of training to stabilize the optimization process. Our network implementation was based on the PyTorch 3.10 framework, ensuring efficient and scalable training.

4.3. Baseline Model Parameters

To help readers better evaluate the effectiveness and fairness of the comparative analysis, we explicitly present the parameters of some baseline models as follows:

MulCon [47]: This model uses a projector Proj() with two linear layers and $ReLU$ activation. Pretraining involves employing an Adam optimizer with a 1-cycle policy (max learning rate = $2\times {10}^{-4}$, while fine tuning involves using SGD with momentum = 0.9, weight decay = ${10}^{-4}$, and an initial learning rate of 0.01 (which is reduced by 10 every 20 epochs). Data augmentation includes resizing to $448\times 448$, random flips, and random augmentation.

VSGCN [48]: The model is trained using the Adam optimizer with an initial learning rate of ${10}^{-4}$ for PASCAL VOC 2007 (decaying by a factor of 10 every 20 epochs) and $5\times {10}^{-5}$ for MS-COCO (decaying by a factor of 5 every 15 epochs). A $LeakyReLU$ activation function (negative slope = 0.2) and dropout layer (drop rate = 0.1) are applied in the Multi-Head GCN mechanism. The backbone network is ResNet-101, which is pre-trained on ImageNet, and 300-dimensional GloVe embeddings are used as semantic prototypes. Input images are resized to $448\times 448$, and training is conducted on two TITAN Xp GPUs with a batch size of 32.

CFMIC [18]: The model is implemented using PyTorch 3.10. Input images are resized to $448\times 448$, and the feature aggregation parameter is set to 0.5. For label co-occurrence embedding, each object is represented as a 300-dimensional GloVe vector. In the cross-modal fusion module, m = 358 is used for feature fusion and g = 2 is applied for group sum-pooling. The model is trained with a batch size of 32.

LGR [49]: The model is implemented using PyTorch with ResNet-101 (pre-trained on ImageNet) as the backbone network. GloVe embeddings are used for label representation, with multi-word labels averaged. Key hyperparameters include s = 0.3, 0.2, and 0.3; and p = 0.2, 0.2, and 0.3 for MS-COCO, Pascal VOC 2007, and Pascal VOC 2012, respectively. The GGNN steps are set to $T_l$ = 1 and $T_f$ = 3. During training, input images are resized to $448\times 448$ with random cropping and flipping. The SGD optimizer is used with a batch size of 32, a momentum of 0.9, a weight decay of ${10}^{-5}$, and an initial learning rate of 0.001 (reduced by a factor of 10 on plateaus).

GCN-MS-SGA [13]: This model uses ResNet-101 (pre-trained on ImageNet) as the backbone network. Label embeddings are generated using 300-dimensional GloVe vectors, with multi-word labels represented as the average of their word embeddings. Input images are resized to $448\times 448$ with random cropping and horizontal flipping for data augmentation. The SGD optimizer is used with a momentum of 0.9, a weight decay of ${10}^{-4}$, and an initial learning rate of 0.05, which decays by a factor of 10 every 50 epochs for a total of 100 epochs.

ML-GCN [12]: This model uses ResNet-101 (pre-trained on ImageNet) as the backbone. Label embeddings are 300-dimensional GloVe vectors, with multi-word labels averaged. $LeakyReLU$ is used for activation. Input images are resized to $448\times 448$ with random cropping and flipping. The SGD optimizer is used with a momentum of 0.9, a weight decay of ${10}^{-4}$, and an initial learning rate of 0.01, decaying by 10 every 40 epochs for 100 epochs.

4.4. Experimental Results

To validate the effectiveness of our model, we conducted experiments on the PASCAL VOC (2007 and 2012) and MS-COCO datasets, and we also compared the results with other state-of-the-art methods. For a fair comparison, we directly adopted the results reported in the corresponding literature for all baseline models, ensuring that each model was evaluated under its optimal settings, as provided by the original authors. For detailed parameter configurations, readers can refer to the original papers cited in this work. Additionally, we performed ablation studies to further analyze the components and evaluated their contributions. Finally, we conducted network visualization analysis to intuitively display the regions of interest identified by the network.

4.4.1. Comparisons with State-of-the-Art Methods

Comparisons on PASCAL VOC 2007. The VOC 2007 dataset serves as a standard benchmark for evaluating the capability of image classification and recognition. The dataset consists of a training set (5011 images) and a test set (4952 images), comprising a total of 9963 images across 20 categories. To ensure a fair comparison with other state-of-the-art methods, we report the results in terms of average precision (AP) and mean average precision (mAP). The experimental results on the VOC 2007 dataset are presented in

Table 1. Comparisons of the AP and mAP with the state-of-the-art methods on the VOC 2007 dataset. (Bold indicates the best performance in its category, and the same applies hereafter).

Methods	Aero	Bike	Bird	Boat	Bot	Bus	Car	Cat	Chair	Cow	Table	Dog	Horse	Mot	Pers	Plant	Sheep	Sofa	Train	Tv	mAP
CNN-RNN [39]	96.7	83.1	94.2	92.8	61.2	82.1	89.1	94.2	64.2	83.6	70.0	92.4	91.7	84.2	93.7	59.8	93.2	75.3	99.7	78.6	84.0
ResNet-101 [26]	99.5	97.7	97.8	96.4	65.7	91.8	96.1	97.6	74.2	80.9	85.0	98.4	96.5	95.9	98.4	70.1	88.3	80.2	98.9	89.2	89.9
FeV+LV [50]	97.9	97.0	96.6	94.6	73.6	93.9	96.5	95.5	73.7	90.3	82.8	95.4	97.7	95.9	98.6	77.6	88.7	78.0	98.3	89.0	90.6
Atten-Reinforce [51]	98.6	97.1	97.1	95.5	75.6	92.8	96.8	97.3	78.3	92.2	87.6	96.9	96.5	93.6	98.5	81.6	93.1	83.2	98.5	89.3	92.0
ATL [52]	99.0	97.2	96.6	96.2	75.4	92.0	96.8	97.2	79.0	93.6	89.3	97.0	97.5	94.0	98.8	81.6	94.3	85.8	98.7	90.6	92.5
ML-GCN [12]	99.5	98.5	98.6	98.1	80.8	94.6	97.2	98.2	82.3	95.7	86.4	98.2	98.4	96.7	99.0	84.7	96.7	84.3	98.9	93.7	94.0
GCN-MS-SGA [13]	99.6	98.3	98.0	97.5	81.0	93.1	97.5	98.5	86.3	88.3	89.2	95.5	98.0	96.1	98.3	89.0	96.7	91.6	97.9	92.3	94.2
LGR [49]	99.6	95.6	97.3	96.4	84.0	95.8	94.1	98.9	86.9	96.8	86.8	98.7	98.6	96.9	98.8	84.8	97.2	83.7	98.8	93.6	94.2
FLNet [53]	99.6	98.7	98.9	97.9	84.6	95.3	96.2	96.5	85.6	96.1	87.2	97.7	98.6	97.0	98.1	86.5	97.4	86.5	98.8	90.8	94.4
CFMIC [18]	99.7	98.5	98.8	98.3	83.9	96.5	97.5	98.8	83.1	96.1	87.4	98.6	98.9	97.2	99.0	85.4	97.1	84.9	99.2	94.2	94.7
SSGRL [14]	99.7	98.4	98.0	97.6	85.7	96.2	98.2	98.8	82.0	98.1	89.7	98.8	98.7	97.0	99.0	86.9	98.1	85.8	99.0	93.7	95.0
VSGCN [48]	99.8	98.6	98.7	98.6	85.2	96.9	98.2	98.6	83.6	96.5	87.8	99.1	99.0	97.5	99.3	85.8	96.3	87.6	98.6	95.2	95.1
MulCon [47]	99.8	98.3	99.3	98.6	83.3	98.4	98.0	98.3	85.8	98.3	90.5	99.3	98.9	96.6	98.8	86.3	99.8	87.3	99.8	96.1	95.6
Ours	99.6	99.1	98.8	98.9	85.8	97.9	98.5	99.1	87.9	98.1	92.9	99.1	98.6	98.6	99.3	90.9	98.8	91.3	99.8	97.0	96.5

, where we compared our method with 13 other state-of-the-art approaches. Thanks to the establishment of our two-dimensional dependencies, our approach yielded significantly better results compared to methods that solely rely on image feature extraction or those that build one-dimensional dependencies. Our proposed model achieved an mAP of 96.5%, indicating superior performance compared to the other methods.

Comparisons on PASCAL VOC 2012. The PASCAL VOC2012 dataset is a challenging and widely recognized benchmark in the field of computer vision, and it is commonly used for tasks such as object detection, image classification, and semantic segmentation. It consists of a diverse collection of images, each annotated with detailed labels that specify object boundaries, categories, and other relevant information. Our experimental results, as presented in

Table 2. Comparisons of the AP and mAP with state-of-the-art methods on the VOC 2012 dataset.

Methods	Aero	Bike	Bird	Boat	Bot	Bus	Car	Cat	Chair	Cow	Table	Dog	Horse	Mot	Pers	Plant	Sheep	Sofa	Train	Tv	mAP
Fev+Lv [50]	98.4	92.8	93.4	90.7	74.9	93.2	90.2	96.1	78.2	89.8	80.6	95.7	96.1	95.3	97.5	73.1	91.2	75.4	97.0	88.2	89.4
HCP [54]	99.1	92.8	97.4	94.4	79.9	93.6	89.8	98.2	78.2	94.9	79.8	97.8	97.0	93.8	96.4	74.3	94.7	71.9	96.7	88.6	90.5
RCP  [55]	99.3	92.2	97.5	94.9	82.3	94.1	92.4	98.5	83.8	93.5	83.1	98.1	97.3	96.0	98.8	77.7	95.1	79.4	97.7	92.4	92.2
MCAR [10]	99.6	97.1	98.3	96.6	87.0	95.5	94.4	98.8	87.0	96.9	85.0	98.7	98.3	97.3	99.0	83.8	96.8	83.7	98.3	93.5	94.3
SSGRL [14]	99.7	96.1	97.7	96.5	86.9	95.8	95.0	98.9	88.3	97.6	87.4	99.1	99.2	97.3	99.0	84.8	98.3	85.8	99.2	94.1	94.8
Ours	99.7	97.5	99.0	97.6	87.1	97.7	96.2	99.7	90.2	98.4	89.3	99.1	99.3	97.4	99.0	88.0	99.1	88.5	99.7	97.3	96.0

, demonstrate that our network achieved an impressive 96.0% on the comprehensive evaluation metric mAP. This performance surpasses the current state-of-the-art by 1.2%. This improvement can be attributed to the effective two-dimensional dependencies we established, which simultaneously consider both image feature dependencies and label semantic dependencies. Additionally, global relationships were also taken into account when constructing these dependencies.

Comparisons on MS-COCO. The COCO dataset is a large and diverse dataset for object detection, segmentation, and captioning. It aims at scene understanding and is primarily composed of images captured from complex everyday scenes, with precise object localization achieved through accurate segmentation. The dataset consists of 91 object categories, 328,000 images, and 2,500,000 labels. It currently stands as the largest dataset for semantic segmentation—with 80 provided categories and over 330,000 images, including 200,000 annotated images—containing over 1.5 million instances in total. The experimental results, as shown in

Table 3. Comparisons with state-of-the-art methods on the MS-COCO dataset.

Methods	ALL							TOP-3
	mAP	CP	CR	CF1	OP	OR	OF1	CP	CR	CF1	OP	OR	OF1
CNN-RNN [39]	61.2	-	-	-	-	-	-	66.0	55.6	66.4	69.2	66.4	67.8
SRN [40]	77.1	81.6	65.4	71.2	82.7	69.9	75.8	85.2	58.8	67.4	87.4	62.5	72.9
ResNet-101 [26]	77.3	80.2	66.7	72.8	83.9	70.8	76.8	84.1	59.4	69.7	89.1	62.8	73.6
ML-GCN [12]	83.0	85.1	72.0	78.0	85.8	75.4	80.3	89.2	64.1	74.6	90.5	66.5	76.7
GCN-MS-SGA [13]	83.4	85.1	71.6	77.8	84.0	75.0	79.3	88.8	63.4	75.7	88.8	66.0	75.7
MCAR [10]	83.8	85.0	72.1	78.0	88.0	73.9	80.3	88.1	65.5	75.1	91.0	66.3	76.7
CFMIC [18]	83.8	85.8	72.7	78.7	86.3	76.3	81.0	89.7	64.5	75.0	90.7	67.3	77.3
LGR [49]	83.9	85.0	73.3	78.7	86.2	76.4	81.0	89.0	64.8	75.0	90.7	67.0	77.1
FLNet [53]	84.1	84.9	73.9	79.0	85.5	77.4	81.1	89.0	65.2	75.2	90.4	67.5	77.3
LGLM [56]	84.2	85.7	72.8	78.7	86.6	76.7	81.3	89.4	64.7	75.0	90.7	67.4	77.3
STMG [57]	84.3	85.8	72.7	78.7	86.7	76.8	81.5	89.3	64.8	75.1	90.8	67.4	77.4
Ours	85.2	84.1	75.9	79.8	85.2	78.7	81.8	88.7	66.5	76.0	90.5	68.1	77.7

, demonstrate a comparison between our proposed method and 11 state-of-the-art approaches, including CNN-RNN [39], SRN [40], ResNet-101 [26], ML-GCN [12], GCN-MS-SGA [13], MCAR [10], CFMIC [18], LGR [49], FLNet [53], LGLM [56], and STMG [57]. Compared to the state-of-the-art methods, our approach exhibited superior performance, indicating the effectiveness of our proposed modeling of two-dimensional dependencies.

4.4.2. Statistical Significance Experiment

To demonstrate that our data were significantly superior to the other groups, we first used the Wilcoxon signed-rank test to check if there was a significant statistical difference between our data and the other groups. The Wilcoxon signed-rank test is a non-parametric method thaat is used to compare two related samples. It involves calculating the differences between data pairs, ranking the absolute differences, assigning signs based on the direction, and then calculating the positive and negative rank sums. The p-value is derived from comparing the rank sums, and if it is below the significance level (typically 0.05), a significant difference is indicated. The PASCAL VOC 2007 dataset was utilized for validation, wherein the results of 13 comparison methods were assessed for significance against those of the method. The Wilcoxon signed-rank test was employed to analyze the significance, with the results depicted in

Table 4. Wilcoxon signed-rank test results table.

Methods	CNN-RNN [39]	ResNet-101 [26]	FeV+LV [50]	Atten-Reinforce [51]	ATL [52]	ML-GCN [12]	GCN-MS-SGA [13]	LGR [49]	FLNet [53]	CFMIC [18]	SSGRL [14]	VSGCN [48]	MulCon [47]
p-value	$8.83\times {10}^{-5}$	$8.84\times {10}^{-5}$	$8.83\times {10}^{-5}$	$8.84\times {10}^{-5}$	$8.82\times {10}^{-5}$	$8.78\times {10}^{-5}$	$1.54\times {10}^{-4}$	$1.96\times {10}^{-4}$	$2.32\times {10}^{-4}$	$2.91\times {10}^{-4}$	$2.44\times {10}^{-4}$	$6.26\times {10}^{-4}$	$2.28\times {10}^{-2}$
Confidence interval	[8.52, 16.50]	[3.38, 9.76]	[3.72, 8.12]	[3.03, 5.96]	[2.57, 5.37]	[1.38, 3.57]	[1.26, 3.41]	[1.30, 3.37]	[1.24, 2.96]	[0.88, 2.81]	[0.66, 2.40]	[0.63, 2.28]	[0.21, 1.64]

. Notably, all p-values were found to be less than the significance threshold (a = 0.05). This indicates that the results of our model showed a significant difference compared to the results of the other 13 models. Secondly, we further compared the mean values (i.e., mAP) of each group to determine which group performed better in terms of numerical value. It was clear that our model had a higher mAP compared to the other 13 models (see Table 1), indicating that the performance of our model is superior to the other groups. Taken together, these two points demonstrate that our method is statistically superior to other methods, and the advantages over different comparison methods are not coincidental, but rather stable and consistent. In addition, we calculated the 95% confidence intervals for the mean differences. The calculation of the confidence intervals was based on the results of multiple independent experiments, which can more comprehensively reflect the stability of the model’s performance. The experimental results indicate that the differences were statistically significant, and the estimation precision was high. By introducing confidence intervals, we were able to more intuitively assess the variability of the model’s performance, thereby enhancing the credibility and scientific rigor of the conclusions.

4.4.3. Ablation Studies

As shown in this section, we conducted ablation studies from four different aspects, including the impact of different modules in TDDM on classification results; the influence of calculation methods for fusion factor $B_f$ and compensation factor $C_f$ in FFM feature fusion; the effects of weight proportion control coefficients $\alpha$ and $\beta$ in FEM feature enhancement; and the influence of threshold values $\tau$ and p in GCN for computing correlation matrices. We conducted the aforementioned ablation experiments on the VOC dataset.

Effects of different modules. To evaluate the effectiveness of our model, we conducted ablation studies by reconstructing the model with different factors. To validate the necessity of modeling dependencies in both the image and semantic dimensions, we conducted ablation experiments by individually removing the SFDM and LSDM branches, as shown in the second and third rows of

Table 5. Ablation study on the impact of different modules (FFM, FEM, GCN, and GREM) on the model performance when using the VOC2007 dataset.

SFDM		LSDM		mAP
FFM	FEM	GCN	GREM
✔	✔	✔	✔	96.5
✔	✔			95.0
		✔	✔	94.3
	✔	✔	✔	94.9
✔		✔	✔	94.6
✔	✔	✔		95.7

. The results indicate that using either the SFDM or LSDM branch alone yielded inferior performance compared to modeling dependencies in both dimensions. This strongly demonstrates the complementarity of the two branches and the effectiveness of modeling image feature spatial dependencies and semantic label dependencies. Furthermore, to further verify the effectiveness of each module in the dual branches, we conducted individual ablation experiments on each module. As the GCN module is concatenated before the GREM and serves as the foundation for LSDM, it cannot be individually ablated. Therefore, we selected FFM, FEM, and GREM as separate ablation modules. The results of the ablation experiments are presented in rows four to six of Table 5. The experimental results thoroughly validated the effectiveness of each module. We conducted statistical significance tests for the individual modules in the ablation study using Wilcoxon signed-rank tests. The results, as shown in

Table 6. Ablation study of the Wilcoxon signed-rank test results.

Modules	LSDM	SFDM	FFM	FEM	GREM
p-value	$2.44\times {10}^{-4}$	$6.34\times {10}^{-4}$	$5.36\times {10}^{-4}$	$3.50\times {10}^{-4}$	$3.44\times {10}^{-2}$

, indicate that the p-values for all comparisons were below 0.05, confirming that the observed differences were statistically significant. Additionally, we further analyzed the impact of each module by examining the mAP values, and the results demonstrate that each module contributed significantly to the model’s performance.

Effects of different ${\mathit{B}}_{\mathit{f}}$ and ${\mathit{C}}_{\mathit{f}}$. Determining the calculation of the fusion factor $B_f$ and the compensation factor $C_f$ in FFM is a key aspect that warrants our attention. Additionally, we aimed to ensure that the computation method does not introduce any additional computational burden. Therefore, we explored relatively straightforward approaches, such as the element-wise addition (add) or multiplication (mul) of feature vectors, to perform these calculations. The combined methods and identification results are presented in

Table 7. Impact of different calculation methods for fusion factor $B_f$ and compensation factor $C_f$ on the model performance when using the VOC2007 dataset.

Methods		All
	mAP	OF1	CF1
add($B_f$)add($C_f$)	94.8	92.2	90.5
mul($B_f$)add($C_f$)	96.3	93.2	92.0
mul($B_f$)mul($C_f$)	95.5	93.0	91.6
add($B_f$)mul($C_f$)	96.5	93.2	92.2

. We explored four fusion methods in various combinations, and it can be observed that adding the feature vectors as the fusion factor $B_f$ and multiplying the feature vectors as the compensation factor $C_f$ yielded the best combination approach.

Effects of different $\mathit{\alpha }$ and $\mathit{\beta }$. To investigate the impact of the weight proportion control coefficients $\alpha$ and $\beta$ on the results of the global enhanced attention in the FEM, we conducted experiments using different weighting coefficients, as shown in Figure 4. We changed the values of $\alpha$ in a set of {0.02, 0.05, 0.1, 0.15, 0.2, 0.25} and $\beta$ in a set of {0.02, 0.03, 0.04, 0.05, 0.06, 0.07}. We can observe that the model achieved the best performance when $\alpha$ = 0.1 and $\beta$ = 0.04. We hypothesized that when $\alpha$ and $\beta$ become excessively large, they accentuate feature attention and fusion attention while neglecting the crucial global average attention, resulting in a decrease in mAP.

Effects of different thresholds $\mathit{\tau }$ and $\mathit{p}$. In order to investigate the impact of threshold values $\tau$ and p on the accuracy of computing correlation matrices, we conducted experiments using different $\tau$ and p settings. Relevant literature [12,13,49] was reviewed to determine the approximate parameter ranges used in similar models. These references have furnished us with preliminary guidelines for the scope. We selected the following values for $\tau$, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1, and for p, 0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, and we also observed the impact of different combinations on the model’s performance. As shown in Figure 5, the best performance was achieved when $\tau$ = 0.4 and p = 0.3. If the threshold $\tau$ was set too large, it may result in too many edges being retained, failing to filter out noise, which affects the model’s performance. On the other hand, if $\tau$ is set too small, too much information may be discarded, preventing the network from capturing sufficient structural information, leading to a decline in model performance. Regarding the value of p, if p is too small, the nodes in the graph may not gather enough information from their related nodes. On the other hand, if p is too large, it can lead to excessive smoothing, causing the node features to become overly similar and losing important distinctions. Both scenarios can negatively impact the model’s performance.

4.4.4. Computational Cost Analysis

Inference Time Analysis In this study, inference time serves as the metric for evaluating model computational complexity. It directly reflects the model’s computational speed in real-world applications, thereby demonstrating its performance under practical conditions. Image inputs were set to 224 × 224, and the inference tests were conducted on an RTX 3060 graphics card. The test results are shown in

Table 8. Inference time comparison results.

Methods	ResNet-101 [26]	ML-GCN [12]	MCAR [10]	GCN-MS-SGA [13]	Ours
Inference time/ms	32.22	139.28	251.70	351.18	172.34

. The experimental findings indicate that the proposed method exhibits slower inference times compared to backbone networks such as ResNet-101 and the classical ML-GCN method. This can be attributed to the simpler structures and lower computational complexities of these base networks, resulting in shorter inference times. Conversely, methods like GCN-MS-SGA and MCAR, despite demonstrating higher performance, exhibit longer inference times due to their complex network structures and extensive computational operations. Overall, the proposed method strikes a balance between performance and computational complexity, delivering superior performance while maintaining an advantage in inference time. As such, it proves suitable for addressing efficient inference requirements in practical applications.

Spatial complexity analysis In the experiments, the different models’ spatial complexity was compared, as shown in

Table 9. Comparison table of parameter quantity.

Methods	ResNet-101 [26]	ML-GCN [12]	LGLM [56]	Ours
Number of parameters/M	44.50	42.50	44.04	45.21

, which lists the parameter counts for each model. Specifically, the parameter counts for ResNet-101, ML-GCN, LGLM, and the proposed method were 44.50 M, 42.50 M, 44.04 M, and 45.21 M, respectively. The proposed method has a relatively higher parameter count, primarily due to the inclusion of FFM, FEM, and GREM. These modules were incorporated to better integrate and enhance feature information to establish more effective dependencies, but they inevitably introduced additional computational overhead. Although the proposed model has a higher parameter count, trading a certain amount of parameter count for accuracy improvements is considered valuable. Efforts have been made to balance performance and parameter count to ensure that the model improves accuracy while keeping the parameter count under control. Additionally, the section on Conclusions and Future Work provides a detailed discussion on how to further reduce model complexity in future research. Plans include optimizing the model structure and incorporating more efficient techniques to reduce computational overhead and the number of parameters, with the aim of achieving a better balance between performance and complexity.

4.5. Visualization

To further validate the effectiveness of our model, we visualized heatmaps, as shown in Figure 6. We utilize CAM (class activation map) to highlight the regions of interest in the images, as perceived by the network. CAM is a technique employed to visualize the internal workings of a neural network. It aids in understanding the network’s focus on different objects or features in classification tasks and is used for visualizing the decision-making process of neural networks in image classification tasks. We compared ResNet-101, MobileNet, and our proposed model. The experimental results demonstrate that our model exhibits a stronger interest in the regions containing the labels, whereas ResNet-101 and MobileNet show a less focused and broader interest, resulting in inferior performance.

5. Conclusions and Future Work

In this paper, we propose the Two-Dimensional Dependency Model (TDDM) for MLIC. The dual branches of TDDM address two main issues: firstly, modeling object spatial feature dependencies to establish long-range dependencies and alleviate the limitations of fixed convolutional kernel receptive fields; secondly, modeling and enhancing global label semantic dependencies to address issues arising from the local semantics introduced by GCN itself. Our model outperforms previous methods on both the PASCAL VOC 2007 and MS-COCO datasets. We believe that its excellent performance and clear architecture provide a strong foundation for future research in multi-label image classification. In future work, we will focus on addressing some of the limitations of the proposed method. Firstly, due to the use of cross-layer feature fusion in the proposed method, a certain degree of feature confusion occurs, making it more challenging to distinguish similar image features. To address this issue, transfer learning will be introduced. Transfer learning allows the model to leverage learned features and representations from related tasks or domains, thereby better distinguishing similar features and reducing confusion. This approach aims to enhance the model’s robustness and generalization capabilities, making it easier for the model to differentiate similar image features and address the limitations of the current method. Secondly, the method uses a fixed correlation matrix, which may limit the model’s ability to capture dynamic relationships in the data and reduce its performance across different tasks and datasets. Therefore, we plan to develop correlation matrices with adaptive algorithms in the future to address this limitation. These adaptive algorithms can dynamically adjust the correlation matrices based on the input data, further improving the model’s performance and adaptability across a broader range of computer vision applications. This improvement will not only enhance the model’s accuracy, but also expand its applicability, ensuring superior performance in various practical scenarios. Thirdly, the focus will be on model lightweighting to enhance computational efficiency and practical applicability. This includes optimizing the model structure, employing depthwise separable convolutions to reduce parameter count and computational complexity, and utilizing knowledge distillation techniques to transfer knowledge from a large model to a smaller one, thereby improving its performance and efficiency. Additionally, model pruning and quantization techniques will be applied to further reduce computational overhead and memory usage. These measures are expected to enhance the applicability of the model in resource-constrained environments while maintaining high performance levels.