Proxy-Based Semi-Supervised Cross-Modal Hashing

Abstract

Due to the difficulty in obtaining label information in practical applications, semi-supervised cross-modal retrieval has emerged. However, the existing semi-supervised cross-modal hashing retrieval methods mainly focus on exploring the structural relationships between data and generating high-quality discrete pseudo-labels while neglecting the relationships between data and categories, as well as the structural relationships between data and categories inherent in continuous pseudo-labels. Based on this, Proxy-based Semi-Supervised Cross-Modal Hashing (PSSCH) is proposed. Specifically, we propose a category proxy network to generate category center points in both feature and hash spaces. Additionally, we design an Adaptive Dual-Label Loss function, which applies different learning strategies to discrete ground truth labels and continuous pseudo-labels and adaptively increases the training weights of unlabeled data with more epochs. Experiments on the MIRFLICKR-25K, NUS-WIDE, and MS COCO datasets show that PSSCH achieves the highest mAP improvements of 3%, 1%, and 4%, respectively, demonstrating better results than the latest baseline methods.

1. Introduction

With the exponential growth of multimedia data on the internet, the ability to efficiently search for and retrieve relevant information has become a crucial task. Deep cross-modal hashing, which maps different modalities (e.g., images, text, and videos) into a common low-dimensional space, has emerged as a powerful solution for fast and efficient retrieval [1,2,3,4,5,6,7,8,9]. This technique enables the use of binary codes to represent data, making retrieval both fast and scalable. However, the effectiveness of cross-modal hashing methods often relies on the availability of large amounts of labeled data for training, which is not always feasible in real-world applications [10,11,12,13].

Due to the difficulty in obtaining labeled data, especially for large-scale datasets, semi-supervised learning techniques have gained increasing attention in the field of cross-modal retrieval. These methods aim to leverage both labeled and unlabeled data, reducing the reliance on expensive and time-consuming manual labeling processes. Semi-supervised learning seeks to effectively utilize the wealth of unlabeled data, which are often more abundant and easier to collect, while maximizing the information extracted from the relatively smaller labeled dataset. By incorporating both labeled and unlabeled data into the training process, these approaches have shown great potential for improving retrieval performance in scenarios where labeled data are scarce [14,15,16].

Motivation. The existing semi-supervised cross-modal hashing retrieval methods focus on exploring the similarity relationships between different modalities, aiming to map similar data closer together and dissimilar data farther apart in a shared low-dimensional space while preserving these relationships [17,18]. However, they often neglect the “data–category” relationships, which leads to suboptimal retrieval performance. Moreover, these methods primarily concentrate on generating high-quality discrete pseudo-labels, overlooking the structural relationships between data and categories inherent in continuous pseudo-labels. For example, when predicting the pseudo-label of an unlabeled data sample A, a confidence distribution indicating its likelihood of belonging to certain categories is obtained. This distribution contains rich structural information about the relationships between data and categories. However, the existing methods fail to fully utilize this information, instead setting a threshold to discretize the continuous labels [19,20,21,22]. Therefore, effectively leveraging the information within continuous pseudo-labels remains a significant challenge.

Our Method. To answer the above questions, we proposed a novel Proxy-based Semi-Supervised Cross-modal Hashing (PSSCH) method for semi-supervised cross-modal retrieval. Our proposed method introduces a category proxy network (CPNet) and a novel Adaptive Dual-Label Loss to address the challenges in semi-supervised cross-modal hashing. The CPNet consists of two key components: feature proxies and hash proxies. Feature proxies are learned and updated based on labeled data features. Each proxy represents the category center point of a category in the feature space, providing a reliable basis for generating pseudo-labels for unlabeled data. Hash proxies represent category centers in the hash space and are obtained by forward-propagating the feature proxies. This process ensures that the hash proxies preserve the structural relationships between feature proxies in the feature space as much as possible in the hash space. By learning excellent feature proxies and hash proxies via the CPNet, we can consider both data–data and data–category relationships during model training.

The Adaptive Dual-Label Loss is specifically designed to handle the differences between discrete ground truth labels (labeled data) and continuous pseudo-labels (unlabeled data). By adopting different learning strategies, it effectively optimizes both types of labels. For discrete ground truth labels, it ensures that labeled data cluster around their corresponding categories in the hash spaces while preserving the structural relationships between the data. Unlike discrete labels, continuous pseudo-labels can capture subtle relationships between data and multiple categories. Directly discretizing pseudo-labels would result in a loss of information. Therefore, in our learning strategy, we fully leverage the rich structural information inherent in continuous pseudo-labels, thereby capturing the confidence relationships between the data and all the categories. Additionally, we dynamically adjust the weight of the loss for unlabeled data based on the epoch. This ensures that the model maintains the importance of labeled data while learning more stably from unlabeled data. In the early stages of training, the model relies more on true labels to avoid negative impacts from unlabeled data. Later, as the quality of the pseudo-labels improves, the weight gradually increases.

Contributions. To sum up, the main contributions of this article are threefold:

• Category Proxy Network: We design a CPNet to generate feature proxies and hash proxies, which work with two-modal hashing networks during training. This enables the model to consider both the relationships between data and the relationships between data and categories.
• Adaptive Dual-Label Loss: To capture the structural relationships between the data and categories contained in continuous pseudo-labels, an Adaptive Dual-Label Loss model is proposed.
• Experimental Validation: Extensive experiments on three public datasets demonstrate the superiority of our PSSCH method.

Roadmap. The overall architecture of this paper is as follows. Section 2 introduces some representative methods in the three fields of supervised cross-modal hashing, unsupervised cross-modal hashing, and semi-supervised cross-modal hashing, respectively. Section 3 introduces the proposed method in detail. The experimental results and analysis are provided in Section 4. Section 5 summarizes the conclusions and future work.

2. Related Works

In recent years, cross-modal hashing has gained significant attention due to its potential to improve the efficiency and scalability of cross-modal retrieval tasks. In this section, we summarize representative approaches in three main categories: supervised cross-modal hashing in Section 2.1, unsupervised cross-modal hashing in Section 2.2, and semi-supervised cross-modal hashing in Section 2.3.

2.1. Supervised Hashing Methods

Supervised cross-modal hashing methods utilize labeled data to learn hash codes, aiming to maintain semantic similarity across different modalities based on known label information. Among these methods, DCMH made a groundbreaking contribution by introducing deep learning into cross-modal hashing. It employs an end-to-end approach to learn the feature representations of raw data and optimize hash codes simultaneously [23]. Building on this, the MESDCH method further considers the distinction in similarity magnitudes between data under a multi-label setting, designing a multi-label semantic similarity module to explore inter-modal similarity relationships [24]. Since mapping data into the hash space involves quantization, HHF addresses the incompatibility between metric loss and quantization loss; it alleviates this issue by setting different thresholds based on varying hash code lengths and the number of categories [25]. SCCGDH designs a label network to generate distinct centers for different labels and uses these label centers to guide data learning [26]. DCPH is the first to introduce the concept of category proxies, generating hash proxies through a proxy network to guide the model learning process, enabling the model to fully consider the relationships between data and categories [27]. Considering that the generated category proxies do not account for the data distribution, DAPH adopts a two-step approach: it first integrates data features and labels into a unified framework to learn category proxies, which are then used for guidance [28]. DSPH proposes trainable category proxies and further adds a negative sample constraint to the proxy loss [29]. SCH, addressing the oscillation problem of the loss function, sets upper and lower bounds for similarity measures between data points to mitigate oscillations [30]. Finally, DCGH simultaneously considers relationships between data points as well as between data and categories, ensuring intra-class cohesion and preserving inter-class structural relationships [31].

2.2. Unsupervised Hashing Methods

In the absence of supervision, unsupervised cross-modal hashing methods utilize structural relationships among data to guide the training of hash models. DBRC models the relationships between different modalities and introduces a reconstruction framework [32]. UDCMH dynamically assigns different weights to data during the optimization process [33]. DJSRH explores the structural relationships of data across modalities and integrates them to guide model training [34]. AGCH employs various metric methods to aggregate structural relationships between different modalities and generates a similarity matrix [35]. CIRH designs a multi-modal collaborative graph to establish heterogeneous multi-modal correlations and performs semantic aggregation on graph networks to generate a complementary multi-modal representation [36].

2.3. Semi-Supervised Hashing Methods

Semi-supervised cross-modal hashing methods leverage both the label information of labeled data and the structural information of unlabeled data, gaining wide attention in real-world applications. For example, MGCH [19] proposes a novel multi-view graph-based cross-modal hashing framework to generate hash codes in a semi-supervised manner, using a graph reasoning module to process the output of multi-view graphs. MCGCN [37] designs an additional network within the two modality-specific networks to generate joint feature representations of the two modalities and employs graph convolution to propagate label information to unlabeled data. SSCH [22] enhances semantic information through an unaligned pseudo-labeling process in the presence of incorrect pairings and learns hash representations for different data using a label enhancement strategy. SCMB [38] introduces a cross-modal memory bank, dynamically storing the feature representations of each cross-modal data instance in a shared space and class probability representations in a label space to guide model training. SKDCH [21] adopts a student–teacher network to propagate knowledge and improve triplet ranking loss, effectively alleviating the heterogeneity gap. SPAL [39] uses shared semantic prototypes to associate labeled and unlabeled data in both modalities, minimizing intra-class variation and maximizing inter-class variation to improve the discriminative representation of unlabeled data. GCSCH [20] employs graph convolution networks to capture semantic information between multi-modal data of both ground truth and pseudo-labels and uses a teacher–student learning framework to transfer knowledge from the fusion module to image and text hashing networks.

3. Methodology

This section introduces the method we propose. First, the problem is described, followed by an overview of the PSSCH framework, and finally, a detailed discussion of our PSSCH method.

3.1. Problem Formulation

Problem Definition. Cross-modal hashing methods aim to map multimodal data (such as images and text) into a shared hash space to enable efficient cross-modal retrieval. Given an image or a text query, the goal is to generate binary hash codes through the learned hashing functions and retrieve the most relevant corresponding modality (for example, retrieving the corresponding text from an image or vice versa) based on their semantic similarity. This paper focuses on image–text cross-modal retrieval. Given a training dataset $\mathcal{D}=\left\{{\mathcal{D}}_l,{\mathcal{D}}_u\right\}$ containing N image–text pairs, ${\mathcal{D}}_l=\left\{{\mathit{X}}^l,{\mathit{Y}}^l,{\mathit{L}}^l\right\}$ represents the labeled sample set with $N_l$ labeled image–text pairs. The labeled images and texts are denoted as ${\mathit{X}}^l\in {\mathbb{R}}^{N_l\times d_x}$ and ${\mathit{Y}}^l\in {\mathbb{R}}^{N_l\times d_y}$, respectively, and ${\mathit{L}}^l\in {\{0,1\}}^{N_l\times C}$, where C is the number of categories. If the i-th sample belongs to the j-th class, then ${\mathit{L}}_{ij}^l=1$; otherwise, ${\mathit{L}}_{ij}^l=0$ [20]. On the other hand, ${\mathcal{D}}_u=\left\{{\mathit{X}}^u,{\mathit{Y}}^u\right\}$ represents the unlabeled sample set, where ${\mathit{X}}^u\in {\mathbb{R}}^{N_u\times d_x}$ and ${\mathit{Y}}^u\in {\mathbb{R}}^{N_u\times d_y}$ denote the unlabeled images and texts, respectively, with $N_u$ being the number of unlabeled samples. Finally, the total number of samples is $N=N_l+N_u$. Our goal is to learn the hashing functions ${\mathcal{H}}^x$ for the image modality and ${\mathcal{H}}^y$ for the text modality, and to obtain high-quality hash codes $B^x\in {\{-1,1\}}^K$ and $B^y\in {\{-1,1\}}^K$ through these hashing functions, where K is the number of bits in the hash code.

Notations. Without loss of generality, sets are denoted by math script upper case letters (e.g., $\mathcal{D}$). Scalers or constants are signed by uppercase letters (e.g., C). Matrices are represented in uppercase bold letters (e.g., $\mathit{A}$), and $(i,j)$-th element of $\mathit{A}$ is denoted by $A_{ij}$ [31]. Vectors are denoted by lowercase bold letters (e.g., $\mathit{a}$), and i-th element of $\mathit{a}$ is represented by $a_i$. Transpose of a matrix or a vector is denoted by a superscript T (e.g., ${\mathit{A}}^{\mathrm{T}}$). Functions are denoted by calligraphy uppercase letters (e.g., $\mathcal{L}$). The frequently used mathematical notations are summarized in

Table 1. A summary of common symbols and notations.

Notation	Definition
$\mathcal{D}$	Training Dataset
N	Number of Samples
$\mathcal{H}$	Hash Functions
$\mathcal{G}$	Feature Extractor
I	Indicator Function
${\mathit{x}}_i$	The i-th Image Data
${\mathit{y}}_i$	The i-th Text Data
C	the number of categories
K	The number of bits in the hash code
$\mathit{F}$	Features of Data
${\mathit{P}}^f$	Feature Proxies
${\mathit{P}}^h$	Hash Proxies
$\mathit{B}$	Hash Code
$\mathit{S}$	Similarity matrix
epoch num	The total number of iterations

for readability [31].

3.2. Overview of PSSCH Framework

As illustrated in Figure 1, the proposed PSSCH framework, PSSCH, consists of three networks: ImgNet, TxtNet, and CPNet.

ImgNet and TxtNet. Building on the ideas from [31,40,41], we introduce feature extraction based on the Transformer encoder in cross-modal retrieval, comprising 12 stacked encoder blocks. Each block includes Layer Normalization, Multi-Head Self-Attention, and MLP components, with a total of 12 attention heads. The process of extracting image semantic features is represented as ${\mathit{F}}_x^l={\mathcal{G}}_x({\mathit{X}}^l,{\mathit{\theta }}_x)$ and ${\mathit{F}}_x^u={\mathcal{G}}_x({\mathit{X}}^u,{\mathit{\theta }}_x)$, where ${\mathit{F}}_x^l$ and ${\mathit{F}}_x^u$ represent the semantic features of labeled and unlabeled images, respectively. ${\mathcal{G}}_x$ denotes the image semantic encoder, and ${\mathit{\theta }}_x$ refers to its parameters.

Similarly, the text Transformer encoder comprises 12 encoder blocks, each with 8 attention heads for Multi-Head Self-Attention [29,31]. The extraction of text semantic features is expressed as ${\mathit{F}}_y^l={\mathcal{G}}_y({\mathit{Y}}^l,{\mathit{\theta }}_y)$ and ${\mathit{F}}_y^u={\mathcal{G}}_y({\mathit{Y}}^u,{\mathit{\theta }}_y)$, where ${\mathit{F}}_y^l$ and ${\mathit{F}}_y^u$ represent the semantic features of labeled and unlabeled text, respectively. ${\mathcal{G}}_y$ represents the text semantic encoder, and ${\mathit{\theta }}_y$ corresponds to its parameters.

CPNet. The CPNet consists of four simple fully connected layers, with the dimensions of the intermediate layers matching the dimensions of image and text semantic features, and the output layer having a dimension equal to the hash code length K. One-hot encoding represents categorical information directly in a sparse discrete format, providing a clear distinction between different categories; we use the one-hot encoding of categories as input, and, through forward propagation, it produces the feature proxies ${\mathit{P}}^f={\left\{{\mathit{p}}_i^f\right\}}_{i=1}^C$ and hash proxies ${\mathit{P}}^h={\left\{{\mathit{p}}_i^h\right\}}_{i=1}^C$.

3.3. Model Learning

For the generation of pseudo-labels, we first calculate the cosine similarity between the features of unlabeled data and each category center (feature proxy) in the feature space. The higher the similarity, the greater the probability we assign to the data belonging to that category. We set a threshold for this; we retain data with a cosine similarity above the threshold, and, for those below the threshold, we consider the data not to belong to the category and set it to 0. Feature proxies are learned from the features of labeled data in the feature space. Since at the initial training stage the feature proxies have not yet been well trained, the quality of the pseudo-labels is not guaranteed. To address this, we set up a dynamic weight adjustment function that gradually increases the weight of unlabeled training data as the number of epochs increases. The hyper-parameters $\alpha$ and $\beta$ are used to explore the importance of considering the relationship between data and the relationships between data and categories during model learning. In this subsection, we will detail the generation method of pseudo-labels and the learning strategy of the model.

3.3.1. Pseudo-Label Generation

The pseudo-label generation process is based on cosine similarity [42,43]. First, we obtain the feature representations of the unlabeled data for the two modalities ${\mathit{F}}_x^u$ and ${\mathit{F}}_y^u$, as well as the feature proxies ${\mathit{P}}^f$. Then, we calculate the cosine similarity between the feature representations of the two modalities, ${\mathit{F}}_x^u$ and ${\mathit{F}}_y^u$, and the feature proxies ${\mathit{P}}^f$ using the pseudo-label generation module, as shown in Equation (1):

$${\mathit{S}}_x^u={\displaystyle \frac{{\mathit{F}}_x^u}{{∥{\mathit{F}}_x^u∥}_2}}·{\left({\displaystyle \frac{{\mathit{P}}^f}{{∥{\mathit{P}}^f∥}_2}}\right)}^T,{\mathit{S}}_y^u={\displaystyle \frac{{\mathit{F}}_y^u}{{∥{\mathit{F}}_y^u∥}_2}}·{\left({\displaystyle \frac{{\mathit{P}}^f}{{∥{\mathit{P}}^f∥}_2}}\right)}^T$$

(1)

where ${\parallel ·\parallel }_2$ is the $L_2$ norm, ${(.)}^T$ is the transpose of vector(or matrix), and ${\mathit{S}}_x^u,{\mathit{S}}_y^u\in {\mathbb{R}}^{N_u\times C}$ represent the similarity matrices between the image and text unlabeled data features and the feature proxies, respectively. Then, compute the average similarity ${\mathit{S}}^u$ by calculating the following Equation (2):

$${\mathit{S}}^u={\displaystyle \frac{{\mathit{S}}_x^u+{\mathit{S}}_y^u}{2}}$$

(2)

Hence, we generate the pseudo-labels ${\mathit{L}}^u$ using ${\mathit{S}}^u$ and a threshold, as shown in Equation (3):

$$\begin{array}{c}\hfill {\mathit{L}}^u=\left\{\begin{array}{cc}{\mathit{S}}^u\hfill & \mathrm{if}{\mathit{S}}^u\ge \mathbf{threshold}\hfill \\ 0\hfill & \mathrm{otherwise}\hfill \end{array}\right.\end{array}$$

(3)

In this paper, we set the threshold to 0.5. If ${\mathit{S}}^u\ge \mathbf{threshold}$, we retain the continuous values in ${\mathit{S}}^u$. If ${\mathit{S}}^u<\mathbf{threshold}$, following [30], we set it to 0.

3.3.2. Feature Proxy Learning

For the learning of feature proxies, we only constrain the relationship between the labeled data and the feature proxies to update the feature proxies. Specifically, we use the label information from the labeled data to bring the relevant feature proxies closer to the data and push the irrelevant feature proxies away from the data. For feature proxies that are related to the samples, we reduce the distance between the features and the relevant feature proxies by calculating the following Equation (4):

$${cos}_{+}(f,p)=-{\displaystyle \frac{|f·p|}{\left|f\right|·\left|p\right|}}$$

(4)

For feature proxies that are not related to the samples, we enforce a larger distance between the features and irrelevant feature proxies by calculating the following Equation (5):

$${cos}_{-}(f,p)=max\left({\displaystyle \frac{|f·p|}{\left|f\right|·\left|p\right|}},0\right)$$

(5)

Hence, the image feature proxy loss ${\mathcal{L}}_f^I$ is defined in Equation (6):

$$\begin{array}{cc}\hfill {\mathcal{L}}_f^I& ={\displaystyle \frac{{\sum }_{i=1}^{N_l}{\sum }_{j=1}^{C}I\left(l_{ij}=1\right){cos}_{+}\left(f_i^x,p_j^f\right)}{{\sum }_{i=1}^{N_l}{\sum }_{j=1}^{C}I\left(l_{ij}=1\right)}}\hfill \\ \hfill & +{\displaystyle \frac{{\sum }_{i=1}^{N_l}{\sum }_{j=1}^{C}I\left(l_{ij}=0\right){cos}_{-}\left(f_i^x,p_j^f\right)}{{\sum }_{i=1}^{N_l}{\sum }_{j=1}^{C}I\left(l_{ij}=0\right)}}\hfill \end{array}$$

(6)

where I is an indicator function. The denominators represent the number of relevant data–proxy pairs and irrelevant data–proxy pairs, respectively, aiming for normalization. Similarly, the text feature proxy loss ${\mathcal{L}}_f^T$ is defined in the following Equation (7):

$$\begin{array}{cc}\hfill {\mathcal{L}}_f^T& ={\displaystyle \frac{{\sum }_{i=1}^{N_l}{\sum }_{j=1}^{C}I\left(l_{ij}=1\right){cos}_{+}\left(f_i^y,p_j^f\right)}{{\sum }_{i=1}^{N_l}{\sum }_{j=1}^{C}I\left(l_{ij}=1\right)}}\hfill \\ \hfill & +{\displaystyle \frac{{\sum }_{i=1}^{N_l}{\sum }_{j=1}^{C}I\left(l_{ij}=0\right){cos}_{-}\left(f_i^y,p_j^f\right)}{{\sum }_{i=1}^{N_l}{\sum }_{j=1}^{C}I\left(l_{ij}=0\right)}}\hfill \end{array}$$

(7)

The feature proxy loss ${\mathcal{L}}_f$ is defined in Equation (8):

$${\mathcal{L}}_f={\mathcal{L}}_f^I+{\mathcal{L}}_f^T$$

(8)

3.3.3. Adaptive Dual-Label Loss

Through forward propagation, the features of labeled and unlabeled data ${\mathit{F}}_{\ast }^l$ and ${\mathit{F}}_{\ast }^u$ are transformed into binary-like codes ${\mathit{H}}_{\ast }^l$ and ${\mathit{H}}_{\ast }^u$, where $\ast \in \{x,y\}$; the feature proxies ${\mathit{P}}^f={\left\{{\mathit{p}}_i^f\right\}}_{i=1}^C$ generate hash proxies ${\mathit{P}}^h={\left\{{\mathit{p}}_i^h\right\}}_{i=1}^C$. In the hash space, we consider both labeled and unlabeled data separately. Specifically, for labeled data, which have true label information, we not only learn the relationship between the data and the hash proxies but also learn the relationship between the data points themselves through their true labels. We obtain the similarity relationship between labeled data using the following Equation (9):

$${\mathit{S}}^l={\displaystyle \frac{{\mathit{L}}^l}{{∥{\mathit{L}}^l∥}_2}}·{\left({\displaystyle \frac{{\mathit{L}}^l}{{∥{\mathit{L}}^l∥}_2}}\right)}^T$$

(9)

The range of ${\mathit{S}}_{ij}^l$ is [0, 1]. If ${\mathit{S}}_{ij}^l$ > 0, then $x_i$ (or $y_i$) and $x_j$ (or $y_j$) are called an relevant pair. If ${\mathit{S}}_{ij}^l$ = 0, then they are considered as irrelevant pairs. To pull relevant pairs closer, we use ${cos}_{\mathrm{p}}\left(h_i,h_j\right)$, and the distance between irrelevant data pairs is pushed away by ${cos}_{\mathrm{n}}\left(h_i,h_j\right)$.

$$\begin{array}{c}\hfill {cos}_{\mathrm{p}}\left(h_i,h_j\right)=max\left({\mathit{S}}_{ij}^l-{\displaystyle \frac{\left|h_i·h_j\right|}{\left|h_i\right|·\left|h_j\right|}},0\right),{cos}_{\mathrm{n}}\left(h_i,h_j\right)=max\left({\displaystyle \frac{\left|h_i·h_j\right|}{\left|h_i\right|·\left|h_j\right|}},0\right)\end{array}$$

(10)

Therefore, the data-related loss for labeled data in hash space ${\mathcal{L}}_h^{l_d}$ is computed using the following Equation (11):

$$\begin{array}{c}\hfill {\mathcal{L}}_h^{l_d}={\displaystyle \frac{{\sum }_{\ast \in \{x,y\}}{\sum }_{i,j=1}^{N_l}I\left({\mathit{S}}_{ij}^l>0\right){cos}_{\mathrm{p}}\left(h_i^{\ast },h_j^{\ast }\right)}{{\sum }_{i,j=1}^{N_l}I\left({\mathit{S}}_{ij}^l>0\right)}}+{\displaystyle \frac{{\sum }_{\ast \in \{x,y\}}{\sum }_{i,j=1}^{N_l}I\left({\mathit{S}}_{ij}^l=0\right){cos}_{\mathrm{n}}\left(h_i^{\ast },h_j^{\ast }\right)}{{\sum }_{i,j=1}^{N_l}I\left({\mathit{S}}_{ij}^l=0\right)}}\end{array}$$

(11)

For the relationship between data and hash proxies in the hash space, we can impose a constraint using the loss ${\mathcal{L}}_h^{l_p}$. Similar to the loss in the feature space ${\mathcal{L}}_f$, ${\mathcal{L}}_h^{l_p}$ is defined as follows:

$$\begin{array}{c}\hfill {\mathcal{L}}_h^{l_p}={\displaystyle \frac{{\sum }_{\ast \in \{x,y\}}{\sum }_{i=1}^{N_l}{\sum }_{j=1}^{C}I\left(l_{ij}=1\right){cos}_{+}\left(h_i^{\ast },p_j^h\right)}{{\sum }_{i=1}^{N_l}{\sum }_{j=1}^{C}I\left(l_{ij}=1\right)}}+{\displaystyle \frac{{\sum }_{\ast \in \{x,y\}}{\sum }_{i=1}^{N_l}{\sum }_{j=1}^{C}I\left(l_{ij}=0\right){cos}_{-}\left(h_i^{\ast },p_j^h\right)}{{\sum }_{i=1}^{N_l}{\sum }_{j=1}^{C}I\left(l_{ij}=0\right)}}\end{array}$$

(12)

For unlabeled data, since their pseudo-labels directly correspond to the structural relationships between the data and the categories, learning the relationships between unlabeled data through pseudo-label information would instead introduce errors. Therefore, we directly map the structural relationship between the unlabeled data features and the feature proxies in the feature space, as contained in the pseudo-label information, to the relationship between the binary-like codes of the unlabeled data and the hash proxies in the hash space. Specifically, we bring the binary-like codes of unlabeled data closer to the relevant hash proxies by calculating ${cos}_r\left(h_i,p_j^h\right)$ and push them apart from irrelevant ones by calculating ${cos}_e\left(h_i,p_j^h\right)$.

$$\begin{array}{c}\hfill {cos}_r\left(h_i,p_j^h\right)=max\left({\mathit{L}}_{ij}^u-{\displaystyle \frac{\left|h_i·p_j^h\right|}{\left|h_i\right|·\left|p_j^h\right|}},0\right),{cos}_e\left(h_i,p_j^h\right)=max\left({\displaystyle \frac{\left|h_i·p_j^h\right|}{\left|h_i\right|·\left|p_j^h\right|}},0\right)\end{array}$$

(13)

Therefore, the loss for unlabeled data in the hash space, ${\mathcal{L}}_h^u$, can be computed using the following Equation (14):

$$\begin{array}{c}\hfill {\mathcal{L}}_h^u={\displaystyle \frac{{\sum }_{\ast \in \{x,y\}}{\sum }_{i=1}^{N_u}{\sum }_{j=1}^{C}I\left({\mathit{L}}_{ij}^u>0\right){cos}_r\left(h_i^{\ast },p_j^h\right)}{{\sum }_{i=1}^{N_u}{\sum }_{j=1}^{C}I\left({\mathit{L}}_{ij}^u>0\right)}}+{\displaystyle \frac{{\sum }_{\ast \in \{x,y\}}{\sum }_{i=1}^{N_u}{\sum }_{j=1}^{C}I\left({\mathit{L}}_{ij}^u=0\right){cos}_e\left(h_i^{\ast },p_j^h\right)}{{\sum }_{i=1}^{N_u}{\sum }_{j=1}^{C}I\left({\mathit{L}}_{ij}^u=0\right)}}\end{array}$$

(14)

The proposed Adaptive Dual-Label Loss ${\mathcal{L}}_h$ in the hash space is defined as in Equation (15):

$${\mathcal{L}}_h=\alpha {\mathcal{L}}_h^{l_d}+\beta {\mathcal{L}}_h^{l_p}+min\left(1,{\displaystyle \frac{\mathbf{epoch}}{\mathbf{epoch}\mathbf{num}}}\right)·{\mathcal{L}}_h^u$$

(15)

where $\alpha$ and $\beta$ are the hyper-parameters; we dynamically adjust the weight of the loss for unlabeled data based on the epoch, ensuring that the model maintains the importance of labeled data while learning more stably from unlabeled data. In the early stages of training, the model relies more on the true labels to avoid negative impacts from unlabeled data, while later, as the quality of the pseudo-labels improves, the weight gradually increases.

3.4. Optimization

The pseudo-code for our proposed PSSCH method is provided in Algorithm 1. For query samples, we utilize the well-trained PSSCH model to map them into the hash space and then generate their hash codes using the sign function.

$$\begin{array}{c}\hfill sign\left(x\right)=\left\{\begin{array}{c}+1,x>0\hfill \\ -1,x<0\hfill \end{array}\right.\end{array}$$

(16)

For image data $x_i$ or text data $y_i$, we generate the hash codes by Equation (17):

$$\begin{array}{c}\hfill \begin{array}{c}{b}_i^x=sign\left({\mathcal{H}}^x\left(x_i\right)\right)\hfill \\ \\ b_i^y=sign\left({\mathcal{H}}^y\left(y_i\right)\right)\hfill \end{array}\end{array}$$

(17)

Algorithm 1 The Pseudo-Code of the PSSCH Method

Input:  Training dataset $\mathcal{D}$; The number of bits in the hash code K; The one-hot encodings of the categories $\mathit{P}$; threshold and Hyper-parameters $\alpha$, $\beta$. Output:  Image and Text Network Parameters: ${\mathsf{\Theta }}_x$, ${\mathsf{\Theta }}_y$; Category Proxy Network Parameters ${\mathsf{\Theta }}_c$.    1: Initialize ${\mathsf{\Theta }}_x$, ${\mathsf{\Theta }}_y$ and ${\mathsf{\Theta }}_c$; iteration number: $\mathbf{epoch}\mathbf{num}$, batch size: 128, learning rate: 0.001.    2: Construct a similarity matrix ${\mathit{S}}^l$ by Equation (9).    3: while  $iter<\mathbf{epoch}\mathbf{num}$  do    4:    Obtain the features ${\mathit{F}}^l$ and binary-like codes ${\mathit{H}}^l$ for labeled samples, as well as the features ${\mathit{F}}^u$ and binary-like codes ${\mathit{H}}^u$ for unlabeled samples, through forward propagation, along with the feature proxies ${\mathit{P}}^f={\left\{{\mathit{p}}_i^f\right\}}_{i=1}^C$ and hash proxies ${\mathit{P}}^h={\left\{{\mathit{p}}_i^h\right\}}_{i=1}^C$.    5:    Generate pseudo-labels for unlabeled data by Equation (3).    6:    Compute feature proxy loss ${\mathcal{L}}_f$ by Equation (8).    7:    Compute Adaptive Dual-Label Loss ${\mathcal{L}}_h$ by Equation (15).    8:    Update Category Proxy Network Parameters ${\mathsf{\Theta }}_c$ by backpropagation.    9:    Update Image and Text Network Parameters ${\mathsf{\Theta }}_x$ and ${\mathsf{\Theta }}_y$ by backpropagation.  10: end while  11: return

4. Experiments

This section evaluates the effectiveness of the proposed PSSCH by performing image–text cross-modal retrieval and comparing the results with those of several excellent methods.

4.1. Experimental Settings

Datasets. We evaluated our method on three widely used benchmark datasets: MIRFLICKR-25K [44], NUS-WIDE [45], and MS COCO [46]. A brief description of each is provided below:

• MIRFLICKR-25K. This small-scale cross-modal dataset consists of 24,581 image–text pairs, spanning 24 categories, with each sample belonging to at least one category.
• NUS-WIDE. Comprising 269,648 image–text pairs, this dataset includes 81 categories. We filtered out categories with fewer samples and selected 21 common categories, resulting in 195,834 image–text pairs.
• MS COCO. A large-scale dataset commonly used in computer vision, containing 82,785 training images and 40,504 validation images. Each image is associated with textual descriptions and labels across 80 categories. For our experiments, we combined the training and validation sets, with each sample belonging to at least one of these categories.

Implementation Details. To facilitate comparison, we follow the settings of the baseline method. The threshold was set to 0.5, $\mathbf{epoch}\mathbf{num}$ was set to 100, we applied the same procedure across the three public datasets, and we randomly selected 10,000 samples as the training set, which were then divided into labeled and unlabeled data according to a certain ratio. In addition, we randomly selected 5000 samples as the query set, with the remaining samples being used as the database. In this process, images are resized to 224 × 224, and text is represented using BPE encoding.

Experimental Environment. We implemented our PSSCH method using PyTorch==1.12.1, with an NVIDIA RTX 3090 GPU. The batch size was set to 128, with 80 labeled samples and 48 unlabeled samples. The hyper-parameters $\alpha$ and $\beta$ were set to 0.5 and 1, respectively. We used two Transformer encoders as the backbone for the PSSCH method: ViT [41] for the image and GPT-2 [47] for the text. The backbone network parameters were initialized using pre-trained CLIP features (ViT-B/32) [48]. The parameters of ImgNet and TxtNet were updated using the Adam optimizer, with a learning rate of 0.00001 for the backbone and 0.001 for the hashing layer. The CPNet network parameters were updated using the SGD optimizer with a learning rate of 0.001.

Baseline Methods. We selected 10 excellent deep cross-modal hashing methods for comparison, including four supervised cross-modal hashing methods, i.e., LEMON [49], EDMH [50], HCCH [51], and HMAH [52], four semi-supervised cross-modal hashing methods, i.e., SSCH [22], MGCH [19], TS3H [14], and GCSCH [20], and two unsupervised cross-modal hashing methods, i.e., DGCPN [53] and UCCH [54]. For supervised methods, we train using only the labeled data. Due to some methods not being open source, we directly cite the results from the published papers. Here is a brief introduction to each baseline:

• LEMON. embeds label information into the hash learning process in order to fully utilize the semantic information of labels to guide the learning of hash functions.
• EDMH. proposes a discrete optimization algorithm that seamlessly integrates three useful discrete constraints into a joint hashing learning model.
• HCCH. mitigates the loss of important discriminative information, a hierarchical hashing scheme from coarse to fine is proposed, which refines useful discriminative information step by step using a two-layer hashing function.
• HMAH. creates a hierarchical message aggregation network within a teacher–student framework, enhancing alignment of heterogeneous modalities and modeling detailed cross-modal correlations.
• SSCH. obtains enhanced semantic information through a pseudo-labeling process that does not require alignment and learns the hash representations of various data via a label enhancement strategy.
• MGCH. employs a multi-view graph to connect the data, utilizing anchor points as a unified semantic hub to achieve semi-supervised cross-modal hashing.
• TS3H. utilizes supervised information; classifiers for different modalities are learned to predict the labels of unlabeled data, and then the hash codes are learned by combining both the new and old labels.
• GCSCH. designs a fusion network to integrate the two modalities and uses a graph convolutional network to capture semantic information from both real-labeled and pseudo-labeled multi-modal data.
• DGCPN. utilizes graph models to explore graph-neighbor consistency, which helps to address the inaccurate similarity calculation in unsupervised cross-modal hashing.
• UCCH. proposes a novel momentum optimizer for learnable hashing in contrastive learning and designs a cross-modal ranking learning loss.

Evaluation Protocols. We evaluated our method by comparing it with baseline approaches on two cross-modal retrieval tasks: image-to-text retrieval (I→T) and text-to-image retrieval (T→I). For performance assessment, we employed standard evaluation metrics, including mean Average Precision (mAP) and precision–recall (PR) curves. mAP represents the average of Average Precision (AP) across all queries and is the most commonly used evaluation metric in cross-modal hashing. The formula for calculating mAP is as follows in Equation (18):

$$mAP={\displaystyle \frac{1}{N_q}}\sum _{i=1}^{N_q}AP\left(i\right)$$

(18)

where $AP\left(i\right)$ represents the average precision for a query sample i and $N_q$ denotes the total number of query samples. PR curve illustrates the relationship between recall and precision. The results from these metrics show that the PSSCH method performs exceptionally well in cross-modal similarity search.

4.2. Performance Comparison

We compared the PSSCH method with baseline methods on three public datasets. The mAP results are shown in

Table 2. Comparison of mAP with baseline methods on MIRFLICKR-25K, NUS-WIDE, and MS COCO based on 30% labeled samples. The best results are in bold font.

Task	Methods	MIRFLICKR-25K			NUS-WIDE			MS COCO
		16 Bits	32 Bits	64 Bits	16 Bits	32 Bits	64 Bits	16 Bits	32 Bits	64 Bits
I→T	DGCPN (AAAI 21)	0.703	0.713	0.720	0.566	0.589	0.601	0.575	0.613	0.630
	UCCH (TPAMI 23)	0.734	0.741	0.739	0.590	0.610	0.618	0.562	0.569	0.590
	LEMON (MM 20)	0.651	0.670	0.682	0.460	0.491	0.507	0.492	0.438	0.527
	EDMH (TKDE 22)	0.651	0.657	0.646	0.460	0.477	0.461	0.502	0.497	0.427
	HMAH (TMM 22)	0.755	0.743	0.753	0.606	0.636	0.639	0.558	0.569	0.594
	HCCH (TMM 23)	0.719	0.730	0.736	0.625	0.638	0.649	0.560	0.606	0.634
	MGCH (IS 22)	0.689	0.705	0.729	0.525	0.514	0.595	0.615	0.562	0.607
	SSCH (TCSVT 23)	0.622	0.670	0.685	0.479	0.524	0.539	0.435	0.441	0.479
	TS3H (TNNLS 23)	0.717	0.741	0.742	0.613	0.642	0.671	0.618	0.624	0.690
	GCSCH (MM 24)	0.772	0.776	0.785	0.658	0.677	0.673	0.619	0.675	0.701
	PSSCH (Ours)	0.794	0.797	0.818	0.666	0.678	0.684	0.644	0.689	0.723
T→I	DGCPN (AAAI 21)	0.692	0.701	0.710	0.578	0.596	0.601	0.572	0.609	0.625
	UCCH (TPAMI 23)	0.722	0.726	0.725	0.600	0.616	0.626	0.553	0.560	0.586
	LEMON (MM 20)	0.666	0.695	0.708	0.472	0.508	0.517	0.487	0.475	0.535
	EDMH (TKDE 22)	0.668	0.677	0.667	0.475	0.487	0.477	0.501	0.494	0.427
	HMAH (TMM 22)	0.721	0.703	0.705	0.546	0.578	0.559	0.549	0.558	0.578
	HCCH (TMM 23)	0.721	0.740	0.742	0.631	0.632	0.649	0.556	0.588	0.647
	MGCH (IS 22)	0.675	0.695	0.719	0.541	0.515	0.607	0.601	0.553	0.586
	SSCH (TCSVT 23)	0.623	0.664	0.688	0.482	0.526	0.557	0.440	0.443	0.474
	TS3H (TNNLS 23)	0.727	0.753	0.748	0.622	0.653	0.674	0.614	0.618	0.687
	GCSCH (MM 24)	0.780	0.791	0.791	0.661	0.673	0.684	0.620	0.661	0.688
	PSSCH (Ours)	0.774	0.787	0.803	0.671	0.683	0.692	0.657	0.702	0.728

. PSSCH generally outperforms other baseline methods, achieving satisfactory performance. On the MIRFLICKR-25K dataset, the PSSCH method achieves an average mAP that is 2.5% higher than the best baseline method in the image-to-text retrieval task. Additionally, PSSCH achieves the best performance in the 64-bit text-to-image retrieval task. However, in the text-to-image retrieval task, PSSCH performed slightly worse than GCSCH at 16 bits and 32 bits. We speculate that the size of the hash space may affect the performance of our method. Moreover, since the MIRFLICKR-25K dataset is not very large, the use of data augmentation in GCSCH on the training image data may also be a reason why it performs better in the text-to-image retrieval task. On the NUS-WIDE dataset, PSSCH also shows an average mAP improvement of about 1% over the best baseline method in both retrieval tasks. On the MS COCO dataset, PSSCH performs most significantly, with an average mAP that is 3% higher than the best baseline method. We speculate that this may be because the MS COCO dataset contains 80 categories, which is much higher than the other two datasets. In scenarios with a larger number of categories, learning the relationships between data and categories may yield better results. To demonstrate that the improvements regarding our method are significant and not due to randomness, we conducted experiments on three datasets at 64 bits to obtain 10 mAP results. We then performed a paired t-test with the best baseline method, and the test results are presented in

Table 3. Paired t-test results comparing mAP values of PSSCH with GCSCH on MIRFLICKR-25K, NUS-WIDE, and MS COCO at 64 bits.

Task	Dataset	t-Statistic	p-Value	Conclusion
I→T	MIRFLICKR-25K	19.42	p < 0.001	PSSCH is significantly better than GCSCH
	NUS-WIDE	5.74	p < 0.001
	MS COCO	15.25	p < 0.001
T→I	MIRFLICKR-25K	11.00	p < 0.001	PSSCH is significantly better than GCSCH
	NUS-WIDE	9.08	p < 0.001
	MS COCO	29.03	p < 0.001

, which proves that the performance enhancement regarding the PSSCH method is significant.

Figure 2 shows the precision–recall curve results for the MIRFLICKR-25K, NUS-WIDE, and MS COCO datasets with 64-bit codes. Precision refers to the proportion of relevant items in the retrieval results, while recall indicates the proportion of all relevant items that are correctly retrieved by the model. The PR curves for the PSSCH method outperform other baseline methods in both retrieval tasks across the three datasets. Figure 3 shows the mAP of the semi-supervised cross-modal hashing methods under different percentages of labeled samples on the MIRFLICKR-25K, NUS-WIDE, and MS COCO datasets. As can be seen, with the reduction in the number of labeled samples, the mAP of the PSSCH method decreases sharply, which was expected. We must acknowledge the importance of true labels. Our feature proxies require real labeled samples for constraints in order to ensure the accurate generation of pseudo-labels for unlabeled samples. Moreover, the mAP results of the PSSCH method with 50% labeled samples are close to those obtained with 90% labeled samples, further confirming the practicality of the PSSCH method.

4.3. Ablation Studies

To validate the effectiveness of the PSSCH method, we implemented three variations to calculate the mAP values for the tasks of I→T and T→I. Specifically: (1) PSSCH-1: Using discrete [0, 1] pseudo-labels, and applying the labeled data loss ${\mathcal{L}}_h^{l_p}$ to the unlabeled data in the hash space. (2) PSSCH-2: Not dynamically adjusting the weight of the loss for unlabeled data ${\mathcal{L}}_h^u$. (3) PSSCH-3: Removing the loss function ${\mathcal{L}}_f$ in the feature space.

The results of the ablation experiment are shown in

Table 4. mAP results of PSSCH and its variants on MIRFLICKR-25K, NUS-WIDE, and MS COCO. The best results are in bold font.

Task	Methods	MIRFLICKR-25K			NUS-WIDE			MS COCO
		16 Bits	32 Bits	64 Bits	16 Bits	32 Bits	64 Bits	16 Bits	32 Bits	64 Bits
Img2Txt	PSSCH-1	0.783	0.789	0.801	0.661	0.669	0.674	0.641	0.672	0.711
	PSSCH-2	0.711	0.723	0.732	0.604	0.615	0.624	0.610	0.623	0.644
	PSSCH-3	0.764	0.771	0.784	0.643	0.652	0.659	0.638	0.664	0.697
	PSSCH (Ours)	0.794	0.797	0.818	0.666	0.678	0.684	0.644	0.689	0.723
Txt2Img	PSSCH-1	0.771	0.776	0.786	0.665	0.673	0.682	0.645	0.684	0.709
	PSSCH-2	0.703	0.709	0.713	0.594	0.603	0.609	0.604	0.618	0.625
	PSSCH-3	0.752	0.758	0.767	0.641	0.647	0.658	0.630	0.662	0.689
	PSSCH (Ours)	0.774	0.787	0.803	0.671	0.683	0.692	0.657	0.702	0.728

. Since PSSCH-1 uses discrete pseudo-labels [0, 1], which require quantization of the results after pseudo-label generation, information is inevitably lost during this process. In contrast, continuous pseudo-labels can directly represent the structural relationship between data and similar categories. By comparing the results of PSSCH-1 and PSSCH on three benchmark datasets, it can be seen that PSSCH effectively utilizes the structural relationships within continuous pseudo-labels to achieve better outcomes. PSSCH-2 eliminates the dynamic weight adjustment function: $min\left(1,{\displaystyle \frac{\mathbf{epoch}}{\mathbf{epoch}\mathbf{num}}}\right)$.

In the early stages of training, when feature proxies have not been sufficiently trained, generating pseudo-labels based on the similarity between pre-trained features of unlabeled data and feature proxies introduces noise. Assigning a large weight to unlabeled data loss in the early stages can negatively impact model performance. Comparing the results of PSSCH-2 and PSSCH indicates that not dynamically adjusting the loss weight for unlabeled data leads to the initial low-quality pseudo-labels severely affecting the model’s training, resulting in suboptimal outcomes. Comparing the results of PSSCH-3 and PSSCH, we see that PSSCH-3 removes the loss function ${\mathcal{L}}_f$ in the feature space and only updates the feature proxy through hash loss during backpropagation without constraining the feature proxy at the feature level. This lack of guarantee in pseudo-label generation leads to suboptimal results. By comparing these three variants, the effectiveness of each component in the PSSCH method can be verified.

4.4. Sensitivity to Hyper-Parameters

$\alpha$ and $\beta$ represent the relative importance of two types of loss in the total loss. Specifically, $\alpha$ denotes the importance of the loss ${\mathcal{L}}_h^{l_d}$, which constrains the relationship between data and data. Meanwhile, $\beta$ indicates the importance of the loss ${\mathcal{L}}_h^{l_p}$, which constrains the relationships between data and categories. We investigate the sensitivity of the parameters $\alpha$ and $\beta$. We set their ranges to 0.001, 0.01, 0.05, 0.1, 0.5, 0.8, 1. As shown in Figure 4, when either $\alpha$ or $\beta$ is close to 0, the retrieval performance decreases. When $\alpha$ and $\beta$ are set to 0.5 and 1, respectively, the model performs optimally. The model is still quite sensitive to hyper-parameters $\alpha$ and $\beta$. This also validates that, during model training, it is essential to consider both the relationship between data and data and the relationships between data and categories.

4.5. Training and Encoding Time

To investigate the efficiency of the PSSCH method, we compared the training time and encoding time with baseline methods on the MIRFlickr-25K dataset with 64 bits. For the training time, we analyzed the time complexity of the PSSCH algorithm. The time complexity of the feature proxy loss ${\mathcal{L}}_f$ is $O\left(N_{l}C\right)$, where C represents the number of categories, while the time complexity of the Adaptive Dual-Label Loss is $O\left(NC+N_l^2\right)$, where $N_l$ denotes the number of labeled samples. Thus, the overall time complexity of PSSCH is $O\left(NC+N_l^2+N_{l}C\right)$, where N represents the total number of training samples. As shown in Figure 5a, our training time is slightly higher than other methods, but the training process is conducted offline, so the training time does not affect retrieval performance. In the field of cross-modal hashing retrieval, more attention is often paid to inference time, which mainly consists of encoding time and Hamming distance computation time. The encoding time, as shown in Figure 5b, is in the millisecond range for all methods, demonstrating that the PSSCH method has a reasonable encoding time.

4.6. Visualization

We validate the proposed PSSCH method by visualizing the learned hash codes using T-SNE [55]. Specifically, we select samples from seven different single-label categories in the NUS-WIDE dataset with a hash code length of 32 bits and compare them with the TS3H and GCSCH methods. The results are shown in Figure 6. Different colored dots represent different single-label category data. To further assess the clustering effectiveness of the data after dimensionality reduction, we computed the Davies–Bouldin Index (DBI) for these three methods. The DBI is a commonly used metric for evaluating clustering performance as it calculates the separation between clusters as well as the compactness within clusters. The smaller the DBI value, the better the clustering effect. The DBI scores for T3SH, GCSCH, and PSSCH are 1.9742, 1.7121, and 1.5935, respectively. This also demonstrates that the PSSCH method achieved superior performance.

5. Conclusions

In this paper, we propose the PSSCH framework for semi-supervised cross-modal hashing retrieval, enabling the model to fully utilize the structural information in continuous pseudo-labels while simultaneously considering the relationships between data and the relationships between data and categories. However, the performance of the PSSCH method is not fully demonstrated on small-scale datasets or when the hash space is reduced. In future work, we plan to address this limitation by performing data augmentation on training data for small-scale datasets or applying balance constraints to hash codes. Additionally, we will explore extending the PSSCH method to other modalities, such as speech and video, to validate its versatility and effectiveness.