Multi-Grained Similarity Preserving and Updating for Unsupervised Cross-Modal Hashing

Abstract

Unsupervised cross-modal hashing is a topic of considerable interest due to its advantages in terms of low storage costs and fast retrieval speed. Despite the impressive achievements of existing solutions, two challenges remain unaddressed: (1) Semantic similarity obtained without supervision is not accurate enough, and (2) the preservation of similarity structures lacks effectiveness due to the neglect of both global and local similarity. This paper introduces a new method, Multi-Grained Similarity Preserving and Updating (MGSPU), to tackle these challenges. To overcome the first challenge, MGSPU employs a newly designed strategy to update the semantic similarity matrix, effectively generating a high-confidence similarity matrix by eliminating noise in the original cross-modal features. For the second challenge, a novel multi-grained similarity preserving method is proposed, aiming to enhance cross-modal hash code learning by learning consistency in multi-grained similarity structures. Comprehensive experiments on two widely used datasets with nine state-of-the-art competitors validate the superior performance of our method in cross-modal hashing.

1. Introduction

As an efficient information retrieval paradigm for big multimedia data [1,2,3], cross-modal retrieval [4,5,6,7,8,9] utilizes one modality as a query to search another modal data. Among the existing techniques [10,11,12,13,14,15,16,17,18], cross-modal hashing [16,17,18,19,20] is popular for its fast retrieval speed and low storage cost. The core of cross-modal hashing is to map high-dimensional data to a low-dimensional common Hamming space, in which multi-modal instances (text, image, audio and video) with similar semantics are closer, and dissimilar instances are far apart. Notwithstanding the strong practical significance, this task is extremely challenging due to the heterogeneity gap between different modalities.

A number of studies are striving to promote the progress of cross-modal hashing. In general, they can be simply divided into two groups: supervised learning methods [21,22,23,24,25] and unsupervised learning methods [26,27,28,29,30]. The supervised learning methods use human-annotated label to guide model learning, which preserves semantic discrimination well to obtain great performance. However, directly obtaining the category information is really difficult in real-world scenarios, while the cost of manual labeling is so huge as to limit the wide application. In contrast, without relying on category annotations, unsupervised learning methods are great at capturing the underlying correlation patterns between different modalities to bridge the heterogeneity gap. Thus, they have gained considerable attention currently.

Motivation. Although existing researches have shown remarkable results on large-scale datasets, unsupervised cross-modal hashing is yet a Herculean task that await further exploration due to the following challenges. The first challenge is how to accurately measure inter-modal and intra-modal similarity without category information. In the early days, several unsupervised learning methods [31,32] used discrete models to calculate binary similarity, namely, defining the similarity between two instances with small distances as “1”, otherwise “−1”. Unfortunately, the naive similarity measurement is far from accurate. To break this limitation, researchers [12,15,33,34] began to use original feature distance to measure the continuous similarity. However, such similarity is still not accurate enough to represent the real relationship between instances due to the noise existing in their original features. The second challenge is how to preserve similarity structure consistency from both local and global view during hash learning process. Undisputedly, using multi-grained similarity relationships across modalities is crucial to narrow cross-modal heterogeneity. However, current methods [35,36,37] either overlook multi-grained similarity learning across various modalities or struggle to capture a consistent distribution of relations within the content details. Additionally, some state-of-the-art methods [38,39,40,41] do not account for the semantic correlation between feature vectors from one modality and the corresponding hash codes from another modality. To conquer these challenges, this paper has made the following two efforts: (1) designing a novel and effective similarity updating strategy to rise the accuracy of underlying similarity measurement, and (2) combining multi-grained similarity information from local and global view to preserve similarity consistency.

Our Method. We propose a novel cross-modal hashing framework, named Multi-Grained Similarity Preserving and Updating (MGSPU). As shown in Figure 1, this framework consists four module, i.e., feature extraction, semantic similarity matrix updating, hash code learning and hash function learning. To overcome the first challenge, a novel semantic similarity matrix update strategy is developed, which removes noises from the similarity matrix so as to obtain high-confidence supervisory signal. To face the second challenge, we capture the multi-grained similarity structure information from global and local view to preserve similarity consistency better. For global view, we use graph convolutional network (GCN for short) to aggregate the similarity structure information from the neighbors of each instance to enrich the coarse-grained similarity information. For local view, cross-modal attention mechanisms are used to perform cross-modal interaction to enhance similarity learning between instance pairs. Furthermore, we use a similarity consistency reconstruction method to ensure the similarity consistency of hash codes.

Contributions. In summary, the contributions of this paper can be summarized as follows:

• We propose an effective unsupervised learning framework, called Multi-Grained Similarity Preserving and Updating, which learns high-quality hash codes by comprehensively improving cross-modal similarity learning.
• We propose a novel semantic similarity matrix updating strategy to effectively remove noises in the original similarity matrix, which produces high-confidence supervisory signal for cross-modal hashing learning.
• We propose a novel multi-grained similarity preserving method to enhance similarity consistency preserving of the cross-modal hash codes.
• We conducted extensive experiments on the widely used datasets MIRFLICKR-25K and NUS-WIDE to validate the superiority of the proposed approach and evaluate the effectiveness of each component.

Roadmap. The paper is structured as follows: Section 2 provides a summary of related works. Section 3 outlines the problem definition and details the MGSPU approach. Section 4 presents the experimental results and analysis. Lastly, Section 5 concludes the paper.

2. Related Work

According to literatures, cross-modal hashing techniques are generally categorized into two groups: supervised and unsupervised cross-modal hashing. This section reviews the prevailing solutions related to this paper, which are summarized in

Table 1. Related works summary.

Type	Reference	Source	Authors	Description
Supervised	SCM [42]	AAAI (2014)	Zhang, D., et al.	Integrate semantic tags into the data modeling process.
	SePH [43]	CVPR (2015)	Lin, Z., et al.	The semantic similarity given by the training data is converted into a probability distribution, and then the hash code is learned by minimizing the KL divergence.
	DCMH [17]	CVPR (2017)	Jiang, Q.Y., et al.	Feature learning and hash learning are integrated for the first time.
	CMHH [44]	ECCV (2018)	Cao, Y., et al.	Hamming distance replaces the inner product and uses pairwise loss based on exponential distribution to solve the problem of uneven positive and negative samples.
	SSAH [45]	AAAI (2020)	Jin, S., et al.	Derive semantic ranking information using data features and tags and integrate it into cross-modal hashes.
	RDCMH [46]	AAAI (2019)	Liu, X., et al.	A multi-level semantic similarity matrix is constructed by considering the bidirectional relation, the relation between the whole instances.
	Bi_NCMH [47]	CVPR (2022)	Sun, C., et al.	The potential semantic relationships of data are captured by constructing a joint semantic matrix, but redundant information is introduced.
Unsupervised	DJSRH [12]	ICCV (2019)	Su, S., et al.	The potential semantic relationships of data are captured by constructing a joint semantic matrix, but redundant information is introduced.
	JDSH [36]	ACM (2020)	Liu, S., et al.	The proposed sampling weighting scheme can generate more efficient hash codes.
	DAEH [27]	TCSVT (2022)	Shi, Y., et al.	Design information mixed similarity estimation integrates distance distribution and similarity rate information.
	DGCPN [48]	AAAI (2021)	Yu, J., et al.	Using a graph model to integrate neighborhood information, the overall loss is maintained by designing three types of data similarity.
	AGCH [26]	TMM (2021)	Zhang, P., et al.	The semantic structure of data is mined by GCN, and the structural information is explored from multiple perspectives by using a variety of similarity measures.
	DRNPH [49]	Mathematics (2022)	Yang, X., et al.	Cross-modal triples are used to maintain data similarity, and the distance between similar samples is required to be smaller than that between dissimilar pairs.

to ease reading.

2.1. Supervised Cross-Modal Hashing

To obtain common binary representations, supervised cross-modal hashing methods use category information to maintain semantic discrimination. For example, semantic correlation maximization (SCM [42]) learns hash functions by constructing and maintaining semantic similarity matrices. Semantics-preserving hashing (SePH [43]) converts the semantic matrix into a probability distribution and minimizes the Kullback-Leibler divergence so that the learned hash codes are approximately distributed in Hamming space. Thanks to the prosperity of deep learning, researchers began to realize cross-modal retrieval by deep models that explore more discriminative features. Deep cross-modal hashing (DCMH [17]) is a pioneering achievement in this field, which perfectly combines hash function learning with deep feature extraction, and cleverly exploits the labelling information to construct similarity matrices, thus preserving the subtle similarity relationships in cross-modal data. Meanwhile, cross-modal hamming hashing (CMHH [44]) introduces a pairwise focusing loss based on exponential distribution. It penalises instances with similar semantic content but the Hamming distance exceeds a predefined threshold. Semi-supervised adversarial deep hashing (SSAH [45]) is a groundbreaking innovation that shifts the focus to self-supervised methods and integrates adversarial learning into cross-modal hashing, thus making significant progress in the field. Ranking-based deep cross-modal hashing (RDCMH [46]) uses maximum marginal loss to learn uniform Hamming representations. In addition, deep normalized cross-modal hashing with bi-direction relation reasoning (Bi-NCMH [47]) achieved excellent retrieval performance by constructing high-quality similarity matrices to capture similarity relations between instances with multiple labels. Despite the impressive performance of these techniques, their inherent limitations cannot be ignored: they rely heavily on manual annotation to obtain supervision.

2.2. Unsupervised Cross-Modal Hashing

Unsupervised cross-modal hashing methods do not rely on labels so as to apply in real-world scenarios easily. In the existing solutions, constructing a semantic similarity matrix is the key to guide cross-modal relationship learning.Among them, an ingenious method is Deep Joint Semantic Reconstruction Hash (DJSRH [12]), which proposes a joint semantic similarity matrix to simultaneously integrate multi-modal similarity information of cross-modal instances. However, the similarity matrix in DJSRH introduces redundant information from intra-modal fusion items. As an improvement, joint-modal distribution-based similarity hashing (JDSH [36]) involves distribution-based similarity decision and weighting to learn more discriminative hash codes. Deep adaptively-enhanced hashing (DAEH [27]) utilizes distance distributions and similarity ratio information to estimate comparable similarity relationships as complementarity of simple feature distance-based metrics. Although competitive performance is achieved, lack of accurate similarity measurement are these methods since the similarity matrix they rely on are constructed from the original features with noises.

Other unsupervised learning studies make efforts on narrowing the heterogeneity gap by preserving similarity structure consistency. For example, deep graph-neighbor coherence preserving network (DGCPN [48]) explores the relationship information between data and their neighbors to capture neighborhood structure coherence. Aggregation-based graph convolutional hashing (AGCH [26]) employs a GCN to deeply explore the underlying neighborhood structure. Deep relative neighbor relationship preserving hashing (DRNPH [49]) method excavates deep relative neighbor relationships in common Hamming space via binary feature vector based intra- and inter-modal neighbor matrix reconstruction. Impressive progress had been made by these works, however, a common weakness they suffer from is lack of both global and local similarity learning from multi-modal contents.

To address the above shortcomings, we attempt to boost cross-modal hashing learning from two aspects: (1) trying to rise the quality of similarity matrix by reducing noises from original features, and (2) inviting multi-grained similarity learning strategy to capture both global and local similarity relationships. As a results, a novel unsupervised cross-modal hashing technique called multi-grained similarity preserving and updating is developed. The detailed details of this approach will be explored in depth in the next section.

3. Methodology

This section details the proposed MGSPU method across six aspects. Initially, we introduce the notations and problem definition in the Preliminary. Next is an overview of MGSPU, feature extraction, semantic similarity matrix updating, hashing learning (including hash code learning and hashing function learning), and finally a discussion of the optimization algorithm. All abbreviations involved are summarized in

Table 2. Abbreviation summary.

Abbreviation	Meaning
MGSPU	Multi-Grained Similarity Preserving and Updating
CNN	Convolutional Neural Network
BoW	Bag of words
GCNs	Graph Convolutional Networks
ResNet	Residual Networks
LSTM	Long Short-Term Memory Network
MGSP	Multi-Grained Similarity Preserving
GSA	Global Similarity Aggregation
LSI	Local Similarity Interaction
SCR	Similarity Consistency Reconstruction

for easier reading.

3.1. Preliminary

Notations. For clarity and simplicity, calligraphy uppercase letters, such as $\mathcal{O}$, denote sets. Bold uppercase letters, such as $\mathit{W}$, represent matrices. Bold lowercase letters, such as $\mathit{w}$, indicate vectors. In addition, the ij-th element of $\mathit{W}$ is represented as ${\mathit{W}}_{ij}$, the i-th row of $\mathit{W}$ is represented as ${\mathit{W}}_{i\ast }$, the j-th column of $\mathit{W}$ is represented as ${\mathit{W}}_{\ast j}$, ${\mathit{W}}^T$ is the transpose of $\mathit{W}$, $\mathit{I}$ represents identity matrix. $∥.∥$ represents the Frobenius norm of a matrix. $sgn(.)$ represents a sign function, shown as below:

$$sgn\left(x\right)=\left\{\begin{array}{cc}1,\hfill & x\ge 0\hfill \\ -1,\hfill & x<0\hfill \end{array}\right..$$

(1)

For ease of comprehension, commonly utilized mathematical notations have been compiled in

Table 3. Notation summary.

Notation	Definition
${\mathit{v}}_i$	The i-th image instance
${\mathit{t}}_i$	The i-th text instance
${\mathit{F}}^v$	The original image features after feature extraction
${\mathit{F}}^t$	The original text features after feature extraction
$f^v$	Image hash function
$f^t$	Text hash function
${\mathit{S}}^v$	Image similarity matrix
${\mathit{S}}^t$	Text similarity matrix
${\mathit{S}}^f$	Fused similarity matrix
${\mathit{S}}^r$	Refined similarity matrix
${\mathit{S}}^h$	Hashing similarity matrix
$\mathit{\Gamma }$	The indicator matrix
$\mathit{\Lambda }$	The indicator matrix
$\mathit{S}$	Unified similarity matrix
${\mathit{L}}^{*}$	The corresponding latent representations generated by the image or text encoder
${\mathit{X}}_g^{*}$	Image or text global similarity aggregation representations by self-attention mechanism
${\mathit{G}}^{*}$	Final outputs of image or text GCN
${\mathit{Z}}^{*}$	Image or text share the outputs of a fully connected network through a two-layer of weights
${\mathit{X}}_l^v$	Image attention-weighted representations by cross-attention mechanism
${\mathit{X}}_l^t$	Text attention-weighted representations by cross-attention mechanism
${\mathit{P}}^v$	Image features output by the local similarity interaction sub-module
${\mathit{P}}^t$	Text features output by the local similarity interaction sub-module
$\widehat{{\mathit{F}}^v}$	Reconstructed image feature representations
$\widehat{{\mathit{F}}^t}$	Reconstructed text feature representations
${\mathit{H}}^v$	Real-valued hash codes for image instances
${\mathit{H}}^t$	Real-valued hash codes for text instances
${\mathit{B}}^v$	Binary code for image instances
${\mathit{B}}^t$	Binary code for text instances

to assist in reference.

Problem Definition. This paper concentrates on cross-modal retrieval between two prevalent modalities, i.e., image v and text t. Let $\mathcal{O}=\left\{({\mathit{v}}_i,{\mathit{t}}_i\right){\}}_{i=1}^n$ be a cross-modal dataset, ${\mathit{v}}_i\in {\mathbb{R}}^{d_v}$ and ${\mathit{t}}_i\in {\mathbb{R}}^{d_t}$ refer to the image and text in the pairwise instance respectively, $d_v$ and $d_t$ represent the dimension of the corresponding features, n denotes the number of instances. Our study aims to learn two hash functions $f^v=(·;{\mathit{\theta }}_v)$ and $f^t=(·;{\mathit{\theta }}_t)$ to map images and texts to a common Hamming space: ${\mathit{B}}^v=sgn(f^v=({\left\{{\mathit{v}}_i\right\}}_{i=1}^n;{\mathit{\theta }}_v))$, ${\mathit{B}}^t=sgn(f^t=({\left\{{\mathit{t}}_i\right\}}_{i=1}^n;{\mathit{\theta }}_t))$, where ${\mathit{B}}^v,{\mathit{B}}^t\in {\left\{-1,1\right\}}^{n\times C}$, C represents the length of the binary code, ${\mathit{\theta }}_v$ and ${\mathit{\theta }}_t$ are the learnable parameters. In addition, the similarity between two binary codes ${\mathit{B}}_i^v$ and ${\mathit{B}}_j^t$ is measured by Hamming distance $D\left({\mathit{B}}_i^v,{\mathit{B}}_j^t\right)=\frac{1}{2}\left(C-<{\mathit{B}}_i^v,{\mathit{B}}_j^t>\right)$, which is the base for implementing cross-modal hashing retrieval.

3.2. Overview of MGSPU

Figure 1 shows the overview of the MGSPU framework. Specifically, it mainly consists of four modules: (1) feature extraction module, (2) semantic similarity matrix updating module, (3) hash code learning module, and (4) hash function learning module. The feature extraction module maps images and texts into feature subspaces by corresponding encoders: deep CNN for image features and Bag-of-Word (BoW) model for text features. The semantic similarity matrix updating module realizes a novel similarity updating strategy to construct a high-confidence similarity matrix, which effectively enhance the hash code learning and hash function learning. To support hash function learning, the hash code learning module aims at generating real-value hash representations with a novel multi-grained semantic preserving learning strategy. It utilizes a pair of GCNs (each for one modality) to capture global semantic correlation, following by a cross-attention mechanism to enhance cross-modal local feature learning. This technique allows for sufficient semantic correlations preserving of both intra- and inter-modality. The hash function learning module is to learn two hash functions that are implemented by two fully-connected neural networks for image and text, respectively.

In a nutshell, the goal of this method is to optimize the hash function module with other three parts to generate high-quality binary hash codes for cross-modal retrieval task. Thereinafter, we discuss the technical details module by module.

3.3. Feature Extraction

The first step of cross-modal hash code learning is feature extraction from both images and texts. For images, we follow previous works [12,37,48] to extract deep features from CNN model (pre-trained on ImageNet). Specifically, we extract the the 4096-dimensional features from the first fc7 layer (after ReLU) of AlexNet [50] as the original image features ${\mathit{F}}^{\mathit{v}}\in {\mathbb{R}}^{m\times 4096}$ for the batch-input images ${\left\{{\mathit{v}}_i\right\}}_{i=1}^m$. While for texts ${\left\{{\mathit{t}}_i\right\}}_{i=1}^m$ we directly adopt the BoW [51] vectors as their original features ${\mathit{F}}^{\mathit{t}}\in {\mathbb{R}}^{m\times d_t}$. It is noted that the proposed MGSPU framework can also be compatible with other deep feature extraction models, such as ResNet [52] for images or LSTM [53] for texts. To better validate the superiority of the proposed technique, we use relatively naive but effective feature extraction models (AlexNet and BoW) when implementing MGSPU. The implementation details are presented in Section 4.3.

3.4. Semantic Similarity Matrix Updating

Unlike supervised hashing learning that do not easily suffer from noises in features due to accurate category annotation, the unsupervised method cannot ensure the high accuracy of latent semantic correlations between instances without category information. To address this issue, we focus on constructing a high-confidence unified similarity matrix $\mathit{S}$ as a supervisory signal to filter out noises as much as possible. Accordingly, a novel semantic similarity matrix updating strategy is designed.

3.4.1. Similarity Matrix Construction

To comprehensively consider cross-modal similarity relationships, we first fuse the similarity matrices from image modality and text modality:

$${\mathit{S}}^f={\alpha }_1{\mathit{S}}^v+(1-{\alpha }_1){\mathit{S}}^t,$$

(2)

where ${\mathit{S}}^f$ is the fused similarity matrix, ${\mathit{S}}^v$ and ${\mathit{S}}^t$ represent the image and text similarity matrices, respectively. ${\alpha }_1\in [0,1]$ is a hyperparameter to balance them. Each element in ${\mathit{S}}^v$ and ${\mathit{S}}^t$ is computed by Cosine similarity. Taking image modality as an example:

$${\mathit{S}}_{ij}^v=cos({\mathit{v}}_i,{\mathit{v}}_j)=\frac{{\mathit{v}}_i^T{\mathit{v}}_j}{{∥{\mathit{v}}_i∥}_F^2{∥{\mathit{v}}_j∥}_F^2},$$

(3)

where ${\mathit{v}}_i$ and ${\mathit{v}}_j$ denote the feature of the i-th and j-th image. To rise the accuracy of similarity measurement without supervision, we design a filter-and-augment strategy to refine the similarity matrix, shown as follows:

$${\mathit{S}}^r=\left\{\begin{array}{cc}sgn\left({\mathit{S}}^f\right),\hfill & \left|{\mathit{S}}_{ij}^f\right|>{\eta }_1\hfill \\ 2\times sigmoid(2\times {\mathit{S}}^f)-1+\mathit{I},\hfill & otherwise\hfill \end{array}\right.,$$

(4)

where ${\mathit{S}}^r$ denotes the refined similarity matrix, ${\eta }_1$ is similarity threshold. the sign function $sgn\left({\mathit{S}}^f\right)$ indicates if ${\mathit{S}}_{ij}^f>{\eta }_1$, ${\mathit{S}}_{ij}^f=1$; if ${\mathit{S}}_{ij}^f<-{\eta }_1$, ${\mathit{S}}_{ij}^f=-1$. Meanwhile, inspired by reference [28], we use the non-linear function $sigmoid(·)$ to compress the remaining values to further expand the distance between similar and non-similar instances. The coefficient 2 of ${\mathit{S}}^f$ is to control the linear compression, then $2\times sigmoid(·)-1$ maps the compressed value to $[-1,1]$. $\mathit{I}$ is an identity matrix to maintain the similarity between instance pairs.

3.4.2. Dual Instruction Fusion Updating

In order to preserve semantic similarity consistency in cross-modal hash codes, a reliable similarity matrix for unsupervised hash function learning is indispensable. During the training process, unfortunately, we found that there is a evident gap between the similarity relationships of cross-modal hash codes and the real relationships of original instances due to the noises existing in the flawed similarity matrix ${\mathit{S}}^r$. In other words, the value of $|{\mathit{S}}_{ij}^r-{\mathit{S}}_{ij}^h|$ is not small enough to correctly guide cross-modal hash function learning. Intuitively, if we rectify these incorrect similarity values, it will effectively embed the correct similarity relationship information into cross-modal hash codes. For this reason, we attempt to construct dual instruction to further denoise the similarity matrix ${\mathit{S}}^r$: the one is the difference of similarity value of ${\mathit{S}}_{ij}^r$ and ${\mathit{S}}_{ij}^h$, the other is difference of the sign of them. Beyond all doubt, these two instructions actually measure the difference between ${\mathit{S}}^r$ and ${\mathit{S}}^h$ from two different perspectives, which could be fused to narrow the gap between ${\mathit{S}}^r$ and ${\mathit{S}}^h$.

From the analysis mentioned, we develop a novel semantic similarity matrix updating strategy, named dual instruction fusion updating, to gradually eliminate the noises. Firstly, we construct the hashing similarity matrix ${\mathit{S}}^h$ via the real-valued hash code ${\mathit{H}}^v$ and ${\mathit{H}}^t$:

$${\mathit{S}}^h=cos({\mathit{H}}^v,{\mathit{H}}^v)+cos({\mathit{H}}^t,{\mathit{H}}^t)+cos({\mathit{H}}^v,{\mathit{H}}^t).$$

(5)

Thereafter, we utilize ${\mathit{S}}^h$ to update ${\mathit{S}}^r$ according to the similarity relationship among them. For the convenience of formal description and implementation, we introduce two indicator matrices $\mathit{\Lambda }$ and $\mathit{\Gamma }$ at first. The former is defined as ${\mathit{\Lambda }}_{ij}={\mathit{S}}_{ij}^r\oplus {\mathit{S}}_{ij}^h$, therein ⊕ is XOR operation, which means if ${\mathit{S}}_{ij}^r\times {\mathit{S}}_{ij}^h>0$, ${\mathit{\Lambda }}_{ij}=0$; otherwise ${\mathit{\Lambda }}_{ij}=1$. The latter indicates the numerical difference between ${\mathit{S}}_{ij}^h$ and ${\mathit{S}}_{ij}^r$, therein the elements $|{\mathit{S}}_{ij}^r-{\mathit{S}}_{ij}^h|>{\eta }_2$ are set to 1 and the others are set to 0. ${\eta }_2\in [0,1]$ be a threshold to measure the difference between two elements in similar matrices. Accordingly, two cases are considered: (1) Not updating: if ${\mathit{\Lambda }}_{ij}=0$ and ${\mathit{\Gamma }}_{ij}=0$, then the updating do not be conducted since ${\mathit{S}}^h$ and ${\mathit{S}}^r$ are similar enough. (2) Updating: according to the value of ${\mathit{\Lambda }}_{ij}$, we introduce two updating rules: if ${\mathit{\Lambda }}_{ij}=0$ and ${\mathit{\Gamma }}_{ij}=1$, we update ${\mathit{S}}_{ij}^r$ by ${\mathit{S}}_{ij}^h$; if ${\mathit{\Lambda }}_{ij}=1$, we update ${\mathit{S}}_{ij}^r$ by “0”, which is a way with maximizing entropy to enhance generalization capability. The reason behind this rule is intuitive: if the difference between ${\mathit{S}}_{ij}^r$ and ${\mathit{S}}_{ij}^h$ is too large, a wise way is not to biased towards either side. To clearly show the update rules, we list a truth table for $\mathit{\Lambda }$ and $\mathit{\Gamma }$ in

Table 4. Indicator matrices truth table for dual instruction fusion updating strategy.

${\mathit{\Lambda }}_{\mathit{i}\mathit{j}}$	${\mathit{\Gamma }}_{\mathit{i}\mathit{j}}$	Update	Rules
0	0		${\mathit{S}}_{ij}^r$
0	1	✓	${\alpha }_2{\mathit{S}}_{ij}^r+(1-{\alpha }_2){\mathit{S}}_{ij}^h$
1	0	✓	0
1	1	✓	0

. Thereby, the updated semantic similarity matrix $\mathit{S}$ is generated by the following updating process:

$${\mathit{S}}_{ij}=\left\{\begin{array}{c}\begin{array}{ccc}\hfill & {\mathit{S}}_{ij}^r,\hfill & \hfill {\mathit{\Lambda }}_{ij}=0\cap {\mathit{\Gamma }}_{ij}=0\\ \hfill & {\alpha }_2{\mathit{S}}_{ij}^r+(1-{\alpha }_2){\mathit{S}}_{ij}^h,\hfill & \hfill {\mathit{\Lambda }}_{ij}=0\cap {\mathit{\Gamma }}_{ij}=1\\ \hfill & 0,\hfill & \hfill otherwise\end{array},\hfill \end{array}\right.$$

(6)

where ${\alpha }_2\in [0,1]$ is a parameter. Finally, we use $\mathit{S}$ to guide the cross-modal hash function learning avoiding the interference from similarity noise. Figure 2 illustrates the updating process.

3.5. MGSP for Hash Code Learning

To enhance the cross-modal similarity consistency preserving for hash code learning, a novel technique, called Multi-Grained Similarity Preserving (MGSP) is developed, as shown in Figure 3a. It consists two key sub-modules, i.e., global similarity aggregation (GSA) and local similarity interaction (LSI), which explore multi-grained similarity information: global and local similarity structure information. Furthermore, another sub-module, named Similarity Consistency Reconstruction (SCR) is involved to narrow heterogeneity gap between original features and reconstructed features from hash codes.

3.5.1. Global Similarity Aggregation

The similarity relationship between instances of any modality (image or text) is essential for similarity preserving, which reflects the latent feature distributions of each kind of data. As we known, the intra-modal similarity relationships can be represented as a graph therein each node is an instance and the similarity relationships are represented by edges. In such a graph, each node may be related to others through complex linked structure that can be captured to learn latent global similarity relationships within each modality. To this end, we developed a global similarity aggregation (GSA) sub-module involving the following process: two graphs firstly are constructed, each per modality. Then, we aggregate global similarity information of each node from its neighbors via a graph convolutional network (GCN). Specifically, let $\ast \in \{v,t\}$, for a batch of samples ${\mathit{F}}^{*}$, we feed them into an encoder to generate the latent representations ${\mathit{L}}^{*}$. To further focus on the crucial features, the key-value self-attention mechanism is used to obtain global similarity aggregation representations ${\mathit{X}}_g^{*}$. The query ${\mathit{Q}}_g^{*}$, key ${\mathit{K}}_g^{*}$ and value ${\mathit{V}}_g^{*}$ are calculated as:

$${\mathit{Q}}_g^{*}={\mathit{L}}^{*}{\mathit{W}}_{g,Q}^{*},{\mathit{K}}_g^{*}={\mathit{L}}^{*}{\mathit{W}}_{g,K}^{*},{\mathit{V}}_g^{*}={\mathit{L}}^{*}{\mathit{W}}_{g,V}^{*},$$

(7)

where ${\mathit{W}}_{g,Q}^{*},{\mathit{W}}_{g,K}^{*},{\mathit{W}}_{g,V}^{*}$ are the learnable weight matrices. The attention map ${\mathit{M}}_g^{*}$ is calculated as:

$${\mathit{M}}_g^{*}=softmax\left(\frac{{\mathit{Q}}_g^{*}{\left({\mathit{K}}_g^{*}\right)}^T}{\sqrt{d}}\right).$$

(8)

Thus, the global similarity enhanced representations is ${\mathit{X}}_g^{*}={\mathit{M}}_g^{*}{\mathit{V}}_g^{*}$. To avoid the issue of smaller feature value caused by attention weighting, we add the latent representation ${\mathit{L}}^{*}$ with ${\mathit{X}}_g^{*}$, then normalize it as the input of the GCN:

$${\mathit{G}}^{*\left(0\right)}=\sigma \left(norm({\mathit{L}}^{*}+{\mathit{X}}_g^{*})\right).$$

(9)

where the superscript $\left(0\right)$ denotes the 0-th layer of convolution, namely the input of GCN, $norm(·)$ is normalization function, $\sigma (·)$ represents a nonlinear activation function, usually the $ReLU(·)$ function. The layer-by-layer propagation rule of GCN is formulated as:

$${\mathcal{G}}^{\ast \left(l\right)}=\sigma \left({\tilde{\mathit{D}}}^{-\frac{1}{2}}\tilde{\mathit{A}}{\tilde{\mathit{D}}}^{-\frac{1}{2}}{\mathcal{G}}^{\ast (l-1)}{\mathit{W}}^{\ast \left(l\right)}\right),$$

(10)

where ${\tilde{\mathit{D}}}_{ii}={\sum }_j{\tilde{\mathit{A}}}_{ij}$, $\tilde{\mathit{A}}$ is an adjacency matrix constructed from the refined similarity matrix ${\mathit{S}}^r$ via kNN algorithm, k represents the number of neighbors, ${\mathit{G}}^{\ast (l-1)}$ and ${\mathit{G}}^{\ast \left(l\right)}$ represent the input and output of the l-th layer, ${\mathit{W}}^{\ast \left(l\right)}$ denotes the learnable parameters of the l-th layer, $\sigma (·)$ is $ReLU(·)$ function. The final outputs of the GCN are denoted by ${\mathit{G}}^{\ast }$. Finally, to further mitigate the cross-modal heterogeneity, a two-layer fully-connected network with weigh-sharing is integrated on the top of GCN, and its output are denoted as ${\mathit{Z}}^{\ast }$.

3.5.2. Local Similarity Interaction

Capturing rich similarity structure information among each image-text pair $({\mathit{v}}_i,{\mathit{t}}_i)$ is another desiderata to further enhance cross-modal similarity preserving. This kind of information is mainly reflected in the latent local feature relationships between modalities. To this end, we develop a local similarity interaction (LSI) sub-module to improve similarity consistency preserving within each image-text pair. Specifically, a key-value cross-attention mechanism is used on the top of GSA sub-module to learn fine-grained cross-modal similarity associations. We obtain query ${\mathit{Q}}_l^v$ and ${\mathit{Q}}_l^t$, key ${\mathit{K}}_l^v$ and ${\mathit{K}}_l^t$, as well as value ${\mathit{V}}_l^v$ and ${\mathit{V}}_l^t$ through linear transformation to obtain the visual and textual attention map ${\mathit{M}}_l^v$ and ${\mathit{M}}_l^t$:

$$\begin{array}{cc}\hfill & {\mathit{Q}}_l^v={\mathit{Z}}^v{\mathit{W}}_{l,Q}^v,{\mathit{K}}_l^v={\mathit{Z}}^v{\mathit{W}}_{l,K}^v,{\mathit{V}}_l^v={\mathit{Z}}^v{\mathit{W}}_{l,V}^v,\hfill \\ \hfill & {\mathit{Q}}_l^t={\mathit{Z}}^t{\mathit{W}}_{l,Q}^t,{\mathit{K}}_l^t={\mathit{Z}}^t{\mathit{W}}_{l,K}^t,{\mathit{V}}_l^t={\mathit{Z}}^t{\mathit{W}}_{l,V}^t.\hfill \end{array}$$

(11)

$$\begin{array}{cc}\hfill & {\mathit{M}}_l^v=softmax\left(\frac{{\mathit{Q}}_l^v{\left({\mathit{K}}_l^t\right)}^T}{\sqrt{d}}\right),\hfill \\ \hfill & {\mathit{M}}_l^t=softmax\left(\frac{{\mathit{Q}}_l^t{\left({\mathit{K}}_l^v\right)}^T}{\sqrt{d}}\right).\hfill \end{array}$$

(12)

Thus, the attention-weighted representations are ${\mathit{X}}_l^v={\mathit{M}}_l^v{\mathit{V}}_l^v$ and ${\mathit{X}}_l^t={\mathit{M}}_l^t{\mathit{V}}_l^t$. Similar to the above self-attention, the local are calculated as follows:

$$\begin{array}{cc}\hfill & {\mathit{P}}^v=\sigma \left(norm({\mathit{X}}_l^v+{\mathit{Z}}^v)\right),\hfill \\ \hfill & {\mathit{P}}^t=\sigma \left(norm({\mathit{X}}_l^t+{\mathit{Z}}^t)\right),\hfill \end{array}$$

(13)

where $\sigma (·)$ is still $ReLU(·)$.

3.5.3. Similarity Consistency Reconstruction

The similarity consistency preserving should not only be reflected in cross-modal features, but more importantly, in hash codes. Deep feature reconstruction, namely reducing the heterogeneity between original features and reconstructed features from continuous hash codes, has been verified to be an effective technique. Current literatures indicate that, two different reconstruction strategies, i.e., intra-modal [28] and inter-modal reconstruction [37], are developed for similarity consistency preserving. In this work, we integrate inter-modal reconstruction into our method to narrow cross-modal gap. Specifically, the continuous hash codes ${\mathit{H}}^v$ and ${\mathit{H}}^t$ are generated by a couple of two-layer fully-connected networks from ${\mathit{P}}^v$ and ${\mathit{P}}^t$, which then are fed into two re-constructors to output reconstructed features. Accordingly, inspired by [37], a similarity consistency reconstruction loss function ${\mathcal{L}}_{rec}$ is introduced to preserve the similarity structure information into hash codes:

$${\mathcal{L}}_{rec}={∥\widehat{{\mathit{F}}^v}-{\mathit{F}}^v∥}_F^2+{∥\widehat{{\mathit{F}}^t}-{\mathit{F}}^t∥}_F^2,$$

(14)

where $\widehat{{\mathit{F}}^v}$ and $\widehat{{\mathit{F}}^t}$ denote the reconstructed feature representations.

3.5.4. Hash Code Learning Loss

Furthermore, to preserve similarity consistency of cross-modal hash codes from both intra-modal and inter-modal perspectives, we introduce hash code similarity consistency loss:

$$\begin{array}{cc}\hfill {\mathcal{L}}_{mod}=& {∥cos({\mathit{H}}^v,{\mathit{H}}^v)-cos({\mathit{H}}^t,{\mathit{H}}^t)∥}_F^2\hfill \\ \hfill +& {∥cos({\mathit{H}}^v,{\mathit{H}}^t)-cos({\mathit{H}}^t,{\mathit{H}}^t)∥}_F^2\hfill \\ \hfill +& {∥cos({\mathit{H}}^v,{\mathit{H}}^t)-cos({\mathit{H}}^v,{\mathit{H}}^v)∥}_F^2.\hfill \end{array}$$

(15)

Meanwhile, to ensure the accuracy of the similarity matrix, we designed a new loss function ${\mathcal{L}}_{sim}$ as follows:

$$\begin{array}{cc}\hfill {\mathcal{L}}_{sim}=& {∥\mathit{S}-cos({\mathit{H}}^v,{\mathit{H}}^v)∥}_F^2\hfill \\ \hfill +& {∥\mathit{S}-cos({\mathit{H}}^t,{\mathit{H}}^t)∥}_F^2\hfill \\ \hfill +& {∥\mathit{S}-cos({\mathit{H}}^v,{\mathit{H}}^t)∥}_F^2\hfill \\ \hfill +& \mathit{\Gamma }{∥\mathit{S}-{\mathit{S}}^r∥}_F^2,\hfill \end{array}$$

(16)

where $\mathit{\Gamma }$ is an indicator function, when $\left|{\mathit{S}}_{ij}^r-{\mathit{S}}_{ij}^h\right|>{\eta }_2$, ${\Gamma }_{ij}=1$, otherwise ${\Gamma }_{ij}=0$.

3.6. Hashing Function Learning

Using the semantic similarity matrix $\mathit{S}$ and the hash codes ${\mathit{H}}^v,{\mathit{H}}^t$, we learn two modality-specific hash functions $sgn\left(f^v(.;{\mathit{\theta }}_v)\right)$ and $sgn\left(f^t(.;{\mathit{\theta }}_t)\right)$ to project deep features ${\mathit{F}}^v$ and ${\mathit{F}}^t$ to Hamming space. Specifically, to preserving the similarity structure between hash codes and cross-modal features, we introduce similarity loss function ${\mathcal{L}}_{f_1}$ as follows:

$$\begin{array}{cc}\hfill {\mathcal{L}}_{f_1}=& {∥\mathit{S}-cos(f^v({\mathit{F}}^v;{\mathit{\theta }}_v),f^v({\mathit{F}}^v;{\mathit{\theta }}_v))∥}_F^2\hfill \\ \hfill +& {∥\mathit{S}-cos(f^t({\mathit{F}}^t;{\mathit{\theta }}_t),f^t({\mathit{F}}^t;{\mathit{\theta }}_t))∥}_F^2\hfill \\ \hfill +& {∥\mathit{S}-cos(f^v({\mathit{F}}^v;{\mathit{\theta }}_v),f^t({\mathit{F}}^t;{\mathit{\theta }}_t))∥}_F^2,\hfill \end{array}$$

(17)

where $f^v({\mathit{F}}^v;{\mathit{\theta }}_v)$ and $f^t({\mathit{F}}^t;{\mathit{\theta }}_t)$ denote the relaxed hash codes generated by the hash functions, ${\mathit{\theta }}_v$ and ${\mathit{\theta }}_t$ denote the parameters. In addition, in order to numerically align the hash codes generated by the hash function with the hash codes ${\mathit{H}}^v$ and ${\mathit{H}}^t$ obtained during the training process, we also introduced loss function ${\mathcal{L}}_{f_2}$:

$$\begin{array}{cc}\hfill {\mathcal{L}}_{f_2}=& {∥{\mathit{H}}^v-f^v({\mathit{F}}^v;{\mathit{\theta }}_v)∥}_F^2\hfill \\ \hfill +& {∥{\mathit{H}}^t-f^t({\mathit{F}}^t;{\mathit{\theta }}_t)∥}_F^2\hfill \\ \hfill +& {∥f^v({\mathit{F}}^v;{\mathit{\theta }}_v)-f^t({\mathit{F}}^t;{\mathit{\theta }}_t)∥}_F^2.\hfill \end{array}$$

(18)

Finally, the quantization loss ${\mathcal{L}}_{f_3}$ transfers semantic information from relaxed hash codes to binary hash codes:

$${\mathcal{L}}_{f_3}={∥{\mathit{H}}^v-sign\left({\mathit{H}}^v\right)∥}_F^2+{∥{\mathit{H}}^t-sign\left({\mathit{H}}^t\right)∥}_F^2.$$

(19)

3.7. Optimization

The learning process of MGSPU can be divided into two stages: (1) hash code learning stage, and (2) hash function learning stage. During the hash code learning stage, the optimization of the objective function ${\mathcal{L}}_{code}$ is performed by minimizing Equations (14)–(16):

$$\underset{{\mathit{\theta }}^c}{min}{\mathcal{L}}_{code}={\lambda }_1{\mathcal{L}}_{rec}+{\lambda }_2{\mathcal{L}}_{mod}+{\lambda }_3{\mathcal{L}}_{sim}.$$

(20)

During the hash function learning stage, the optimization of the objective function ${\mathcal{L}}_{hash}$ entails minimizing Equations (17)–(19):

$$\underset{{\mathit{\theta }}^v,{\mathit{\theta }}^t}{min}{\mathcal{L}}_{hash}={\beta }_1{\mathcal{L}}_{f_1}+{\beta }_2{\mathcal{L}}_{f_2}+{\beta }_3{\mathcal{L}}_{f_3}.$$

(21)

The entire algorithm process of MGSPU is outlined in Algorithm 1, which is implemented by Adam adaptive algorithm [54].

Algorithm 1 Algorithm of MGSPU.

Input: Training set size n; number of epochs e; hash code length C; batch size m; learning rate of the network $l{r}_{code}$ and $l{r}_{hash}$; hyperparameter ${\alpha }_1,{\alpha }_2,{\eta }_1,{\eta }_2,k,{\lambda }_1,{\lambda }_2,{\lambda }_3,{\beta }_1,{\beta }_2,{\beta }_3$; Output: Network parameters for the optimal hash function learning part; 1: Initialize network parameters ${\mathit{\theta }}^c,{\mathit{\theta }}^v,{\mathit{\theta }}^t$; 2: Construct a unified semantic matrix ${\mathit{S}}^r$ using image features ${\mathit{F}}^v$ and text features ${\mathit{F}}^t$; 3: repeat 4:    Iterative training e times; 5:    for $i=1,2,...,\lceil \frac{n}{m}\rceil$ do 6:   Arbitrarily select m training data; 7:   Generate continuous hash codes ${\mathit{H}}^v,{\mathit{H}}^t$ by forward propagation algorithm; 8:   Generate the semantic matrix ${\mathit{S}}^h$ through the hash code ${\mathit{H}}^v,{\mathit{H}}^t$, and update the original semantic matrix ${\mathit{S}}^r$ to generate a unified semantic matrix $\mathit{S}$; 9:   Calculate the loss function Equations (14)–(16) and update the parameters through backpropagation ${\mathit{\theta }}^c$; 10:    end for 11:    for $i=1,2,...,\lceil \frac{n}{m}\rceil$ do 12:   Select m training image-text pairs and hash code ${\mathit{H}}^v,{\mathit{H}}^t$; 13:   Map the original feature ${\mathit{F}}^v,{\mathit{F}}^t$ into a hash code through $f^v$ and $f^t$; 14:   Calculate the loss Equations (17)–(19), and update the parameter ${\mathit{\theta }}^v,{\mathit{\theta }}^t$ with backpropagation; 15:    end for 16: until Convergence.

4. Experiments

This section presents experiments and analysis to assess the retrieval performance of the proposed method. We begin by introducing the experimental settings, including datasets, evaluation metrics, baselines, and implementation details. Subsequently, we provide a performance comparison between our method and several baselines, along with an ablation analysis to validate the impact of each component.

4.1. Datasets

We conduct thorough experiments on two prominent multimedia benchmark datasets, namely MIRFLICKR-25K [55] and NUS-WIDE [56], widely employed for cross-modal retrieval evaluation. A concise introduction to these datasets is provided below.

MIRFLICKR-25K. The MIRFLICKR-25K dataset includes 25,000 image-text pairs from the popular photo-sharing platform Flickr. Each image is accompanied by multiple text labels. In our experiment, we specifically selected instances with at least 20 text tags. With AlexNet [50], we converted each image into a depth feature of 4096 dimensions, while the text labels were converted into a BoW [51] vector of 1386 dimensions. In addition, each instance is manually annotated with at least one of 24 unique tags. We experimented with 20,015 examples selected from the dataset.

NUS-WIDE. The NUS-WIDE dataset is a substantial real-world web image collection, featuring more than 269,000 images accompanied by over 5000 user-provided tags and 81 concepts across the entire dataset. Using AlexNet [50], each image instance is represented as a 4096-dimensional deep feature, while the textual content is condensed into a 1000-dimensional BoW [51] vector. For our experiment, we excluded instances lacking labels and focused on those associated with the 10 most frequent categories, resulting in a curated set of 186,577 image-text pairs.

Table 5. Details about the datasets used in our experiments.

Dataset	Train	Query	Retrieval	Text Feature
MIRFLICKR-25K	5000	2000	18,015	1386 d
NUS-WIDE	5000	2000	184,577	1000 d

presents the statistics of the above two datasets, and some samples of these two datasets are shown in Figure 4.

4.2. Evaluation Metrics

In our experiments, we conducted two types of cross-modal retrieval tasks: retrieving texts using image queries (denoted as “I2T”) and retrieving images using text queries (denoted as “T2I”). Next, we utilized two standard hashing performance protocols, Hamming ranking and hash lookup [57], to assess the effectiveness of our method and its competitors. For the Hamming ranking protocol, we utilized mean average precision (mAP) to measure accuracy, while precision-recall curves (P-R curves) were employed for the Hash lookup protocol. For mAP and P-R curves, we considered images and texts to be similar if they shared at least one label; otherwise, they were considered dissimilar. Specifically, given a query $q_i$, the average precision (AP) of the top-N results is defined as:

$$AP\left(q_i\right)=\frac{1}{N}\sum _{r=1}^{R}p\left(r\right)d\left(r\right),$$

(22)

where N is the number of relevant instances in the result set, R represents the total amount of data. $p\left(r\right)$ denotes the precision of the top-r results. If the r-th retrieved result is relevant to the query instances, $d\left(r\right)=1$; otherwise, $d\left(r\right)=0$. The mAP value is defined as the average AP across all queries $q_i$:

$$mAP=\frac{1}{M}\sum _{i=1}^{M}AP\left(q_i\right),$$

(23)

where M represents the number of queries.

4.3. Baselines and Implementation Details

Baselines. we compare the proposed MGSPU method with nine baselines, including CMFH [31], DBRC [58], UDCMH [35], DJSRH [12], JDSH [36], DSAH [37], AGCH [26], DAEH [27], DRNPH [49], which are briefly described as follows:

• CMFH: This method learns uniform binary feature vectors for different modalities through collective matrix factorization of latent factor models.
• DBRC: This approach proposes a deep binary reconstruction model to preserve inter-modal correlation.
• UDCMH: This method utilizes deep learning and matrix factorization with binary latent factor models for multi-modal data search.
• DJSRH: This approach integrates original neighborhood information from different modalities into a joint-semantics affinity matrix to extract latent intrinsic semantic relations.
• JDSH: This method introduces a distribution-based similarity decision and weighting scheme for generating a more discriminative hash code.
• DSAH: This approach explores similarity information across modalities and incorporates a semantic-alignment loss function to align features’ similarities with those between hash codes.
• AGCH: This method utilizes GCNs to uncover semantic structures, coupled with a fusion module for correlating different modalities.
• DAEH: This approach attempts to train hash functions with discriminative similarity guidance and an adaptively-enhanced optimization strategy.
• DRNPH: This method implements unsupervised deep relative neighbor relationship preserving cross-modal hashing for achieving cross-modal retrieval in a common Hamming space.

Except for CMFH, All other approaches use deep features to generate cross-modal hash codes.

Implementation Details. As discussed above, the learning process is divided into two stages. In hash code learning stage, three hyperparameters ${\lambda }_1,{\lambda }_2,{\lambda }_3$ are used to weight ${\mathcal{L}}_{rec},{\mathcal{L}}_{mod},{\mathcal{L}}_{sim}$, respectively. In hash function learning stage, three other hyperparameter ${\beta }_1,{\beta }_2,{\beta }_3$ are used to adjust the ratio between ${\mathcal{L}}_{f_1},{\mathcal{L}}_{f_2},{\mathcal{L}}_{f_3}$. On the MIRFLICKR-25K dataset, we set ${\lambda }_1=0.1,{\lambda }_2=1,{\lambda }_3=10,{\beta }_1=1,{\beta }_2=0.01,{\beta }_3=1$. On NUS-WIDE dataset, we set ${\lambda }_1=1,{\lambda }_2=1,{\lambda }_3=10,{\beta }_1=10,{\beta }_2=0.01,{\beta }_3=10$. In the process of semantic similarity matrix construction, we set ${\alpha }_1=0.6,{\eta }_1=0.8,{\alpha }_2=0.4,{\eta }_2=0.7$ on the MIRFLICKR-25K, and set ${\alpha }_1=0.4,{\eta }_1=0.5,{\alpha }_2=0.7,{\eta }_2=0.6$ on the NUS-WIDE. In GSA sub-module, kNN algorithm is used to aggregate nodes in a certain neighborhood for each modality. We set $k=40$ and $k=60$ for MIRFLICKR-25K and NUS-WIDE, respectively. The optimization algorithm used is the Adam optimization algorithm [54]. For MIRFLICKR-25K, we set the learning rates for hash code learning and hash function learning to 0.001 and 0.0001, respectively, and for NUS-WIDE, both are set to 0.0001. The batch size is consistently set at 512. The number of iterations is defined as 60 for MIRFLICKR-25K and 100 for NUS-WIDE. It’s worth noting that, under the same experimental setup, we directly utilize mAP@50 results provided in the original papers of the baseline methods.

Experimental Environment. All the experiments are performed on a workstation with Intel(R) Core i9-12900K 3.9 GHz CPU, 128 GB RAM, 1 TB SSD storage, 2TB HDD storage, and 1 NVIDIA GeForce RTX 3090Ti GPU with ubuntu-22.04.1 operating system. All the techniques are implemented by Python 3.9 on PyTorch 2.0.1.

4.4. Performance Evaluation

We compare the proposed method with nine baselines on MIRFLICKR-25K and NUS-WIDE datasets. The performance of all these methods are evaluated by Hamming Ranking protocol and hash lookup protocol.

Table 6. mAP@50 score of our method and the baselines at various code lengths (bits) on MIRFLICKR-25K.

Methods	MIRFLICKR-25K
	I2T				T2I
	16 bits	32 bits	64 bits	128 bits	16 bits	32 bits	64 bits	128 bits
CMFH	0.621	0.624	0.625	0.627	0.462	0.662	0.676	0.685
DBRC	0.617	0.619	0.620	0.621	0.618	0.622	0.626	0.628
UDCMH	0.689	0.698	0.714	0.717	0.692	0.704	0.718	0.733
DJSRH	0.810	0.843	0.862	0.876	0.786	0.822	0.835	0.847
JDSH	0.832	0.853	0.882	0.892	0.825	0.864	0.878	0.880
DSAH	0.863	0.877	0.895	0.903	0.846	0.860	0.881	0.882
AGCH	0.865	0.887	0.892	0.912	0.829	0.845	0.852	0.880
DAEH	0.812	0.835	0.847	0.845	0.778	0.819	0.825	0.831
DRNPH	0.876	0.902	0.914	0.933	0.860	0.872	0.885	0.897
Ours	0.898	0.915	0.927	0.936	0.876	0.883	0.889	0.900

and

Table 7. mAP@50 score of our method and the baselines at various code lengths (bits) on NUS-WIDE.

Methods	NUS-WIDE
	I2T				T2I
	16 bits	32 bits	64 bits	128 bits	16 bits	32 bits	64 bits	128 bits
CMFH	0.455	0.459	0.465	0.467	0.529	0.577	0.614	0.652
DBRC	0.424	0.459	0.447	0.447	0.455	0.459	0.468	0.473
UDCMH	0.511	0.519	0.524	0.558	0.637	0.653	0.695	0.716
DJSRH	0.724	0.773	0.798	0.817	0.712	0.744	0.771	0.789
JDSH	0.736	0.793	0.832	0.835	0.721	0.785	0.794	0.804
DSAH	0.775	0.805	0.818	0.827	0.770	0.790	0.804	0.815
AGCH	0.809	0.830	0.831	0.852	0.769	0.780	0.798	0.802
DAEH	0.766	0.789	0.809	0.822	0.718	0.751	0.766	0.767
DRNPH	0.790	0.811	0.826	0.837	0.780	0.795	0.804	0.811
Ours	0.811	0.826	0.844	0.858	0.780	0.786	0.806	0.813

illustrates the mAP@50 results of our method and the competitors varying hash code lengths (16, 32, 64, 128 bits) on MIRFLICKR-25K and NUS-WIDE. Figure 5 and Figure 6 show the P-R curves on these two datasets in various code length. The detailed analysis and observation are presented as follows.

Hamming Ranking. It is clearly from the Table 6 and Table 7 that the proposed method performs better than the baselines. Specifically, on MIRFLICKR-25K, we found that our method achieved the highest mAP@50 score for both retrieval tasks (I2T: mAP@50 = 0.898 (16 bits), mAP@50 = 0.915 (32 bits), mAP@50 = 0.927 (64 bits), mAP@50 = 0.936 (128 bits); T2I: mAP@50 = 0.876 (16 bits), mAP@50 = 0.883 (32 bits), mAP@50 = 0.889 (64 bits), mAP@50 = 0.900 (128 bits)). For example, our method beats out the strongest competitor, DRNPH, by a significant margin on both two tasks, especially in shorter hash code length: 0.022 (16 bits), 0.013 (32 bits), 0.013 (64 bits) on I2T task, and 0.016 (16 bits), 0.011 (32 bits) on T2I task. The reason behind these results is obvious: with the engagement of the proposed similarity matrix updating strategy, our method can gradually eliminate the noise of the original features used in the similarity relationship construction so as to improve similarity consistency preserving, which is unfortunately ignored by DRNPH. In all but a few case (32 bits code length on I2T and I2T task, MGSPU was defeated by AGCH and DSAH marginally), our method won the competition again on NUS-WIDE by stand-out performance: mAP@50 = 0.811 (16 bits), 0.826 (32 bits), 0.844 (64 bits), 0.858 (128 bits) on I2T task, and mAP@50 = 0.780 (16 bits), 0.786 (32 bits), 0.806 (64 bits), 0.813 (128 bits) on T2I task. Compared with mainstream solutions, complex similarity correlations can be greatly mined through the semantic similarity matrix update strategy, and the MGSP module further retains the potential similarity structure between data.

Hash Lookup. To comprehensively showcase the comprehensive performance comparison of MGSPU with baselines, we draw P-R curves in Figure 5 and Figure 6 with different code lengths on both two datasets. As expected, in addition to dramatically defeating hand-crafted feature based method CMFH, MGSPU outperforms state-of-the-art competitors DRNPH, DSAH and AGCH on various hash code length. This observation is mainly due to the search performance boosting from the interplay of similarity updating and multi-grained similarity preserving of hash codes.

Discussion. It is no secret that, the main reason of poor performance of early models (such as CMFH and DBRC) is their shallow feature extraction techniques that cannot obtain feature representations with rich semantic information. With the help of powerful deep learning techniques, deep neural networks based methods such as DJSRH, JDSH, AGCH and DAEH achieved good results. Among them, DJSRH is equiped with a reconstruction framework for training, which is more competitive than batch training. AGCH uses GCN to aggregate neighborhood information and enhance feature expression. DAEH leverages teacher networks to enhance weaker hashing networks. However, all of them build similarity matrices based on original features, which inevitably bring noises into semantic relationships so as to introduce biases. Furthermore, these methods either maintain local or global similarities to preserve the semantic relationships. For example, the similarity matrix in DJSRH contains redundant information from intra-modal fusion items, while DAEH ignores the semantic relationships of intra-modal details. Comparing with these solutions, therefore, we argue that stepwise denoising through a similarity matrix update strategy can greatly mine complex similarity correlations, thereby generating high-confidence supervision signals. In addition, the MGSP method can effectively improve the hash code quality due to further preserving the potential similar structures within and between modalities. Both the mAP@50 score on the hash ranking protocol and the area under the P-R curves on the hash lookup procotol strongly support our view.

4.5. Ablation Study

To verify the validity of each design in MGSPU, we conducted ablation experiments on the MIRFLICKR-25K and NUS-WIDE datasets, serveral variations were considered for this purpose:

• MGSPU-1: it removes semantic similarity matrix updating from MGSPU.
• MGSPU-2: it removes the similarity consistency reconstruction from MGSPU.
• MGSPU-3: it modifies similarity consistency reconstruction by replacing inter-modal reconstruction with intra-modal reconstruction.
• MGSPU-4: it removes the GCN module from MGSPU.

From

Table 8. mAP@50 score for ablation study on MIRFLICKR-25K.

Methods	MIRFLICKR-25K
	I2T				T2I
	16 bits	32 bits	64 bits	128 bits	16 bits	32 bits	64 bits	128 bits
MGSPU-1	0.894	0.912	0.924	0.933	0.872	0.876	0.883	0.892
MGSPU-2	0.886	0.902	0.909	0.913	0.849	0.857	0.866	0.876
MGSPU-3	0.897	0.913	0.926	0.934	0.872	0.885	0.889	0.894
MGSPU-4	0.895	0.904	0.924	0.926	0.868	0.873	0.886	0.887
Ours	0.898	0.915	0.927	0.936	0.876	0.883	0.889	0.900

and

Table 9. mAP@50 score for ablation study on NUS-WIDE.

Methods	NUS-WIDE
	I2T				T2I
	16 bits	32 bits	64 bits	128 bits	16 bits	32 bits	64 bits	128 bits
MGSPU-1	0.803	0.821	0.837	0.849	0.763	0.783	0.804	0.805
MGSPU-2	0.796	0.809	0.819	0.821	0.746	0.775	0.777	0.776
MGSPU-3	0.801	0.821	0.827	0.851	0.768	0.791	0.799	0.812
MGSPU-4	0.794	0.826	0.843	0.847	0.773	0.786	0.798	0.797
Ours	0.811	0.826	0.844	0.858	0.780	0.786	0.806	0.813

, the following observations can be obtained: firstly, the comparison of MGSPU-1 with our the full MGSPU method verifies that the proposed dual instruction fusion updating strategy can improve the quality of instance similarity matrix to enhance the retrieval performance. Specifically, the retrieval accuracy of MGSPU-1 for both I2T and T2I task show decrease in some extent: on MIRFLICKR-25K, mAP@50 results of I2T task drop from 0.898 (16 bits), 0.915 (32 bits), 0.927 (64 bits), 0.936 (128 bits) to 0.894 (16 bits), 0.912 (32 bits), 0.924 (64 bits), 0.933 (128 bits), respectively; mAP@50 results of T2I task drop from 0.876 (16 bits), 0.883 (32 bits), 0.889 (64 bits), 0.900 (128 bits) to 0.872 (16 bits), 0.876 (32 bits), 0.883 (64 bits), 0.892 (128 bits), respectively. It indicates that without the semantic similarity matrix updating, complex similarity relationship learning suffers from disturbance by noise. Secondly, we can clearly observe that MGSPU-2 has a remarkably performance degradation compared with the full version of MGSPU. This phenomenon confirms that similarity consistency reconstruction is beneficial to preserve semantic information into hash code. Thirdly, with intra-modal reconstruction, MGSPU-3 performs better than MGSPU-2 especially on long hash codes (e.g., 64 or 128 bits). However, compared with the inter-modal reconstruction used in our method, the performance of MGSPU-3 is slightly weaker, which indicates that the inter-modal reconstruction is more helpful to reduce the heterogeneity between the original feature and the hash code. Lastly, after removing GCN module, MGSPU-4 achieves lower retrieval accuracy than ours. these results show that the structural similarity aggregated from neighborhoods by the GCN module is essential to enrich the similarity relationship information of each instance.

4.6. Sensitivity to Hyperparameters

This section, we conduct an analysis of the sensitivity of all hyperparameters used in the model: ${\alpha }_1,{\eta }_1;{\alpha }_2,{\eta }_2;k;{\lambda }_1,{\lambda }_2,{\lambda }_3;{\beta }_1,{\beta }_2,{\beta }_3$. To explore the comprehensive impact of them, the accuracy of I2T and T2I task is used to visualize the trend of cross-modal hash performance. All these analysis are carried out in 16 bit hash code length on NUS-WIDE dataset.

Hyperparameter ${\alpha }_1,{\eta }_1,{\alpha }_2,{\eta }_2$. In semantic similarity matrix updating module, ${\alpha }_1,{\eta }_1$ are used to construct the refined similarity matrix ${\mathit{S}}^r$, ${\alpha }_2,{\eta }_2$ are used to execute the updating strategy to generate semantic similarity matrix $\mathit{S}$. We observe the performance change of MGSPU by varying ${\alpha }_1,{\eta }_1$ and ${\alpha }_2,{\eta }_2$. According to the experimental results in Figure 7, it is clearly that the retrieval accuracy is more susceptible to changes of ${\alpha }_1$ when the value of ${\eta }_1$ is small. We speculate that this phenomenon is caused by the noises that are injected in refined similarity matrix when the value of ${\eta }_1$ is small. On the other hand, when we change the ${\alpha }_2$ and ${\eta }_2$, the fluctuations in model performance are relatively less severe, which still shows that our model performs a bit better if ${\eta }_2$ is set to a large value (r.g., ${\eta }_2=0.70$). The reason behind this results is understandable: if a larger threshold ${\eta }_2$ is taken, the discrimination of whether ${\mathit{S}}_{ij}^r$ and ${\mathit{S}}_{ij}^h$ are dissimilar will be more rigorous. Under this circumstance, the semantic similarity matrix updating will be executed more cautiously to preserve robust.

Hyperparameter k. We recorded the performance change by varying the value of k on NUS-WIDE dataset to evaluate the effect by the the number of neighbors in kNN algorithm. As demonstrated in Figure 8, when $k\in [40,80]$, the curve changes sharply, while the curve changes more modestly in the rest of the interval. We conjecture that when too many or too few neighbors we selected, noise will be introduced into the intra-modal similarity relationship representation, thereby affecting the latent similarity relationship learning within modality. Particularly, if k is set to 60, MGSPU achieves the highest mAP@50 score for both I2T and T2I task. It indicates that by selecting an appropriate number of neighbors, high-quality intra-modal similarity structure information can be aggregated by GCN to improve intra-modal similarity consistency preserving.

Hyperparameter${\lambda }_1,{\lambda }_2,{\lambda }_3$. As presented in Equation (20), ${\lambda }_1,{\lambda }_2,{\lambda }_3$ are used to balance three components, i.e., ${\mathcal{L}}_{re}$, ${\mathcal{L}}_{mod}$ and ${\mathcal{L}}_{sim}$ of hash code learning loss function. To analyze the effect by these three losses, we recorded the performance change of our method in Figure 8 by varying ${\lambda }_1,{\lambda }_2,{\lambda }_3$ from 0.001 to 100 with a 10-fold increase. It is noteworthy that our method obtains the best performance if we set ${\lambda }_1=1,{\lambda }_2=1,{\lambda }_3=10$. Among these three losses, we found that the new designed loss ${\mathcal{L}}_{sim}$ has a relatively greater effect on the learning of hash code. We argue that this is mainly due to the indispensability of a reliable similarity matrix for unsupervised hash learning.

Hyperparameter ${\beta }_1,{\beta }_2,{\beta }_3$. As depicted in Equation (21), ${\beta }_1,{\beta }_2,{\beta }_3$ serves as weight factor to balance ${\mathcal{L}}_{f_1}$, ${\mathcal{L}}_{f_2}$, and ${\mathcal{L}}_{f_3}$. The observations in Figure 8 indicates that ${\mathcal{L}}_{f_1}$ and ${\mathcal{L}}_{f_3}$ contributes more than ${\mathcal{L}}_{f_2}$. We infer that although the duty of ${\mathcal{L}}_{f_2}$ is to ensure numerical consistency between the generated hash codes ${\mathit{H}}^{*}$ and trained hash codes and $sgn\left(f^{*}({\mathit{F}}^{*};{\mathit{\theta }}_{*})\right)$, the main goal of cross-modal hash learning is still to eliminate cross-modal heterogeneity, which is only achieved by ${\mathcal{L}}_{f_1}$. Besides, the quantization error is cannot reduced by other losses but ${\mathcal{L}}_{f_3}$. In addition, by setting ${\beta }_1=10,{\beta }_2=0.01,{\beta }_3=10$, our method achieves the best performance.

5. Conclusions

In this paper, we propose a novel unsupervised hashing learning framework, called multi-grained similarity preserving and updating to improve cross-modal hashing performance. To obtain a high-confidence similarity matrix, we develop an update strategy that corrects the similarity values after the original feature fusion using a matrix constructed in Hamming space, and a loss function is designed to guide the similarity update. Also to learn high-quality hash codes, we co-model from multiple granularity to preserve semantic correlations within and between modalities. Specifically, GCNs is used to capture the global similarity relationship within each modality, and a cross-attention mechanism is used to perform interactions between modalities to bridge the heterogeneous gap. In addition, deep feature reconstruction further enhances inter-modal correlations and reduces modal gaps.

In our experiments, we use the hamming ranking protocol and the hash lookup protocol to compare with other benchmark models, and the experimental results on both datasets show that our approach achieves impressive performance. In addition, we set up four different ablation experiments to verify the performance of the designed module, and the results also validate the effectiveness of the designed module.

Although our experimental results achieved excellent results, for text data, we only used a simple bag-of-words model, and the gap between the results of image-retrieval text and text-retrieval image is relatively large. As an exploratory work, we plan to further improve the feature extraction stage in the future to achieve a more accurate alignment of the semantic information of images and text, which will lead to a more balanced cross-modal retrieval.