Cross-Gen: An Efficient Generator Network for Adversarial Attacks on Cross-Modal Hashing Retrieval

Abstract

Research on deep neural network (DNN)-based multi-dimensional data visualization has thoroughly explored cross-modal hash retrieval (CMHR) systems, yet their vulnerability to malicious adversarial examples remains evident. Recent work improves the robustness of CMHR networks by augmenting training datasets with adversarial examples. Prior approaches typically formulate the generation of cross-modal adversarial examples as an optimization problem solved through iterative methods. Although effective, such techniques often suffer from slow generation speed, limiting research efficiency. To address this, we propose a generative-based method that enables rapid synthesis of adversarial examples via a carefully designed adversarial generator network. Specifically, we introduce Cross-Gen, a parallel cross-modal framework that constructs semantic triplet data by interacting with the target model through query-based feedback. The generator is optimized using a tailored objective comprising adversarial loss, reconstruction loss, and quantization loss. The experimental results show that Cross-Gen generates adversarial examples significantly faster than iterative methods while achieving competitive attack performance.

1. Introduction

The exploration of multi-dimensional data visualization in deep learning has garnered significant attention due to its interpretability [1,2]. However, applying such techniques to cross-modal hash retrieval (CMHR) systems remains challenging as heterogeneous modalities—such as images and text—exhibit distinct representation formats and require modality-specific feature extraction methods. A typical CMHR system allows users to submit a query in one modality (e.g., an image or text) and retrieve semantically relevant results from the other modality. To bridge this gap, recent work widely employs deep learning approaches [3,4,5] for cross-modal feature embedding.

Despite their effectiveness, deep neural network (DNN)-based models are inherently vulnerable to imperceptible adversarial perturbations [6,7,8]. Seminal work by Szegedy et al. [9] demonstrated that minor input perturbations can induce substantial output changes due to the highly nonlinear nature of DNNs. This vulnerability enables adversaries to manipulate retrieval results, posing serious security risks. Critically, such perturbations are often imperceptible to humans. For instance, image perturbations involve subtle pixel modifications, while text perturbations may include minor alterations such as reordering words, inserting punctuation, or replacing characters.

Adversarial attacks are commonly categorized based on the adversary’s knowledge of the target model: white-box attacks [10,11,12], where full access to model parameters and architecture is assumed, and black-box attacks [13,14], where such information is unavailable. In practice, black-box settings are more realistic but also more challenging. Black-box attacks are further divided into query-based and transfer-based approaches. Query-based methods [15,16] estimate gradients through numerous queries to the target model, while transfer-based methods [17] generate perturbations using a surrogate model and exploit transferability to attack the target. Both paradigms, however, incur high computational costs or require extensive querying, highlighting the need for efficient general-purpose methods for rapid adversarial example generation.

While extensive research has addressed adversarial attacks in single-modality systems—such as image classification [18] and machine translation [19]—the cross-modal retrieval setting remains underexplored. The existing attack strategies generally fall into two categories: iterative-based [9] and generative-based methods [20,21]. Iterative approaches formulate the attack as an optimization problem solved via gradient descent, progressively refining perturbations toward an optimal solution. However, they demand repeated gradient computations or backward passes through the target model, resulting in high time overhead. In contrast, generative-based methods [21,22] leverage trained generator networks to produce adversarial examples efficiently, showing promise for scalable attacks in deep hash-based CMHR systems. For example, Wang et al. [21] employed a generator–discriminator architecture trained with target labels encoded as semantic guidance within a self-reconstruction framework.

In this work, we focus on the white-box setting and propose Cross-Gen, a parallel generator framework comprising dedicated image and text generators, designed to learn a general adversarial mapping that bypasses costly instance-specific optimization. Cross-Gen aims to approximate and invert the decision boundary of the target CMHR model—i.e., to generate inputs that cause the model to retrieve semantically dissimilar results. During training, we first collect cross-modal data pairs by querying the target model. The original samples are then fed into Cross-Gen to produce adversarial counterparts, which are evaluated by the target model to compute loss signals for end-to-end parameter updates. Specifically, we design three complementary loss functions: (1) an adversarial loss that minimizes the Hamming distance between hash codes of adversarial and target (dissimilar) samples; (2) a quantization loss that reduces the discrepancy between continuous outputs and binary hash codes; and (3) a reconstruction loss to ensure that the generated examples remain visually or semantically close to the originals. The entire framework is optimized using the Adam optimizer [23]. Once trained, Cross-Gen enables rapid generation of effective adversarial examples without per-instance optimization.

We evaluate our approach on two standard benchmarks—FLICKR-25K [24] and NUS-WIDE [25]—across multiple backbone architectures. The experimental results demonstrate that Cross-Gen generates high-quality adversarial examples significantly faster than iterative methods while achieving competitive attack performance.

In summary, our contributions are as follows:

• We propose Cross-Gen, an efficient parallel generator framework that rapidly learns the decision behavior of a target CMHR model and generates cross-modal adversarial examples at inference time without iterative optimization.
• We design a training strategy that acquires relevant cross-modal pairs via interaction with the target model and optimizes the generator using a composite objective combining adversarial, quantization, and reconstruction losses.
• We conduct comprehensive experiments on FLICKR-25K and NUS-WIDE with diverse model backbones, demonstrating that Cross-Gen outperforms iterative baselines in generation speed while maintaining strong attack efficacy.

The rest of this paper is organized as follows. Section 2 provides background on cross-modal retrieval systems. Section 3 formulates the attack problem against CMHR systems and presents our Cross-Gen solution. Section 4 describes the experimental setup—including the datasets, implementation details, results, and analysis. Section 5 reviews the related work on CMHR and adversarial attacks, covering both iterative- and generative-based approaches. Finally, Section 6 concludes the paper and outlines directions for future work.

2. Background

In this section, we detail the composition and training process of CMHR and systematically describe its adversarial attack formulation.

2.1. Cross-Modal Hash Retrieval System

Compared to traditional feature extraction methods—such as spectral hashing [26], color histograms [27], and bag-of-words (BoW) models [28]—recent deep neural network (DNN)-based cross-modal hash retrieval (CMHR) systems [3,4,5,29] have gained prominence due to their powerful data representation capabilities. When presented with cross-modal inputs such as images or text, a DNN-based CMHR system encodes them into feature vectors using dedicated neural networks and retrieves relevant results based on vector similarity. This paradigm eliminates the need for manual feature engineering; instead, optimal network parameters are learned through large-scale training, enabling high-performance retrieval in an end-to-end manner.

Deep hashing has become a key enabler of efficient CMHR, particularly when combined with approximate nearest neighbor (ANN) search [30], which offers low storage requirements and fast retrieval. ANN-based hashing maps high-dimensional features into compact binary codes by optimizing semantic similarity in the hash space. Through this process, well-trained DNNs learn to preserve semantic relationships across modalities—e.g., aligning image and text representations—thereby establishing cross-modal correlations. A typical DNN-based cross-modal retrieval framework is illustrated in Figure 1.

CMHR systems rely on a shared latent space to generate compact hash codes from heterogeneous modalities. The existing DNN-based approaches fall broadly into two categories: unsupervised frameworks [31,32] and supervised frameworks [33,34,35]. Supervised methods leverage labeled data to bridge the semantic gap and achieve higher retrieval accuracy, whereas unsupervised methods, although more flexible in label-scarce scenarios, require sophisticated designs to capture high-level semantic structures.

The training of a CMHR model typically follows three sequential steps:

• Embedding semantically aligned cross-modal data pairs into a joint vector space.
• Designing an optimization objective that enhances both semantic consistency and Hamming-space similarity between paired samples.
• Generating binary hash codes for all cross-modal data and storing them in a database to support efficient retrieval.

Given a cross-modal dataset with M data points $O={\left\{o_i\right\}}_i^M$ and $o_i=\{o_i^v,o_i^t\}$, where $o^v$ and $o^t$ represent image and text modality data with similar semantics, the goal of the deep cross-modal hash retrieval model is to construct two efficient functions ${\mathcal{F}}^{\ast }(·)$ and $\ast \in \{v,t\}$, which project the original data points of two modality types, $o^v$ and $o^t$, into shared hash space. The cross-modal data point can be represented as a binary-like representation $H^{\ast }$, which is the output of the target model. Hence, CMHR outputs the hash code $B^{\ast }$ by the sign function, and the above process can be formally described as

$$\begin{array}{c}\hfill H^{\ast }={\mathcal{F}}^{\ast }\left(o^{\ast }\right),B^{\ast }=\mathtt{sign}\left(H^{\ast }\right),H^{\ast }\in {[-1,1]}^K\end{array}$$

(1)

where K is the hash code length, and $\ast \in \{v,t\}$ denote the image and text data modality.

2.2. Problem Formulation

Suppose a well-trained CMHR model $\mathcal{F}(·)$ can efficiently transfer two similar semantic modal data into a related hash space. For instance, the CMHR model encourages that the hash code of the query data point $o_A^v$ is similar to the positive text data point $o_P^t$ with semantic content. On the contrary, promote the hash code of the negative text data point $o_N^t$ with $o_A^v$ dissimilar. Hence, the CMHR model aims to obtain the optimal model parameter ${\delta }_F$ to retain the semantic relationship of cross-modal data, which can be formulated as

$$\begin{array}{cc}\hfill \underset{{\delta }_F}{argmin}& \mathtt{Ham}({\mathcal{F}}^v\left(o_A^v\right),{\mathcal{F}}^t\left(o_P^t\right))-\mathtt{Ham}({\mathcal{F}}^v\left(o_A^v\right),{\mathcal{F}}^t\left(o_N^t\right))+\phi \hfill \end{array}$$

(2)

$\phi$ is the margin constant and generally set as ${\displaystyle \frac{K}{2}}$. $\mathtt{Ham}(·,·)$ is the Hamming distance function to measure the similarity between hash codes; i.e., $\mathtt{Ham}={\displaystyle \frac{1}{2}}(K-<V,T>)$, where $<·,·>$ is the dot product function. Moreover, the generated adversarial examples are bounded by the projection function, which aims to encourage visual imperceptibility. We formulated it as

$$\begin{array}{c}\hfill o_A^{\prime }=\mathtt{CLIP}\left(\mathtt{min}(o_A+ϵ,\mathtt{max}(\mathcal{G}\left(o_A\right),o_A-ϵ))\right)\end{array}$$

(3)

where $ϵ$ is the constant to ensure the imperceptibility of adversarial perturbations. Finally, given the image–text triplet instances $\{o_A^v,o_P^t,o_N^t\}$, we aim to create imperceptible adversarial perturbation to damage the relationship between the triplet cross-modal data, finally degrading the performance of the CMHR model.

3. Framework

In this section, we first introduce the framework overview of our Cross-Gen on adversarial attacks on CMHR, and then we present the details of the proposed Cross-Gen.

3.1. Overview

We present the complete framework of our proposed Cross-Gen in Figure 2. First, we construct triplet data pairs—comprising anchor, positive, and negative samples—by querying the target model. Next, we introduce two parallel generator networks: one for adversarial image generation and another for adversarial text generation. Specifically, the original image is fed into the adversarial image generator to produce a perturbed image, which is then passed through an image discriminator to generate hash codes. An analogous pipeline is employed for text: the original text input is processed by the adversarial text generator, and the resulting output is used to derive hash codes. Finally, we design three loss functions to guide training: reconstruction loss, adversarial loss, and quantization loss.

In the following sub-section, we present the steps of the proposed framework in detail.

3.2. Cross-Modal Data Adversarial Generator

As mentioned above, since the adversarial networks are subject to the label input for training and finally obtain attack effects (i.e., unsupervised scenarios), the challenge is to effectively promote more scenarios. This paper mainly explores the cross-modal generator network (Cross-Gen) in unsupervised scenarios (i.e., without data labels). We are inspired by the advanced research [36] to maximize the distance between original and adversarial examples in the hash space. Hence, the Cross-Gen network we proposed needs to achieve three goals:

• Adversarial and original input data have significant Hamming distance.
• In the training process, the network needs to generate compact hash codes.
• The generated adversarial examples are invisible from the original input examples.

The Cross-Gen framework is mainly composed of adversarial images, text generators, and discriminator (this paper sets the target model as the discriminator). Cross-Gen aims to learn the target model’s decision knowledge and conducts adversarial attacks by combining a two-stream cross-modal retrieval network (i.e., image and text generator) of images and text. Cross-Gen includes the adversarial generator ${\mathcal{G}}_{{\delta }_g}$ and discriminator ${\mathcal{D}}_{{\delta }_d}$. ${\delta }_g$ and ${\delta }_d$ are the network parameters of the adversarial generator and discriminator, respectively. Among them, we focus on optimizing the adversarial generator $\mathcal{G}$ with its parameter ${\delta }_g$ and setting the target model as the adversarial discriminator ${\mathcal{D}}_{{\delta }_d}$. Our designed framework aims to make the adversarial generator $\mathcal{G}$ learn the decision knowledge through the distribution of cross-modal data points returned by the target model. In many common scenarios, researchers cannot easily obtain the data labels. Hence, we attempt to acquire its decision knowledge by communicating with the target model. Specifically, by inputting cross-modal data $o^v\left(o^t\right)$, we obtain a cross-modal query set of size Q and the corresponding cross-modal database $S={o^v,o^t}^Q$ returned by the target model based on its internally designed hash space. Continuously, we can generate cross-modal triples for every query data point $o_A^v\left(o_A^t\right)$, which include positive data points $o_P^t\left(o_P^v\right)$ and negative data points $o_N^t\left(o_N^v\right)$, i.e., $\{o_A^v,o_P^t,o_N^t\},\{o_A^t,o_P^v,o_N^v\}$. The smaller Hamming distances of hash codes of data points $o_A^v\left(o_A^t\right)$ are positive cross-modal data points $o_P^t\left(o_P^v\right)$, and the remaining ones are negative cross-modal data points $o_N^t\left(o_N^v\right)$.

Following the work [37] to train the local model via a similar querying process, the well-trained adversarial generator $\mathcal{G}$ setting in the Cross-Gen will generate the effective adversarial cross-modal data point. Similarly, the Cross-Gen will inherit the decision knowledge of the target model and can easily destroy the relationship of the original hash space. Furthermore, we will optimize the parameter of ${\delta }_g$ by the adversarial generator $\mathcal{G}$ and initialize the discriminator ${\mathcal{D}}_{{\delta }_d}$ with the target model, which will be fixed while training.

As previously noted regarding the three-point goal of the model, to guarantee the attacking quality of adversarial examples generated by the adversarial generator $\mathcal{G}$, we set the objective function ${\mathcal{L}}_{gen}$ of adversarial generator $\mathcal{G}$, which can be defined as

$$\begin{array}{c}\hfill \underset{{\delta }_g}{argmin}{\mathcal{L}}_{gen}=\sum _{\{o_A,o_P,o_N\}\in S_{tri}}({\mathcal{J}}_{adv}+\beta {\mathcal{J}}_{qua}+\gamma {\mathcal{J}}_{rec})\end{array}$$

(4)

$\beta$ and $\gamma$ are the weighting constants to balance the term in Equation (4). $S_{tri}$ is the triplet dataset that is constructed based on the target model. The main goal of Equation (4) is to obtain the optimal parameter ${\delta }_g^{\ast }$ of adversarial generator $\mathcal{G}$. ${\mathcal{J}}_{adv}$ is the adversarial loss function, ${\mathcal{J}}_{qua}$ is the quantization loss function, and ${\mathcal{J}}_{rec}$ is the reconstruction loss function. Next, we will introduce the above three loss functions below, respectively.

Adversarial loss function ${\mathcal{J}}_{adv}$ aims to minimize the Hamming distance between the hash code of query data point $o_A$ and negative data point $o_N$, and maximize the Hamming distance between the hash code of query data point $o_A$ and positive data point $o_P$. Hence, we can formulate the ${\mathcal{J}}_{adv}$ as follows:

$$\begin{array}{c}\hfill {\mathcal{J}}_{adv}=\mathtt{Ham}(\mathcal{D}\left(\mathcal{G}\left(o_A\right)\right),H_N)-\mathtt{Ham}(\mathcal{D}\left(\mathcal{G}\left(o_A\right)\right),H_P)\end{array}$$

(5)

where $H_P=\mathtt{sign}\left(\mathcal{D}\left(o_P\right)\right)$ and $H_N=\mathtt{sign}\left(\mathcal{D}\left(o_N\right)\right)$. $\{o_A,o_P,o_N\}$ are the anchor data point, positive data point, and negative data point, respectively.

Reconstruction loss function ${\mathcal{J}}_{re}$ minimizes the difference between the adversarial cross-modal data point and the original, e.g., to minimize the pixel difference between the adversarial image and the original image. We adopt the general $L_2$-norm loss to measure the reconstruction error as follows:

$$\begin{array}{c}\hfill {\mathcal{J}}_{re}=\left|\right|o_A-\mathcal{G}\left(o_A\right){\left|\right|}_2^2\end{array}$$

(6)

where $\mathcal{G}\left(o_A\right)$ denote the adversarial cross modal data; $\left|\right|·{\left|\right|}_2$ is the $l_2$ norm function. In fact, many norm functions are available, such as the $l_{\infty }$ norm function and so on. Here, we construct a reconstruction loss function based on the $l_2$ norm function to ensure that the function is differentiable for updating the parameters of adversarial generator ${\mathcal{G}}_{{\delta }_g}$.

Quantization loss function ${\mathcal{J}}_{qua}$ reduces quantization errors between binary-like representations and binary hash codes, and we can formulate it as follows:

$$\begin{array}{c}\hfill {\mathcal{J}}_{qua}=\left|\right|H_A-B_A{\left|\right|}_2^2\end{array}$$

(7)

where $B_A=\mathtt{sign}\left(H_A\right)$ and $H_A$ is the binary-like representation of data point $o_A$; i.e., $H_A=\mathcal{D}\left(o_A\right)$. We create cross-modal adversarial examples to generate more compact hash codes in the target model to improve the attack effect by minimizing the quantization loss. We adopt the tanh-based quantization loss to approximate the sign function during backpropagation. No explicit orthogonality or balance constraints are imposed as the adversarial training itself encourages diverse and discriminative hash codes.

3.3. Implementation

We implement all experiments based on a 32-bit hash code and run on a server equipped with two GeForce RTX 2080Ti GPUs. We use the already constructed target cross-modal hashing model as the discriminator $\mathcal{D}$ and keep its parameters fixed throughout training. In cross-modal data retrieval, we consider two standard tasks: image-to-text retrieval (query: image; database: text) and text-to-image retrieval (query: text; database: image).

• For the image-to-text task, we build the adversarial image generator ${\mathcal{G}}_{\mathrm{img}}$ using a modified AlexNet or ResNet-34/50. All backbones are initialized with ImageNet pre-trained weights, and the final classification layer is replaced by a deconvolutional head that outputs perturbed images of the same size as the input (e.g., $224\times 224\times 3$). The generator takes a clean image $\mathbf{x}$ and produces an adversarial version $\widehat{\mathbf{x}}=\mathbf{x}+\delta$, where the perturbation $\delta$ is constrained by ${\parallel \delta \parallel }_{\infty }\le ϵ=8/255$ to ensure visual imperceptibility.
• For the text-to-image task, we construct the adversarial text generator ${\mathcal{G}}_{\mathrm{text}}$ based on a 2-layer LSTM [38] with 512 hidden units per layer. The input is the original bag-of-words (BoW) vector, and the output is a perturbed textual representation in the same embedding space. Since text perturbations operate in the continuous feature domain, no explicit norm constraint is applied.

To train the Cross-Gen framework, we construct cross-modal triplets as follows: for each query, we retrieve 10 positive pairs (smallest Hamming distance in hash space) and 10 negative pairs (largest Hamming distance) from the database to form $(q,p,n)$ triplets. The entire Cross-Gen network is optimized using the Adam optimizer [23] with ${\beta }_1=0.9$, ${\beta }_2=0.999$, and weight decay ${10}^{-4}$, with an initial learning rate of ${10}^{-2}$ (decayed by a factor of 0.1 every 20 epochs). We train for 50 epochs and only update the parameters ${\theta }_g$ of the generator $\mathcal{G}$ while keeping the discriminator $\mathcal{D}$ frozen. The hyperparameter $\beta$ in Equation (4) (balancing adversarial loss and perceptual loss) is set to $1\times {10}^{-1}$, chosen via grid search over $\{{10}^{-3},{10}^{-2},{10}^{-1},1\}$ on the validation set.

4. Experiments

In this section, we present the procedure used for the experiments of our proposed Cross-Gen. We exploit the widely used Mean Average Precision (MAP) and perceptibility (PER) evaluation metrics to measure the attack performance and perceptibility of the generated adversarial examples. Finally, we show the time consumption of Cross-Gen in generating adversarial examples to demonstrate its superiority.

4.1. Datasets

In our experiment, we evaluate our method on two popular datasets, NUS-WIDE [25] and FLICKR-25K [24]. One dataset is NUS-WIDE, which consists of 190,421 image–text pairs in 81 categories, divided into retrieval, training, and query parts with the numbers 188,321, 10,500, and 2100. FLICKR-25K contains 20,015 image–text pairs, with 38 classes. Similarly, it is divided into three sections: 18,015, 10,000, and 2000. To construct the stolen dataset for adversarial attack generation, we randomly sample 1000 query pairs from each dataset. This choice is motivated by two practical considerations:

• Statistical significance: 1000 samples provide sufficient diversity to cover multiple semantic categories while ensuring stable evaluation of attack success rate.
• Computational feasibility: Generating high-quality adversarial examples is resource-intensive, and 1000 queries represent a commonly adopted scale in recent adversarial cross-modal studies [36].

The entire stolen set (i.e., all 1000 sampled pairs) is then used to craft adversarial perturbations targeting the victim hashing model.

4.2. Evaluation Metrics

To evaluate the performance of our framework, we follow the work [36] to set the Mean Average Precision (MAP). Further, the Average Precision (AP) can be calculated as

$$\mathtt{AP}={\displaystyle \frac{1}{N}}\times \sum _{i=1}^N{\displaystyle \frac{i}{pos\left(i\right)}}$$

where N is the size of the retrieval list. $pos\left(i\right)$ indicates the position of the i-th correct retrieval result in the returned list. Hence, the Mean Average Precision (MAP) is the mean value of the average accuracy (AP) of multiple retrieval results and is widely used in cross-modal Hamming retrieval. The lower the MAP value, the stronger the attack performance. Moreover, we follow the work [9] to measure the cross-modal data difference between adversarial and query data by calculating PER as

$$\mathtt{PER}=\sqrt{{\displaystyle \frac{1}{M}}\left|\right|o-o^{\prime }{\left|\right|}_2^2}$$

where $\mathbf{o}$ denotes the original cross-modal data and ${\mathbf{o}}^{\prime }$ represents its adversarially perturbed version. Here, M is the total number of dimensions of the data (e.g., the total number of pixels for an image). A lower PER value indicates smaller distortion between the adversarial and original samples, implying better visual imperceptibility.

4.3. Compared Method

We deploy an advanced cross-modal hash retrieval method, CMHR [5], to train the target model cross-modal Hamming retrieval on both datasets. Specifically, we use AlexNet [39], ResNet-34, and ResNet-50 [18] as the backbone for image modality retrieval, and the text retrieval network is constructed through three fully connected layers. We set the training epoch as 200, and its data batch size is 32. To evaluate our performance, we follow the work [36] to set the AACH attack method for comparison, which is similar to ours. AACH demonstrates competitive performance relative to the advanced white-box attack methods [40]; hence, we use AACH as our comparison method. Specifically, we build an image retrieval module with vgg-f as the backbone and a text retrieval module with three fully connected layers for the substitute model. AACH runs 500 iterations to obtain the adversarial attack based on the target model.

4.4. Results

Table 1. MAP on the attack methods with the 32-bit hash code on two image datasets. The best results are marked in bold.

Tasks	Method	Iteration	FLICKER-25k
			AlexNet	ResNet-34	ResNet-50
I→T	Original		74.40%	80.24%	79.78%
	AACH	100	59.97%	64.15%	63.43%
		500	58.60%	63.02%	62.98%
	Ours	1	56.21%	60.58%	60.24%
T→I	Original		71.87%	77.58%	76.81%
	AACH	100	59.59%	67.47%	62.84%
		500	58.64%	66.68%	61.80%
	Ours	1	57.83%	65.93%	60.50%
Tasks	Method	Iteration	NUS-WIDE
			AlexNet	ResNet-34	ResNet-50
I→T	Original		56.25%	66.04%	65.97%
	AACH	100	33.93%	40.16%	41.47%
		500	33.52%	39.94%	40.52%
	Ours	1	30.13%	36.92%	39.43%
T→I	Original		54.08%	56.95%	59.14%
	AACH	100	36.71%	47.07%	47.91%
		500	36.55%	46.37%	46.73%
	Ours	1	34.33%	43.26%	45.58%

shows the MAP metrics for the two retrieval tasks, where “I→T” represents retrieving text through images and “T→I” represents retrieving images using text. Original represents the regular retrieval performance. We show the MAP values of AACH and our method on two datasets. The lower the MAP value, the better the attack performance. From Table 1, we can obtain the following conclusions: (1) For the AACH method, the attack performance will gradually improve as the number of iterations increases. For example, in the “I→T” task in FLICKR-25K, the MAP for 100 iterations is $59.97\%$, but the attack performance is only improved by $1.37\%$ when it reaches 500 iterations. This shows the flaw of the instance-specific iterative attack method that it needs to consume significant computing resources, and the effect of the attack is not significantly improved. (2) Comparing two retrieval tasks on the FLICKR-25K and NUS-WIDE datasets, our method outperforms AACH. For example, in the ResNet-34 target model, our method in the $I\to T$ task is higher than the existing methods on the two datasets by $2.54\%$ and $3.02\%$, respectively. Moreover, in both datasets, our method is higher by $0.75\%$ and $3.11\%$ in the “T→I” task. Finally, a desirable feature is that we can increase the speed of generating adversarial examples while improving the attack performance. We only need the trained generator for one forward pass to generate adversarial examples with a high attack rate. This is more efficient than running backpropagation multiple times on the target model.

4.5. Running Time and Imperceptibility

To better demonstrate the superiority of Cross-Gen in terms of generation speed and imperceptibility of adversarial examples,

Table 2. Running time (min) and perceptibility (PER) on the attack methods with the 32-bit hash code on two image datasets. The best results are marked in bold.

Tasks	Method	Iteration	FLICKER-25k		Method	Iteration	NUS-WIDE
			Time	Per			Time	Per
I→T	AACH	100	0.31	1.14	AACH	100	0.34	1.28
		500	1.27	0.93		500	1.44	1.19
	Ours	1	0.011	1.95	Ours	1	0.006	1.77
T→I	AACH	100	0.98	1.84	AACH	100	0.56	1.30
		500	3.81	1.67		500	2.03	1.26
	Ours	1	0.004	2.06	Ours	1	0.009	1.98

shows the 32-bit hash code performance on two datasets, PER, as well as the running speed of AACH and Cross-Gen. We can observe that Cross-Gen has the fastest generation speed. We observe that the PER value of AACH decreases gradually with the increase in the number of iterations, which shows that the iterative attack method can generate enough adversarial examples to be concealed. It is worth noting that Cross-Gen can also produce high-visual-quality and covert adversarial examples. We show examples in Figure 3 of two adversarial attack methods (i.e., iterative- and generative-based attack methods) and two categories of cross-modal retrieval tasks (i.e., image retrieval text and text retrieval image).

5. Related Work

In this section, we introduce the main work of CMHR and describe the necessity of conducting a generative-based adversarial attack method.

5.1. Deep Cross-Modal Hamming Retrieval

Cross-modal search engines enable users to retrieve semantically relevant content by uploading either images or text queries. For instance, Google processed 89.3 billion searches in a single month in 2022 (https://www.oberlo.com/blog/google-search-statistics; accessed on 23 October 2025), while Baidu reported a user base of 795 million in 2021 (https://www.chyxx.com/industry/202109/973941.html; accessed on 23 October 2025). Given the scale of such multimodal data, traditional feature extraction methods—such as stroke penetration, spectral analysis, and directional line elements—are inadequate for achieving efficient cross-modal retrieval. Consequently, recent research has shifted toward deep neural network-based approaches for cross-modal retrieval.

Studies have demonstrated the effectiveness of deep networks in mapping heterogeneous modalities into a shared hash space, significantly advancing retrieval performance [30,31,35]. Despite substantial progress in single-modality and basic cross-modal retrieval, challenges remain in designing robust cross-modal hash retrieval systems, particularly in aligning feature representations from disparate modalities into a unified semantic space.

To address this, Ref. [3] introduces a deep hybrid architecture that learns a joint visual–semantic embedding space for images and text. Ref. [4] generates compact binary hash codes to mitigate precision loss caused by relaxation during optimization, thereby improving retrieval accuracy. Ref. [5] proposes a novel deep hashing method that incorporates a pairwise focal loss based on an exponential distribution to refine the hashing objective. Ref. [41] constructs a joint matrix to integrate cross-modal semantic information and capture latent semantic similarities. More recently, Ref. [42] enhances cross-modal retrieval by modeling one-to-many embedding relationships through probabilistic cross-modal embeddings.

5.2. Adversarial Attacks on Deep Cross-Modal Hamming Retrieval

The successful application of deep neural networks (DNNs) has significantly enhanced the retrieval performance of cross-modal hash retrieval (CMHR) models. However, recent studies [6,7,8] have shown that well-crafted adversarial examples can effectively degrade DNN performance. CMHR systems, being DNN-based, inherit this vulnerability and are similarly susceptible to adversarial perturbations. Such attacks are typically categorized as white-box or black-box depending on whether the adversary has access to the target model’s parameters.

In a white-box setting, Ref. [40] generates adversarial examples for CMHR by minimizing intra-modal similarity and maximizing inter-modal similarity. Ref. [43] proposes a decoupling strategy that separates cross-modal data into modality-dependent and modality-invariant components to craft adversarial inputs. The most closely related work to ours [36], constructs adversarial examples through iterative optimization using a stolen triplet dataset. As previously discussed, however, such iterative methods are computationally expensive.

In contrast, this paper investigates generative-based adversarial attacks in unsupervised settings. To the best of our knowledge, the existing generative approaches [20,21] have been limited to single-modality data and have rarely been explored in the context of cross-modal retrieval.

6. Conclusions

Adversarial sample generation for multimodal data can significantly facilitate research on high-dimensional visualization tasks. Prior work typically employed iterative adversarial attacks to generate examples against target models; however, such approaches are computationally inefficient and often require access to target labels, which is impractical in real-world settings. In contrast, this paper proposes an adversarial optimization framework based on generator networks. To the best of our knowledge, we are the first to construct a triplet database by querying the target model to obtain semantically aligned data pairs and subsequently train a parallel architecture comprising adversarial image and text generators. Our well-trained generator can efficiently produce adversarial examples with strong performance guarantees. Moreover, the method is straightforward to deploy and provides a practical foundation for researchers to scale adversarial evaluation to large cross-modal retrieval datasets.

Meanwhile, the explorations in this study also offer useful insights for improving the use of multimodal data in comprehensive quality evaluation. The proposed methods help enhance the stability and reliability of multimodal data processing, thereby laying a foundation for further refinements in evaluation practices and the development of more robust analytical tools in related research contexts.

For future work, we aim to improve the accuracy of the acquired cross-modal data to better approximate decision boundaries and enhance the cross-domain transferability of the adversarial generator. Indeed, the current training process may be constrained by the use of randomly sampled cross-modal queries to collect semantic pairs from the target model—a limitation we identify as a key challenge for future investigation.