DA-GAN: Dual Attention Generative Adversarial Network for Cross-Modal Retrieval

Abstract

Cross-modal retrieval aims to search samples of one modality via queries of other modalities, which is a hot issue in the community of multimedia. However, two main challenges, i.e., heterogeneity gap and semantic interaction across different modalities, have not been solved efficaciously. Reducing the heterogeneous gap can improve the cross-modal similarity measurement. Meanwhile, modeling cross-modal semantic interaction can capture the semantic correlations more accurately. To this end, this paper presents a novel end-to-end framework, called Dual Attention Generative Adversarial Network (DA-GAN). This technique is an adversarial semantic representation model with a dual attention mechanism, i.e., intra-modal attention and inter-modal attention. Intra-modal attention is used to focus on the important semantic feature within a modality, while inter-modal attention is to explore the semantic interaction between different modalities and then represent the high-level semantic correlation more precisely. A dual adversarial learning strategy is designed to generate modality-invariant representations, which can reduce the cross-modal heterogeneity efficiently. The experiments on three commonly used benchmarks show the better performance of DA-GAN than these competitors.

1. Introduction

Cross-modal retrieval [1,2] is a hot issue in the field of multimedia [3]. As shown in Figure 1, it is aiming to find objects of one modality by queries of another modality. Recently, multimedia data [4] is growing exponentially, which is widely used in several scenarios, such as information retrieval, recommendation system [5], social network [6], etc. It makes this problem attract increasing interest by a growing number of researchers.

Figure 1. Illustration of cross-modal retrieval.

The main challenge of cross-modal retrieval is how to eliminate the heterogeneity between multimedia objects and how to bridge the semantic gap [7,8] by understanding cross-modal consistent semantic concepts. In the existing literature, the classic way to overcome this challenge is to construct a common latent subspace [9], in which the multimedia instances are represented in the same form and the semantic features can be aligned [10]. As a traditional approach, Canonical Correlation Analysis (CCA) [11] is adopted by many researches [12,13,14,15] to learn correlation between cross-modal instances with the same category label. Although these CCA-based methods are supported by classical statistical theory, they cannot represent the complex non-linear semantic correlation. To break this limitation, some non-linear extensions such as KCCA [11], RCCA [16], LPCCA [17], etc. have been proposed to enhance the cross-modal representation.

Thanks to the powerful representation ability of deep learning models [18,19,20,21], cross-modal semantic representation learning has been boosted significantly. For instance, several CCA-based approaches, e.g., deep CCA [22], DisDCCA [23], DCCAE [24], are extended by integrating CCA with DNNs. In recent years, attention mechanisms are exploited to support cross-modal feature learning, which is used to discover more significant semantic details from heterogeneous cross-modal representations. With the help of the attention techniques, high-level semantics can be selectively focused on during the learning, which augments the semantic modeling and reduces the influence of noise on representation learning [25,26,27,28,29].

Our method. To implement the above idea, this paper proposes a new approach, named Dual Attention Generative Adversarial Network (DA-GAN). This method combines adversarial learning, intra-modal, and inter-modal attention mechanism to improve cross-modal representation capability. Specifically, the inputs are divided into three groups: an image-text pair $⟨{\mathit{I}}_i,{\mathit{T}}_i,{\mathit{L}}_i⟩$ with category label ${\mathit{L}}_i$, a group of images and a group of texts with the same label ${\mathit{L}}_i$. For the generator, we utilize visual CNN and textual CNN to generate visual and textual feature vectors respectively. Then these feature vectors are fed into a two-channel intra-attention model (each channel per modality) to learn intra-modal high-level semantic feature representation with the help of a group of images and texts. At the top of this model, a two-channel encoder is implemented by DNN to learn modality-consistent representations, at the top of which an inter-attention model captures the important semantic features across different modalities. Besides, a two-channel decoder is to re-construct the feature representation for intra-modal adversarial learning. In addition, two types of discriminators are used to form a dual adversarial learning strategy to narrow the heterogeneity gap.

Contributions. This paper has three-fold contributions, which are listed as follows.

• We propose a novel Dual Attention Generative Adversarial Network (DA-GAN) for cross-modal retrieval, which is an integration of the adversarial learning method with a dual attention mechanism.
• To narrow semantic gap and learn high-level semantic features, a dual attention mechanism is designed to capture important semantic features from cross-modal instances in both intra-modal view and inter-modal view, which enhances abstract concepts learning across different modalities.
• To reduce heterogeneity gap, a cross-modal adversarial learning model is employed to learn consistent feature distribution via intra-modal and inter-modal adversarial loss.

Roadmap. The rest of this paper is organized as follows: related works on cross-modal retrieval, attention models, and generative adversarial network are introduced in Section 2. In Section 3, the problem definition and related concepts are proposed. In Section 4, we discuss the details of the proposed DA-GAN. Section 5 presents the experiments and the results. At last, Section 6 concludes this paper.

2. Related Work

2.1. Cross-Modal Retrieval

The main challenge of cross-modal retrieval [30,31,32,33] is to diminish the heterogeneity gap and semantics gap by learning a consistent semantic subspace, in which the cross-modal similarity can be directly measured. The existing methods include CCA-based methods, deep learning-based methods, and hashing-based methods. We review them in brief as follows.

CCA-Based Methods. Rasiwasia et al. [34] is the first to use CCA [11] for cross-modal correlation learning. After this work, several CCA-based methods are proposed to enhance cross-modal representation learning. For example, Sharma et al. [14] studied a supervised extension of CCA, which is a general multi-view and kernelizable feature learning method. Pereira et al. [12] proposed three CCA-based approaches, namely correlation matching (CM), semantic matching (SM), and semantic correlation matching (SCM). Gong et al. [13] presented a three-view CCA model in which the abstract semantic information is learned by a third view module to support semantic correlation learning. In [15], cluster-CCA method is developed to generate discriminant cross-modal representations.

Deep Learning-Based Methods. Recently, deep learning [18,19,35] techniques have made great progress, which empowers the multimedia analysis [36,37,38,39] and cross-modal representation [40,41]. To learn non-linear correlations from different data modalities, Andrew et al. [42] proposed to integrate deep neural networks into the CCA method. It is a two-channel model, each of which is for one modality. Benton et al. [22] introduced Deep Generalized Canonical Correlation Analysis (DGCCA) to learn non-linear transformations of arbitrarily many views. Gu et al. [43] designed generative processes so as to learn global and local features from cross-modal samples. Zhen et al. [44] introduced a method named Deep Supervised Cross-modal Retrieval (DSCMR) with a weight-sharing strategy to explore the cross-modal consistent relationship.

2.2. Attention Models

Attention mechanism [45] is widely applied in image caption [46], action recognition [47], fine-grained image classification [48], visual questing answering [49], cross-modal retrieval [25] and etc. For example, Wu et al. [50] introduced a deep attention-based spatially recursive model to consider spatial dependencies during feature learning. Sudhakaran et al. [51] proposed Long Short-Term Attention method to capture features from spatial relevant parts across the video frames.

For cross-modal task, Peng et al. [25] proposed a modality-specific cross-modal similarity approach by using a recurrent attention network. Wang et al. [52] designed a hierarchically aligned cross-modal attention (HACA) model to fuse both global and local temporal dynamics of different modalities. Xu et al. [26] developed a Cross-modal Attention with Semantic Consistency (CASC) method to realize local alignment and multi-label prediction for image-text matching. Liu et al. [53] proposed a cross-modal attention-guided erasing approach to comprehend and align cross-modal information for referring expression grounding. Huang et al. [54] used object-oriented encoders along with inter-modal and intra-modal attention networks to improve inter-modal dependencies. Fang et al. [27] introduced subjective attention-based multi-task auxiliary cross-modal fusion method to enhance the robustness and contextual awareness of image fusion.

2.3. Generative Adversarial Network

Generative adversarial network (GAN) is devised by Goodfellow et al. [55], which is a powerful generative model applied in various multimedia tasks [56]. Wang et al. [57] is the first to employ GAN to learn modality-invariant features to diminish cross-modal heterogeneity. Liu et al. [58] presented an adversarial learning-based image-text embedding method to make the distributions of different modalities consistent. Huang et al. [59] studied an adversarial-based transfer model to realize knowledge transfer, and generate modality-indiscriminative representations.

With the support of GAN, many works proposed effective cross-modal hashing methods to realize efficient retrieval in binary Hamming space [60,61]. For example, [62] presented a GAN-based semi-supervised cross-modal hashing approach is presented, which is to learn semantic correlations from unlabeled samples via a minimax game.

3. Preliminaries

In this section, the formal problem definition and related notions are presented. Then, we review the theory of generative adversarial networks, which is the base of the proposed technique. Table 1 summarizes the mathematical notations used in this paper.

Table 1. The mathematical notations.

Notation	Definition
$\mathcal{D}$	a multimedia dataset
${\mathit{I}}_i$	the i-th image sample
${\mathit{T}}_i$	the i-th text sample
${\mathit{L}}_i$	a label vector
Q	a cross-modal query
$\mathcal{R}$	the set of results
$\Phi (·)$	a non-linear mapping
$\mathcal{C}$	the set of semantic concepts
$\mathbf{\theta }$	the parameter vector of model
${\mathbf{\xi }}_I^{\left(i\right)}$	the i-th visual convolutional representation
${\mathbf{\xi }}_T^{\left(i\right)}$	the i-th textual convolutional representation
${\mathbf{\xi }}_I^{\prime \left(i\right)}$	the attention-aware representation of image ${\mathit{I}}_i$
${\mathbf{\xi }}_T^{\prime \left(i\right)}$	the attention-aware representation of text ${\mathit{T}}_i$
${\mathit{F}}_I^{\left(i\right)}$	the cross-modal common semantic representation of image ${\mathit{I}}_i$
${\mathit{F}}_T^{\left(i\right)}$	the cross-modal common semantic representation of text ${\mathit{T}}_i$
${\mathit{F}}_I^{\prime \left(i\right)}$	the attention-aware cross-modal common semantic representation of image ${\mathit{I}}_i$
${\mathit{F}}_T^{\prime \left(i\right)}$	the attention-aware cross-modal common semantic representation of text ${\mathit{T}}_i$
$\mathit{h}$	a hidden vector
$\mathit{K}$	a convolutional kernel
$\mathit{M}$	a semantic correlation matrix
$\mathit{A}$	an attention map
$\mathit{U}$	a cross-modal semantic correlation matrix
$\mathbf{\zeta }{\left(i\right)}_I$	a reconstructed representation of i-th image
$\mathbf{\zeta }{\left(i\right)}_T$	a reconstructed representation of i-th text

3.1. Problem Definition

This work considers two common modalities: image and text. Let $\mathcal{D}={\left\{⟨{\mathit{I}}_i,{\mathit{T}}_i,{\mathit{L}}_i⟩\right\}}_{i=1}^n$ be a multimedia dataset that contains n image-text pairs, where ${\mathit{I}}_i\in {\mathbb{R}}^{{\lambda }_I}$ and ${\mathit{T}}_i\in {\mathbb{R}}^{{\lambda }_T}$ represent i-th image sample and text sample in their original space respectively, ${\lambda }_I$ and ${\lambda }_T$ are the dimensions of image and text original space. Each pair is assigned a semantic label vector that is denoted as ${\mathit{L}}_i=(L_i^{\left(1\right)},L_i^{\left(2\right)},\cdots ,L_i^{\left({\lambda }_L\right)})\in {\mathbb{R}}^{{\lambda }_L}$, where ${\lambda }_L$ is the number of semantic categories in $\mathcal{D}$. If ${\mathit{I}}_i$ and ${\mathit{T}}_i$ belong to the same semantic category, then $L_i^{\left(c\right)}=1$; otherwise $L_i^{\left(j\right)}=0$. Cross-modal retrieval aims to search multimedia instances, which are different from the modality of the query Q but similar enough to Q. If the query is an image, denoted as $Q_I$, we call this type of cross-modal as image-to-text (I2T) retrieval; otherwise text-to-image (T2I) retrieval. In the following the definition of I2T and T2I retrieval are formulated.

Definition 1.

Cross-Modal Retrieval.Given a multimedia dataset $\mathcal{D}={\left\{⟨{\mathit{I}}_i,{\mathit{T}}_i,{\mathit{L}}_i⟩\right\}}_{i=1}^n$ and two queries $Q_I$ and $Q_T$. The I2T retrieval is to return a set of results

$$\begin{array}{c}\hfill {\mathcal{R}}_{I2T}={\left\{{\mathit{T}}_j|Sim\left({\mathit{T}}_i,Q_I\right)\ge Sim\left({\mathit{T}}^{\prime },Q_I\right),{\mathit{T}}_i\in \mathcal{D},{\mathit{T}}^{\prime }\in \mathcal{D}\setminus {\mathcal{R}}_{I2T}\right\}}_{j=1}^k,\end{array}$$

(1)

where $Sim(·)$ denotes the similarity function, k is the number of results.

Apparently, Definition 1 indicates that the key problem of cross-modal retrieval is to realize the function $Sim(·)$. However, due to the heterogeneity gap and the semantic gap, it is hard to measure the semantic similarity between instances of different modalities in their original space. Therefore, two non-linear mappings ${\Phi }_I(·):{\mathbb{R}}^{{\lambda }_I}\mapsto {\mathbb{R}}^{{\lambda }_C}$ and ${\Phi }_T(·):{\mathbb{R}}^{{\lambda }_T}\mapsto {\mathbb{R}}^{{\lambda }_C}$ need to be learned, which is to transform images and texts into a ${\lambda }_C$-dimensional common semantic subspace. Thus, the heterogeneity of different modalities can be diminished and the cross-modal representations can be described by a set of semantic concepts $\mathcal{C}={\left\{C\right\}}_{l=1}^{{\lambda }_C}$, As a result, the cross-modal similarity can be measured accurately by the following function.

Definition 2.

Cross-Modal Similarity Function.Given a multimedia dataset $\mathcal{D}$, an image $\mathit{I}\in \mathcal{D}$ and a text $\mathit{T}\in \mathcal{D}$, the cross-modal similarity between $\mathit{I}$ and $\mathit{T}$ is defined as

$$\begin{array}{ccc}\hfill Sim& (\mathit{I},\mathit{T})=\hfill & \hfill \frac{{\sum }_{i=1}^{{\lambda }_C}\left({\Phi }_I{\left(\mathit{I}\right)}^{\left(i\right)}\times {\Phi }_I{\left(\mathit{T}\right)}^{\left(i\right)}\right)}{\sqrt{{\sum }_{i=1}^{{\lambda }_C}{\left({\Phi }_I{\left(\mathit{I}\right)}^{\left(i\right)}\right)}^2}\times \sqrt{{\sum }_{i=1}^{{\lambda }_C}{\left({\Phi }_I{\left(\mathit{T}\right)}^{\left(i\right)}\right)}^2}},\end{array}$$

(2)

where ${\Phi }_I\left(\mathit{I}\right)$ and ${\Phi }_I\left(\mathit{T}\right)$ denote the cross-modal representations in the common semantic subspace. ${\Phi }_I{\left(\mathit{I}\right)}^{\left(i\right)}$ and ${\Phi }_I{\left(\mathit{T}\right)}^{\left(i\right)}$ are the i-th element of representation vectors, respectively.

To learn these two non-linear mappings, we propose a deep architecture by using adversarial learning, which generates modality-invariant representations from multi-modality data and realizes cross-modal semantic augmentation via a dual attention mechanism.

3.2. Review of Generative Adversarial Netw

As a powerful technique, generative adversarial networks (GANs) [55] have be utilized in many multimedia tasks, such as image synthesis, video generation, motion generation, face aging, etc. It consists of two components: a generator $G(·;{\mathbf{\theta }}_G)$ and a discriminator $D(·;{\mathbf{\theta }}_D)$, where ${\mathbf{\theta }}_G$ and ${\mathbf{\theta }}_D$ are the model parameter vectors. During the training, the generator $G(·;{\mathbf{\theta }}_G)$ tries to make the synthetic image more realistic to fool the discriminator $D(·;{\mathbf{\theta }}_D)$. The discriminator $D(·;{\mathbf{\theta }}_D)$ makes its efforts to distinguish the fake samples from real samples. In other words, $G(·;{\mathbf{\theta }}_G)$ and $D(·;{\mathbf{\theta }}_D)$ are diametrically against to each other.

Specifically, let $\mathit{I}$ be a real image sample obey natural data distribution $P_{data}\left(\mathit{I}\right)$, $\mathit{z}\in {\mathbb{R}}^{{\lambda }_z}$ be a random noise vector generated from distribution $P_z\left(\mathit{z}\right)$. After fed into the generator $G(·;{\mathbf{\theta }}_G)$, $\mathit{z}$ is transformed into a synthetic sample $G(\mathit{z};{\mathbf{\theta }}_G)$ that obeys the generative distribution $P_G$. The discriminator receives the real sample $\mathit{I}$ and the synthetic sample $G(\mathit{z};{\mathbf{\theta }}_G)$ as inputs, and outputs the discriminant result $D(G(\mathit{z};{\mathbf{\theta }}_G);{\mathbf{\theta }}_D)$, a probability that $G(\mathit{z};{\mathbf{\theta }}_G)$ is produced by the generator. This adversarial process can be formulated as

$$\begin{array}{cc}\hfill arg\underset{G\left(·;{\mathbf{\theta }}_G\right)}{min}& \underset{D\left(·;{\mathbf{\theta }}_D\right)}{max}{\mathcal{L}}_{GAN}\left(G(·;{\mathbf{\theta }}_G),D(·;{\mathbf{\theta }}_D)\right)=\hfill \\ & {\mathbb{E}}_{\mathit{I}\sim P_{data}\left(\mathit{I}\right)}\left[logD\left(\mathit{I};{\mathbf{\theta }}_D\right)\right]+\hfill \\ & {\mathbb{E}}_{\mathit{z}\sim P_{\mathit{z}}\left(\mathit{z}\right)}\left[log\left(1-D\left(G\left(\mathit{z};{\mathbf{\theta }}_G\right);{\mathbf{\theta }}_D\right)\right)\right],\hfill \end{array}$$

(3)

where ${\mathbb{E}}_{\mathit{I}\sim P_{data}\left(\mathit{I}\right)}\left[·\right]$ and ${\mathbb{E}}_{\mathit{z}\sim P_{\mathit{z}}\left(\mathit{z}\right)}\left[·\right]$ denote mathematical expectations:

$$\begin{array}{c}\hfill {\mathbb{E}}_{\mathit{I}\sim P_{data}\left(\mathit{I}\right)}\left[logD\left(\mathit{I};{\mathbf{\theta }}_D\right)\right]={\int }_{\mathit{I}}{P}_{data}\left(\mathit{I}\right)log\left(D\left(\mathit{I};{\mathbf{\theta }}_D\right)\right)d\mathit{I},\end{array}$$

$$\begin{array}{c}\hfill {\mathbb{E}}_{\mathit{z}\sim P_{\mathit{z}}\left(\mathit{z}\right)}\left[log\left(1-D\left(G\left(\mathit{z};{\mathbf{\theta }}_G\right);{\mathbf{\theta }}_D\right)\right)\right]={\int }_{\mathit{z}}{P}_{\mathit{z}}\left(\mathit{z}\right)log\left(1-D\left(G\left(\mathit{z};{\mathbf{\theta }}_G\right);{\mathbf{\theta }}_D\right)\right)d\mathit{z}.\end{array}$$

In the training process, the generator $G(·;{\mathbf{\theta }}_G)$, on one hand, synthesizes images as authentic as possible to fool the discriminator $D(·;{\mathbf{\theta }}_D)$ by minimizing the loss function. On the other hand, the discriminator $(·;{\mathbf{\theta }}_D)$ does its utmost to recognize the fake samples from real samples by maximizing the loss function, shown as follows:

$$\begin{array}{c}\hfill arg\underset{G\left(·;{\mathbf{\theta }}_G\right)}{min}{\mathcal{L}}_{GAN}\left(G\left(·;{\mathbf{\theta }}_G\right),D\left(·;{\mathbf{\theta }}_D\right)\right)={\int }_{\mathit{I}}{P}_{data}\left(\mathit{I}\right)log\left(D\left(\mathit{I};{\mathbf{\theta }}_D\right)\right)d\mathit{I},\end{array}$$

$$\begin{array}{c}\hfill arg\underset{D\left(·;{\mathbf{\theta }}_D\right)}{max}{\mathcal{L}}_{GAN}\left(G\left(·;{\mathbf{\theta }}_G\right),D\left(·;{\mathbf{\theta }}_D\right)\right)={\int }_{\mathit{z}}{P}_{\mathit{z}}\left(\mathit{z}\right)log\left(1-D\left(G\left(\mathit{z};{\mathbf{\theta }}_G\right);{\mathbf{\theta }}_D\right)\right)d\mathit{z}.\end{array}$$

4. Methodology

In this section, we discuss the proposed Dual Attention Generative Adversarial Network (DA-GAN). This method is to learn cross-modal non-linear mappings in an adversarial manner, in which a dual attention mechanism is developed to mine important semantic details to bridge heterogeneity gap and semantic gap. In Section 4.1 we introduce the overview of DA-GAN, and in Section 4.2 and Section 4.3 discuss the multi-modal feature learning and adversarial learning with dual attention mechanism. The implementation details are described in Section 4.4.

4.1. Overview of DA-GAN

Figure 2 illustrates the framework of DA-GAN. It consists of three layers: the input layer, generation layer, and discrimination layer.

Figure 2. The framework of DA-GAN. The input layer feeds two types of samples into the generation layer: (1) the image-text sample pairs ${\left\{⟨{\mathit{I}}_i,{\mathit{T}}_i,{\mathit{L}}_i⟩\right\}}_{i=1}^n$ and (2) for each pair, a group of images ${\left\{⟨{\mathit{I}}_j,{\mathit{L}}_i⟩\right\}}_{j=1}^m$ and a group of texts ${\left\{⟨{\mathit{T}}_j,{\mathit{L}}_i⟩\right\}}_{j=1}^m$ that have the same semantic label are selected from multimedia dataset. The generation layer consists of a two-channel CNN-based multi-modal feature learning model, a two-channel intra-attention model, a two-channel encoder, a two-channel decoder, as well as an inter-attention model. The discrimination layer includes: a two-channel intra-modal discriminator to discriminate the convolutional feature representation and common semantic representation, a two-channel semantic discriminator and an inter-modal discriminator to distinguish the common semantic representations of different modalities.

The Input Layer. The input layer is responsible for training data preparation. To capture more semantic knowledge, two types of samples are selected from the training dataset. The one type is the image-text sample pairs ${\left\{⟨{\mathit{I}}_i,{\mathit{T}}_i,{\mathit{L}}_i⟩\right\}}_{i=1}^n$, and the other type is a group of images ${\left\{⟨{\mathit{I}}_j,{\mathit{L}}_i⟩\right\}}_{j=1}^m$ and a group of texts ${\left\{⟨{\mathit{T}}_j,{\mathit{L}}_i⟩\right\}}_{j=1}^m$ that have the same semantic label. They are fed into the generation layer to produce the common semantic representations.

The Generation Layer. The generation layer is a deep cross-modal generative model with intra-modal attention (intra-attention) and inter-modal attention (inter-attention). Specifically, the visual and textual features are extracted by a two-channel multi-modal feature learning model $\mathit{ImgCNN}(·;{\mathbf{\theta }}_{Fea}^I)$ and $\mathit{TxtCNN}(·;{\mathbf{\theta }}_{Fea}^T)$, one channel per modality, where ${\mathbf{\theta }}_{Fea}^I$ and ${\mathbf{\theta }}_{Fea}^T$ denote parameter vectors. For image modality, it consists of several layers of convolutional networks, which generates visual convolutional representations ${\mathbf{\xi }}_I^{\left(i\right)}$ and ${\mathbf{\xi }}_{Is}^{\left(i\right)}$ of inputs ${\mathit{I}}_i$ and ${\left\{⟨{\mathit{I}}_j⟩\right\}}_{j=1}^m$, respectively. For text modality, the feature learning model consists of a word2vec model to produce word embeddings, and a combination of a bidirectional LSTM [63] (BiLSTM) and a textual CNN to output textual convolutional representations ${\mathbf{\xi }}_T^{\left(i\right)}$ and ${\mathbf{\xi }}_{Ts}^{\left(i\right)}$. A two-channel intra-attention modal is proposed to capture the important semantic details from each category (each channel per modality). It receives the convolutional representation pair $⟨{\mathbf{\xi }}_I^{\left(i\right)},{\overline{\mathbf{\xi }}}_I^{\left(i\right)}⟩$ and $⟨{\mathbf{\xi }}_T^{\left(i\right)},{\overline{\mathbf{\xi }}}_T^{\left(i\right)}⟩$ and generates the intra-attention masks for both image and text, then outputs the attention-aware representations ${\mathbf{\xi }}_I^{\prime \left(i\right)}$ and ${\mathbf{\xi }}_T^{\prime \left(i\right)}$. To narrow the heterogeneity gap, a two-channel encoder with weight-sharing strategy over two branches is used following the intra-attention model. Under weight sharing constraint, it generates ${\lambda }_C$-dimensional visual and textual representations ${\mathit{F}}_I^{\left(i\right)}\in {\mathbb{R}}^{{\lambda }_C}$ and ${\mathit{F}}_T^{\left(i\right)}\in {\mathbb{R}}^{{\lambda }_C}$, which are fed into an inter-attention model to realize cross-modal semantic feature augmentation. Besides, a two-channel decoder (one channel per modality) is employed to reconstruct the image and text representations ${\mathbf{\zeta }}_I^{\left(i\right)}$ and ${\mathbf{\zeta }}_T^{\left(i\right)}$ from distribution-consistent representations ${\mathit{F}}_I^{\left(i\right)}$ and ${\mathit{F}}_T^{\left(i\right)}$.

The Discrimination Layer. In discrimination layer, there are three types of discriminators, i.e., semantic category discriminator $D_S(·;{\mathbf{\theta }}_S)$, intra-modal discriminator $D_{Intra}(·;{\mathbf{\theta }}_{Intra})$ and inter-modal discriminator $D_{Inter}(·;{\mathbf{\theta }}_{Inter})$, to conduct semantic discrimination, intra-modality and inter-modality discrimination. $D_S(·;{\mathbf{\theta }}_S)$ and $D_{Intra}(·;{\mathbf{\theta }}_{Intra})$ are two-channel models (one channel per modality). The Former is to predict the semantic labels of convolutional representations ${\mathbf{\xi }}_I^{\prime \left(i\right)}$ and ${\mathbf{\xi }}_T^{\prime \left(i\right)}$, as well as common semantic representations ${\mathit{F}}_I^{\prime \left(i\right)}$ and ${\mathit{F}}_T^{\prime \left(i\right)}$ by semantic discrimination loss. The latter is to distinguish the reconstructed representations ${\mathbf{\zeta }}_I^{\left(i\right)}$ and ${\mathbf{\zeta }}_T^{\left(i\right)}$ from convolutional representations ${\mathbf{\xi }}_I^{\prime \left(i\right)}$ and ${\mathbf{\xi }}_T^{\prime \left(i\right)}$ via intra-modality discrimination loss. The inter-modal discriminator $D_{Inter}(·;{\mathbf{\theta }}_{Inter})$ aims to discriminate the outputs of inter-attention model, i.e., ${\mathit{F}}_I^{\prime \left(i\right)}$ and ${\mathit{F}}_T^{\prime \left(i\right)}$ from image and text modality.

4.2. Multi-Modal Feature Learning

The multi-modal feature learning model consists of two channels: visual feature learning model $\mathit{ImgCNN}(·;{\mathbf{\theta }}_{Fea}^I)$ and textual feature learning model $\mathit{TxtCNN}(·;{\mathbf{\theta }}_{Fea}^T)$ to generate convolutional representations of image and text samples.

4.2.1. Visual Feature Learning

The visual feature learning model is to project visual samples from original data space into convolutional feature space. Formally, ${\mathbf{\xi }}_I^{\left(i\right)}=\mathit{ImgCNN}({\mathit{I}}_i;{\mathbf{\theta }}_{Fea}^I)$, ${\mathbf{\xi }}_I^{\left(i\right)}=({\xi }_I^{\left(i\right)\left(1\right)},{\xi }_I^{\left(i\right)\left(2\right)},\cdots$, ${\xi }_I^{\left(i\right)\left(\gamma \right)})\in {\mathbb{R}}^{\gamma }$. We use a pre-trained AlexNet [64] to implement visual feature learning. We refine this model on the training dataset via squared loss. Suppose the training set $\mathcal{D}={\left\{⟨{\mathit{I}}_i,{\mathit{T}}_i,{\mathit{L}}_i⟩\right\}}_{i=1}^n$ contains n image samples, the ground-true probability vector of i-th sample is denoted as ${\mathit{p}}^{\prime }\left({\mathit{I}}_i\right)={\mathit{L}}_i/\left|\right|{\mathit{L}}_i{\left|\right|}_1$, where $\left|\right|·{\left|\right|}_1$ is the L1 norm. The predictive probability vector is $\mathit{p}\left({\mathit{I}}_i\right)=(p_i^{\left(1\right)},p_i^{\left(2\right)},\cdots ,p_i^{\left({\lambda }_L\right)})$. Thus, the objective function is

$$arg\underset{{\mathbf{\theta }}_{Fea}^I}{min}{\mathcal{L}}_{Fine}\left({\mathbf{\theta }}_{Fea}^I\right)=\frac{1}{n}\sum _{i=1}^n\sum _{j=1}^{{\lambda }_L}{\left(p_i^{\left(j\right)}-p_i^{\prime \left(j\right)}\right)}^2.$$

(4)

4.2.2. Textual Feature Learning

The textual feature learning model is a combination of a Word2Vec model, a BiLSTM model and a textual convolutional network [65]. It generates textual convolutional representations, i.e., ${\mathbf{\xi }}_T^{\left(i\right)}=\mathit{TxtCNN}({\mathit{T}}_i;{\mathbf{\theta }}_{Fea}^T)$, ${\mathbf{\xi }}_T^{\left(i\right)}=({\xi }_T^{\left(i\right)\left(1\right)},{\xi }_T^{\left(i\right)\left(2\right)},\cdots ,{\xi }_T^{\left(i\right)\left(\gamma \right)})\in {\mathbb{R}}^{\gamma }$. More concretely, a Word2Vec model $\mathit{Word}\mathit{2}\mathit{Vec}(·;{\mathbf{\theta }}_{w2v})$ generates $ϵ$-dimensional word embedding ${\mathit{w}}_j\in {\mathbb{R}}^{ϵ}$ for each word in ${\mathit{T}}_i$. Suppose the length of each text sample ${\mathit{T}}_i\in \mathcal{D}$ is l (padded if necessary), then the embedding of it is denoted as

$$\begin{array}{cc}\hfill {\mathit{E}}_i^{(1,l)}& =\mathit{Word}\mathit{2}\mathit{Vec}\left({\mathit{T}}_i;{\mathbf{\theta }}_{w2v}\right)={\mathit{w}}_1\bowtie {\mathit{w}}_2\bowtie \cdots \bowtie {\mathit{w}}_l,\hfill \end{array}$$

(5)

where ⋈ denotes vector concatenation operator. The word embeddings are fed into a BiLSTM model to encode the contextual semantic information from both the previous and future context on forward and reverse direction, $\mathit{h}\left(t\right)=\mathit{BiLSTM}({\mathit{E}}^{(1,l)};{\mathbf{\theta }}_{Bi})$, $\mathit{h}\left(t\right)\in {\mathbb{R}}^{{\lambda }_B}$.

The following textual CNN model receives $\mathit{h}\left(t\right)$ at time t and encode local semantic information. Let the convolutional kernels be ${\left\{{\mathit{K}}_j\right\}}_{j=1}^{\kappa }$ with size ${\lambda }_B\times m$, for the d-th window of the input vector covered by j-th kernel ${\mathit{K}}_j$, namely $\left(\mathit{h}\right(t),\mathit{h}(t+1),\cdots ,\mathit{t}(t+m-1\left)\right)$, the value of convolution is:

$${\widehat{h}}_j^{\left(d\right)}\left(t\right)=\sigma \left(\left(\sum _{i=0}^{m-1}\mathit{h}(t+i-1)*{\mathit{K}}_j\right)+\beta \right),$$

(6)

where $\sigma (·):\mathbb{R}\mapsto \mathbb{R}$ denotes an activation function, * denotes convolutional operator, and $\beta$ is a bias term. For j-th kernel, the result of the convolution at each window on vector $\mathit{h}\left(t\right)$ is

$${\widehat{\mathit{h}}}_j\left(t\right)=\left({\widehat{h}}_j^{\left(1\right)}\left(t\right),{\widehat{h}}_j^{\left(2\right)}\left(t\right),\cdots ,{\widehat{h}}_j^{(l-d+1)}\left(t\right)\right).$$

(7)

Then, a max pooling operation is conducted on the all the vectors $({\widehat{\mathit{h}}}_1\left(t\right),{\widehat{\mathit{h}}}_2\left(t\right),\cdots ,{\widehat{\mathit{h}}}_{\kappa }\left(t\right))$ as follows:

$$\begin{array}{cc}\hfill \left({\dot{h}}_1\left(t\right),{\dot{h}}_2\left(t\right),\cdots ,{\dot{h}}_{\kappa }\left(t\right)\right)& =\mathit{MaxPooling}\left({\widehat{\mathit{h}}}_1\left(t\right),{\widehat{\mathit{h}}}_2\left(t\right),\cdots ,{\widehat{\mathit{h}}}_{\kappa }\left(t\right)\right)\hfill \\ \hfill & =\left(max\left({\widehat{\mathit{h}}}_1\left(t\right)\right),max\left({\widehat{\mathit{h}}}_2\left(t\right)\right),\cdots ,max\left({\widehat{\mathit{h}}}_{\kappa }\left(t\right)\right)\right),\hfill \end{array}$$

(8)

where $max(·)$ is the function to choose the maximal element of a vector. This $\kappa$-dimensional vector is fed into the last FC layer with drop-out to restraint over-fitting:

$$\begin{array}{ccc}\hfill & \left({\xi }_T^{\left(i\right)\left(1\right)},{\xi }_T^{\left(i\right)\left(2\right)},\cdots ,{\xi }_T^{\left(i\right)\left(\gamma \right)}\right)\hfill & \hfill ={\mathit{W}}_{fc}\times \left({\dot{h}}_1\left(t\right),{\dot{h}}_2\left(t\right),\cdots ,{\dot{h}}_{\kappa }\left(t\right)\right)\odot \Omega +\mathbf{\beta },\end{array}$$

(9)

where ${\mathit{W}}_{fs}$ is the parameters of FC layer, $\mathbf{\beta }$ is the bias term, ⊙ denotes element-wise multiplication operator, and $\Omega$ is a mask to realize drop-out.

4.2.3. Semantic Grouping of Samples

As described in Section 4.1, for each pair $⟨{\mathit{I}}_i,{\mathit{T}}_i,{\mathit{L}}_i⟩$, the input layer produces a group of images and a group of texts, which belong to the same semantic category to $⟨{\mathit{I}}_i,{\mathit{T}}_i,{\mathit{L}}_i⟩$. In other words, it randomly samples $\alpha$ images ${\left\{⟨{\mathit{I}}_j,{\mathit{L}}_i⟩\right\}}_{j=1}^{\alpha }$ and texts ${\left\{⟨{\mathit{T}}_j,{\mathit{L}}_i⟩\right\}}_{j=1}^{\alpha }$ according to the semantic label ${\mathit{L}}_i$ from training set $\mathcal{D}$. After that, these two groups are fed into visual and textual feature learning model, respectively, i.e.,

$${\left\{{\mathbf{\xi }}_I^{\left(i\right)\left(j\right)}\right\}}_{j=1}^{\alpha }=\mathit{ImgCNN}\left({\left\{{\mathit{I}}_j\right\}}_{j=1}^{\alpha };{\mathbf{\theta }}_{Fea}^I\right),$$

(10)

$${\left\{{\mathbf{\xi }}_T^{\left(i\right)\left(j\right)}\right\}}_{j=1}^{\alpha }=\mathit{TxtCNN}\left({\left\{{\mathit{T}}_j\right\}}_{j=1}^{\alpha };{\mathbf{\theta }}_{Fea}^T\right).$$

(11)

The final convolutional representations of the two groups are the average of each representations, i.e.,

$${\overline{\mathbf{\xi }}}_I^{\left(i\right)}=\frac{1}{\alpha }\sum _{j=1}^{\alpha }{\mathbf{\xi }}_I^{\left(i\right)\left(j\right)},{\overline{\mathbf{\xi }}}_T^{\left(i\right)}=\frac{1}{\alpha }\sum _{j=1}^{\alpha }{\mathbf{\xi }}_T^{\left(i\right)\left(j\right)}.$$

(12)

In this work, ${\overline{\mathbf{\xi }}}_I^{\left(i\right)}$ and ${\overline{\mathbf{\xi }}}_T^{\left(i\right)}$ are used to represent the common semantic features of the category labeled by ${\mathit{L}}_i$.

4.3. Adversarial Learning with Dual Attention

In DA-GAN, a novel dual attention mechanism is proposed to learn more discriminative representations via modeling intra-modal and inter-modal semantic correlations by two attention models: intra-attention and inter-attention. Besides, three types of discriminative models are integrated into the framework to achieve modality-invariant representations in an adversarial manner.

4.3.1. Intra-Attention

Intra-Attention model aims to learn more discriminative feature representations by modeling the intra-modal semantic correlations. In our method, it is a two-channel model, one channel per modality. Since the images and texts are processed in the same way, we take the image intra-attention as an example. For the feature representation pair $⟨{\mathbf{\xi }}_I^{\left(i\right)},{\overline{\mathbf{\xi }}}_I^{\left(i\right)}⟩$, ${\mathbf{\xi }}_I^{\left(i\right)},{\overline{\mathbf{\xi }}}_I^{\left(i\right)}\in {\mathbb{R}}^{x\times y\times d}$, $x,y,d$ denote the weight, height and depth of the tensors. For convenience of discussion, we reshape these two tensors as ${\mathbf{\xi }}_I^{\left(i\right)}=({\mathbf{\xi }}_I^{\left(i\right)\left(1\right)},{\mathbf{\xi }}_I^{\left(i\right)\left(2\right)},\cdots ,{\mathbf{\xi }}_I^{\left(i\right)\left(p\right)})$ and ${\overline{\mathbf{\xi }}}_I^{\left(i\right)}=({\overline{\mathbf{\xi }}}_I^{\left(i\right)\left(1\right)},{\overline{\mathbf{\xi }}}_I^{\left(i\right)\left(2\right)},\cdots ,{\overline{\mathbf{\xi }}}_I^{\left(i\right)\left(p\right)})$, where $p=x\times y$ is the number of spatial positions of each tensor. The semantic correlation between ${\mathbf{\xi }}_I^{\left(i\right)}$ and ${\overline{\mathbf{\xi }}}_I^{\left(i\right)}$ can be modeled by the semantic correlation matrix ${\mathit{M}}_I^{\left(i\right)}\in {\mathbb{R}}^{p\times p}$:

$$\begin{array}{cc}\hfill & {\mathit{M}}_I^{\left(i\right)}={\left(\begin{array}{cccc}{\mathit{M}}_I^{\left(1\right)\left(1\right)}& {\mathit{M}}_I^{\left(1\right)\left(2\right)}& \cdots & {\mathit{M}}_I^{\left(1\right)\left(p\right)}\\ {\mathit{M}}_I^{\left(2\right)\left(1\right)}& {\mathit{M}}_I^{\left(2\right)\left(2\right)}& \cdots & {\mathit{M}}_I^{\left(2\right)\left(p\right)}\\ ⋮& ⋮& \ddots & ⋮\\ {\mathit{M}}_I^{\left(p\right)\left(1\right)}& {\mathit{M}}_I^{\left(p\right)\left(2\right)}& \cdots & {\mathit{M}}_I^{\left(p\right)\left(p\right)}\end{array}\right)}^{\left(i\right)},\hfill \\ \hfill & {\mathit{M}}_I^{\left(j\right)\left(k\right)}={\mathbf{\xi }}_I^{\left(i\right)}⨂{\overline{\mathbf{\xi }}}_I^{\left(i\right)}={\left(\frac{{\mathbf{\xi }}_I^{\left(i\right)\left(j\right)}}{{∥{\mathbf{\xi }}_I^{\left(i\right)\left(j\right)}∥}_2}\right)}^{\top }\left(\frac{{\overline{\mathbf{\xi }}}_I^{\left(i\right)\left(k\right)}}{{∥{\overline{\mathbf{\xi }}}_I^{\left(i\right)\left(k\right)}∥}_2}\right),\hfill \\ \hfill & j,k=1,2,\cdots ,p.\hfill \end{array}$$

(13)

where ${∥·∥}_2$ is the L2 norm, notation ⨂ is called semantic correlation multiplication. Obviously, ${\mathit{M}}_I^{\left(i\right)}$ encode the semantic correlation between the single-sample ${\mathit{I}}_i$ and the corresponding group ${\left\{{\mathit{I}}_j\right\}}_{j=1}^{\alpha }$. We reshape it in the following form:

$${\mathit{M}}_I^{\left(i\right)}=\left({\mathit{m}}_I^{\left(i\right)\left(1\right)},{\mathit{m}}_I^{\left(i\right)\left(2\right)},\cdots ,{\mathit{m}}_I^{\left(i\right)\left(p\right)}\right).$$

(14)

where ${\mathit{m}}_I^{\left(i\right)\left(j\right)}\in {\mathbb{R}}^p$ is the encoding of semantic correlation between the local single-sample feature representation ${\mathbf{\xi }}_I^{\left(i\right)\left(j\right)}$ and all the grouping-sample feature representations ${\left\{{\overline{\mathbf{\xi }}}_I^{\left(i\right)\left(k\right)}\right\}}_{k=1}^p$. Therefore, the local semantic correlation between a specific feature representation ${\mathbf{\xi }}_I^{\left(i\right)}$ and the average semantic representation ${\overline{\mathbf{\xi }}}_I^{\left(i\right)}$ of the corresponding category can be measured directly.

The intra-attention map ${\mathit{A}}_I^{\left(i\right)}$ is generated from the semantic correlation matrices ${\mathit{M}}_I^{\left(i\right)}$ via learning a convolutional operation to fuse the semantic correlations between local single-sample feature vector ${\mathbf{\xi }}_I^{\left(i\right)\left(j\right)}$ and all the grouping-sample features ${\left\{{\overline{\mathbf{\xi }}}_I^{\left(i\right)\left(k\right)}\right\}}_{k=1}^p$. Specifically, let ${\mathit{K}}_I^{\left(i\right)}\in {\mathbb{R}}^{p\times 1}$ be the convolutional kernel, which is learned from the inputs $⟨{\mathbf{\xi }}_I^{\left(i\right)},{\overline{\mathbf{\xi }}}_I^{\left(i\right)}⟩$ by meta learning as follows:

$$\begin{array}{ccc}\hfill {\mathit{K}}_I^{\left(i\right)}=& {\mathit{W}}_2\times \sigma ({\mathit{W}}_1\times (\frac{1}{p}\sum _{j=1}^p{\mathit{M}}_I^{\left(1\right)\left(j\right)},\hfill & \hfill \frac{1}{p}\sum _{j=1}^p{\mathit{M}}_I^{\left(2\right)\left(j\right)},\cdots ,\frac{1}{p}\sum _{j=1}^p{\mathit{M}}_I^{\left(p\right)\left(j\right)}{)}^{\top }),\end{array}$$

(15)

where ${\mathit{W}}_1$ and ${\mathit{W}}_2$ denote the model parameter vectors, $\sigma (·)$ is a non-linear activation function, here we employ ReLU function. Then a softmax operation is conducted on the convolution result to generate intra-attention map ${\mathit{A}}_I^{\left(i\right)}\in {\mathbb{R}}^{x\times y}$:

$$\begin{array}{ccc}\hfill & {\mathit{A}}_I^{\left(i\right)}=\left({\mathit{A}}_I^{\left(i\right)\left(1\right)},{\mathit{A}}_I^{\left(i\right)\left(2\right)},{\mathit{A}}_I^{\left(i\right)\left(p\right)}\right),\hfill & \hfill {\mathit{A}}_I^{\left(i\right)\left(j\right)}=\frac{exp\left(\frac{1}{\Gamma }{\left({\mathit{K}}_I^{\left(i\right)}\right)}^{\top }\times {\mathit{m}}_I^{\left(i\right)\left(j\right)}\right)}{{\sum }_{j=1}^{p}exp\left(\frac{1}{\Gamma }{\left({\mathit{K}}_I^{\left(i\right)}\right)}^{\top }\times {\mathit{m}}_I^{\left(i\right)\left(j\right)}\right)},\end{array}$$

(16)

where $\Gamma$ is the temperature hyperparameter that influences the entropy. In the same way, the intra-attention map of text modality ${\mathit{A}}_I^{\left(i\right)}\in {\mathbb{R}}^{x\times y}$ is achieved. Finally, a residual attention mechanism is utilized to calculate the results for both modalities:

$$\begin{array}{cc}\hfill & {\mathbf{\xi }}_I^{\prime \left(i\right)}={\mathbf{\xi }}_I^{\left(i\right)}⨀\left(1+{\mathit{A}}_I^{\left(i\right)}\right),\hfill \\ \hfill & {\mathbf{\xi }}_T^{\prime \left(i\right)}={\mathbf{\xi }}_T^{\left(i\right);}⨀\left(1+{\mathit{A}}_T^{\left(i\right)}\right),\hfill \end{array}$$

(17)

where ⨀ is the element-wise multiplication.

Following intra-attention model, a two-channel encoder $E(·;{\mathbf{\theta }}_{Enc}^I)$ and $E(·;{\mathbf{\theta }}_{Enc}^T)$ is to generate common representations ${\mathit{F}}_I^{\prime \left(i\right)}$ and ${\mathit{F}}_T^{\prime \left(i\right)}$. In this model, weight-sharing constraint is applied in last few layers to learn the cross-modal consistent joint distribution, which diminishes heterogeneity effectively.

4.3.2. Inter-Attention

To realize semantic augmentation in the common representation subspace, an inter-attention model is designed to learn the semantic relationship between image and text, i.e.,

$$⟨{\stackrel{⃛}{\mathit{F}}}_I^{\left(i\right)},{\stackrel{⃛}{\mathit{F}}}_T^{\left(i\right)}⟩=\mathit{InterAtt}\left(⟨{\mathit{F}}_I^{\left(i\right)},{\mathit{F}}_T^{\left(i\right)}⟩;{\mathbf{\theta }}_{Inter}\right).$$

(18)

Similar to the intra-attention mechanism, it calculates the cross-modal semantic correlation matrix ${\mathit{U}}^{\left(i\right)}$ from ${\mathit{F}}_I^{\left(i\right)}$ and ${\mathit{F}}_T^{\left(i\right)}$:

$$\begin{array}{cc}\hfill {\mathit{U}}^{\left(j\right)\left(k\right)}& ={\mathit{F}}_I^{\left(i\right)}⨂{\mathit{F}}_T^{\left(i\right)}={\left(\frac{{\mathit{F}}_I^{\left(i\right)\left(j\right)}}{{∥{\mathit{F}}_I^{\left(i\right)\left(j\right)}∥}_2}\right)}^{\top }\left(\frac{{\overline{\mathit{F}}}_T^{\left(i\right)\left(k\right)}}{{∥{\overline{\mathit{F}}}_T^{\left(i\right)\left(k\right)}∥}_2}\right),\hfill \\ \hfill & j,k=1,2,\cdots ,p.\hfill \end{array}$$

(19)

and then generates two correlation matrices:

$$\begin{array}{cc}\hfill & {\mathit{U}}_I^{\left(i\right)}={\mathit{U}}^{\left(i\right)}=\left({\mathit{u}}_I^{\left(i\right)\left(1\right)},{\mathit{u}}_I^{\left(i\right)\left(2\right)},\cdots ,{\mathit{u}}_I^{\left(i\right)\left(p\right)}\right),\hfill \\ \hfill & {\mathit{U}}_T^{\left(i\right)}={\left({\mathit{U}}^{\left(i\right)}\right)}^{\top }=\left({\mathit{u}}_T^{\left(i\right)\left(1\right)},{\mathit{u}}_T^{\left(i\right)\left(2\right)},\cdots ,{\mathit{u}}_T^{\left(i\right)\left(p\right)}\right).\hfill \end{array}$$

(20)

Similar to Equation (14), ${\mathit{u}}_I^{\left(i\right)\left(j\right)},\in {\mathbb{R}}^p$ encodes semantic correlation between the local image feature vector ${\mathit{F}}_I^{\left(i\right)\left(j\right)}$ in j-th position and all the text feature vectors ${\left\{{\mathit{F}}_T^{\left(i\right)\left(k\right)}\right\}}_{k=1}^p$. ${\mathit{u}}_T^{\left(i\right)\left(2\right)}\in {\mathbb{R}}^p$ encodes the semantic correlation between the local text feature vector ${\mathit{F}}_T^{\left(i\right)\left(k\right)}$ in j-th position and all the image feature vectors ${\left\{{\mathit{F}}_I^{\left(i\right)\left(j\right)}\right\}}_{j=1}^p$. Then, two convolutional kernels ${\widehat{\mathit{K}}}_I^{\left(i\right)}$ and ${\widehat{\mathit{K}}}_T^{\left(i\right)}$ are learned in the same way as Equation (15) and the inter-attention maps ${\widehat{\mathit{A}}}_I^{\left(i\right)}$ and ${\widehat{\mathit{A}}}_T^{\left(i\right)}$ for image and text are achieved by Equation (16). Thus, more discriminative cross-modal representations ${\mathit{F}}_I^{\prime \left(i\right)}$ and ${\mathit{F}}_T^{\prime \left(i\right)}$ can be achieved by residual attention mechanism.

4.3.3. Discriminative Model

Three types of discrimination model are integrated into DA-GAN framework: (1) a semantic discriminator $D_S(·;{\mathbf{\theta }}_S)$ to realize semantic discrimination, (2) a two-channel intra-modal discriminator $D_I(·;{\mathbf{\theta }}_D^I)$ and $D_T(·;{\mathbf{\theta }}_D^I)$ and (3) a two-channel inter-modal discriminator ${\widehat{D}}_I(·;{\widehat{\mathbf{\theta }}}_D^I)$ and ${\widehat{D}}_T(·;{\widehat{\mathbf{\theta }}}_D^T)$ to realize intra-modal and inter-modal adversarial learning.

Semantic Discriminator. Semantic discriminator $D_S(·;{\mathbf{\theta }}_S)$ is used to recognize the semantic category of the instance in common semantic representation subspace. To this end, a two-channel network with softmax function is added on the top of inter-attention model (one channel per modality), which takes ${\mathit{F}}_I^{\prime \left(i\right)}$ and ${\mathit{F}}_T^{\prime \left(i\right)}$ and inputs and outputs the predicted probability distribution $P_I\left({\stackrel{⃛}{\mathit{F}}}_I^{\left(i\right)}\right)$ and $P_T\left({\stackrel{⃛}{\mathit{F}}}_T^{\left(i\right)}\right)$ to calculate the semantic discrimination loss:

$$\begin{array}{c}\hfill {\mathcal{L}}_{Sem}\left({\mathbf{\theta }}_S\right)=-\frac{1}{m}\sum _{i=1}^m\left[{\mathit{L}}_i\left(log{P}_I\left({\stackrel{⃛}{\mathit{F}}}_I^{\left(i\right)}\right)+log{P}_I\left({\stackrel{⃛}{\mathit{F}}}_T^{\left(i\right)}\right)\right)\right],\end{array}$$

(21)

where ${\mathbf{\theta }}_S=({\mathbf{\theta }}_G^I,{\mathbf{\theta }}_G^T,{\mathbf{\theta }}_C)$ denotes the parameter vector of this model, ${\mathbf{\theta }}_C$ is the parameter vector of the classifier. ${\mathbf{\theta }}_G^I$ and ${\mathbf{\theta }}_G^T$ denote the parameter vector of the image and text generation model respectively, i.e., ${\mathbf{\theta }}_G^I=({\mathbf{\theta }}_{Fea}^I,{\mathbf{\theta }}_{Intra}^I,{\mathbf{\theta }}_{Enc}^I,{\mathbf{\theta }}_{Inter}^I)$ and ${\mathbf{\theta }}_G^T=({\mathbf{\theta }}_{Fea}^T,{\mathbf{\theta }}_{Intra}^T,{\mathbf{\theta }}_{Enc}^T,{\mathbf{\theta }}_{Inter}^T)$.

Intra-Model Discriminator. The intra-modal discriminator tries to discriminate the real representations ${\mathbf{\xi }}_I^{\prime \left(i\right)}$ (${\mathbf{\xi }}_T^{\prime \left(i\right)}$) from intra-attention model and the synthetic representations ${\mathbf{\zeta }}_I^{\left(i\right)}$ (${\mathbf{\zeta }}_T^{\left(i\right)}$) from decoder as inputs. For simplicity, we denote this branch network as $GAN1$, whose objective function is:

$$\begin{array}{cc}\hfill arg\underset{G_I,G_T}{min}\underset{D_I,D_T}{max}{\mathcal{L}}_{GAN1}\left({\mathbf{\theta }}_G^I,{\mathbf{\theta }}_G^T,{\mathbf{\theta }}_D^I,{\mathbf{\theta }}_D^T\right)& ={\mathbb{E}}_{\mathit{I}\sim P_I\left(\mathit{I}\right)}\left[log{D}_I\left(\mathit{I};{\mathbf{\theta }}_D^I\right)\right]+\hfill \\ & {\mathbb{E}}_{\mathit{I}\sim P_I\left(\mathit{I}\right)}\left[log\left(1-D_I\left(G_I\left(\mathit{I};{\mathbf{\theta }}_G^I\right);{\mathbf{\theta }}_D^I\right)\right)\right]+\hfill \\ & {\mathbb{E}}_{\mathit{T}\sim P_T\left(\mathit{T}\right)}\left[log{D}_T\left(\mathit{T};{\mathbf{\theta }}_D^T\right)\right]+\hfill \\ & {\mathbb{E}}_{\mathit{T}\sim P_T\left(\mathit{T}\right)}\left[log\left(1-D_T\left(G_T\left(\mathit{T};{\mathbf{\theta }}_G^T\right);{\mathbf{\theta }}_D^T\right)\right)\right].\hfill \end{array}$$

(22)

Inter-Modal Discriminator. Similar to intra-modal discriminator, the inter-modal discriminator has two channels, the subnetwork for image modality is to recognize the visual common representation as the real sample. By contrast, the subnetwork for text modality aims to recognize the textual common representation as the real sample. This branch of the adversarial network is denoted as $GAN2$. The objective function is:

$$\begin{array}{cc}\hfill arg\underset{G_I,G_T}{min}\underset{{\widehat{D}}_I,{\widehat{D}}_T}{max}{\mathcal{L}}_{GAN2}\left({\mathbf{\theta }}_G^I,{\mathbf{\theta }}_G^T,{\widehat{\mathbf{\theta }}}_D^I,{\widehat{\mathbf{\theta }}}_D^T\right)& ={\mathbb{E}}_{\mathit{I},\mathit{T}\sim P_I\left(\mathit{I}\right),P_T\left(\mathit{T}\right)}[log{\widehat{D}}_I\left(G(\mathit{I};{\mathbf{\theta }}_G^I);{\widehat{\mathbf{\theta }}}_D^I\right)-\hfill \\ \hfill & log{\widehat{D}}_I\left(G\left(\mathit{T};{\mathbf{\theta }}_G^T\right);{\widehat{\mathbf{\theta }}}_D^I\right)+\hfill \\ \hfill & log{\widehat{D}}_T\left(G_T\left(\mathit{T};{\mathbf{\theta }}_G^T\right);{\widehat{\mathbf{\theta }}}_D^T\right)-\hfill \\ \hfill & log{\widehat{D}}_T\left(G\left(\mathit{I};{\mathbf{\theta }}_G^I\right);{\widehat{\mathbf{\theta }}}_D^T\right)].\hfill \end{array}$$

(23)

4.3.4. Optimization

According to the above discussion, the DA-GAN model can be optimized by the following objective functions:

$$\begin{array}{c}\hfill arg\underset{G_I,G_T}{min}\underset{D_I,D_T}{max}{\mathcal{L}}_{GAN1}\left({\mathbf{\theta }}_G^I,{\mathbf{\theta }}_G^T,{\mathbf{\theta }}_D^I,{\mathbf{\theta }}_D^T\right)+{\mathcal{L}}_{GAN2}\left({\mathbf{\theta }}_G^I,{\mathbf{\theta }}_G^T,{\widehat{\mathbf{\theta }}}_D^I,{\widehat{\mathbf{\theta }}}_D^T\right),\end{array}$$

(24)

$$arg\underset{{\mathbf{\theta }}_S}{min}{\mathcal{L}}_{Sem}\left({\mathbf{\theta }}_S\right).$$

(25)

For discrimination in $GAN1$, the intra-modal discriminator takes the convolutional representation ${\mathbf{\xi }}_I^{\prime \left(i\right)}$ (${\mathbf{\xi }}_T^{\prime \left(i\right)}$) and the reconstruction representation from ${\mathbf{\zeta }}_I^{\left(i\right)}$ (${\mathbf{\zeta }}_T^{\left(i\right)}$) decoder as inputs. It maximizes the log-likelihood for discriminating the real data ${\mathbf{\xi }}_I^{\prime \left(i\right)}$ (${\mathbf{\xi }}_T^{\prime \left(i\right)}$) and the synthetic data ${\mathbf{\zeta }}_I^{\left(i\right)}$ (${\mathbf{\zeta }}_T^{\left(i\right)}$) by stochastic gradient ascending:

$$\begin{array}{c}\hfill {\mathbf{\theta }}_D^I\leftarrow {\mathbf{\theta }}_D^I+\eta {\nabla }_{{\mathbf{\theta }}_D^I}\frac{1}{m}\sum _{i=1}^n\left[log\left(D_I\left({\mathbf{\xi }}_I^{\prime \left(i\right)};{\mathbf{\theta }}_D^I\right)\right)+log\left(1-D_I\left({\mathbf{\zeta }}_I^{\left(i\right)};{\mathbf{\theta }}_D^I\right)\right)\right],\end{array}$$

(26)

$$\begin{array}{c}\hfill {\mathbf{\theta }}_D^T\leftarrow {\mathbf{\theta }}_D^T+\eta {\nabla }_{{\mathbf{\theta }}_D^T}\frac{1}{m}\sum _{i=1}^n\left[log\left(D_T\left({\mathbf{\xi }}_T^{\prime \left(i\right)};{\mathbf{\theta }}_D^T\right)\right)+log\left(1-D_T\left({\mathbf{\zeta }}_T^{\left(i\right)};{\mathbf{\theta }}_D^T\right)\right)\right].\end{array}$$

(27)

For discrimination in $GAN2$, the subnetwork for image modality receives the image common representation ${\mathit{F}}_I^{\prime \left(i\right)}$ as the real instance and the text common representation ${\mathit{F}}_T^{\prime \left(i\right)}$ as the fake instance. The stochastic gradient ascending is calculated as:

$$\begin{array}{c}\hfill {\widehat{\mathbf{\theta }}}_D^I\leftarrow {\widehat{\mathbf{\theta }}}_D^I+\eta {\nabla }_{{\widehat{\mathbf{\theta }}}_D^I}\frac{1}{m}\sum _{i=1}^n\left[log\left({\widehat{D}}_I\left({\stackrel{⃛}{\mathit{F}}}_I^{\left(i\right)},{\mathbf{\xi }}_I^{\prime \left(i\right)};{\widehat{\mathbf{\theta }}}_D^I\right)\right)+log\left(1-{\widehat{D}}_I\left({\stackrel{⃛}{\mathit{F}}}_T^{\left(i\right)},{\mathbf{\xi }}_I^{\prime \left(i\right)};{\widehat{\mathbf{\theta }}}_D^I\right)\right)\right],\end{array}$$

(28)

$$\begin{array}{c}\hfill {\widehat{\mathbf{\theta }}}_D^T\leftarrow {\widehat{\mathbf{\theta }}}_D^T+\eta {\nabla }_{{\widehat{\mathbf{\theta }}}_D^T}\frac{1}{m}\sum _{i=1}^n\left[log\left({\widehat{D}}_T\left({\stackrel{⃛}{\mathit{F}}}_T^{\left(i\right)},{\mathbf{\xi }}_T^{\prime \left(i\right)};{\widehat{\mathbf{\theta }}}_D^T\right)\right)+log\left(1-{\widehat{D}}_T\left({\stackrel{⃛}{\mathit{F}}}_T^{\left(i\right)},{\mathbf{\xi }}_I^{\prime \left(i\right)};{\widehat{\mathbf{\theta }}}_D^T\right)\right)\right].\end{array}$$

(29)

For the two-channel generative model, it aims to generate more authentic data from the original sample to fit the real semantic distribution by minimizing the objective function. Both of the subnetworks are optimized by stochastic gradient descent (SGD) as follows:

$$\begin{array}{c}\hfill {\mathbf{\theta }}_G^I\leftarrow {\mathbf{\theta }}_G^I-\eta {\nabla }_{{\mathbf{\theta }}_G^I}\frac{1}{m}\sum _{i=1}^n\left[log\left({\widehat{D}}_T\left({\stackrel{⃛}{\mathit{F}}}_I^{\left(i\right)},{\mathbf{\xi }}_T^{\prime \left(i\right)};{\widehat{\mathbf{\theta }}}_D^T\right)\right)+log\left(D_I\left({\mathbf{\zeta }}_T^{\left(i\right)};{\mathbf{\theta }}_D^I\right)\right)\right],\end{array}$$

(30)

$$\begin{array}{c}\hfill {\mathbf{\theta }}_G^T\leftarrow {\mathbf{\theta }}_G^T-\eta {\nabla }_{{\mathbf{\theta }}_G^T}\frac{1}{m}\sum _{i=1}^n\left[log\left({\widehat{D}}_I\left({\stackrel{⃛}{\mathit{F}}}_T^{\left(i\right)},{\mathbf{\xi }}_I^{\prime \left(i\right)};{\widehat{\mathbf{\theta }}}_D^I\right)\right)+log\left(D_T\left({\mathbf{\zeta }}_T^{\left(i\right)};{\mathbf{\theta }}_D^T\right)\right)\right].\end{array}$$

(31)

Besides, the generative model is optimized by the semantic discrimination to learning abstract semantic concepts:

$$\begin{array}{c}\hfill {\mathbf{\theta }}_S\leftarrow {\mathbf{\theta }}_S-\eta {\nabla }_{{\mathbf{\theta }}_S}\left(-\frac{1}{m}\sum _{i=1}^m\left[{\mathit{L}}_i\left(log{P}_I\left({\stackrel{⃛}{\mathit{F}}}_I^{\left(i\right)}\right)+log{P}_I\left({\stackrel{⃛}{\mathit{F}}}_T^{\left(i\right)}\right)\right)\right]\right),\end{array}$$

(32)

where $\eta$ denotes the learning rate, m denotes the number of samples in each mini-batch.

The pseudocode of optimizing the proposed model is shown in Algorithm 1. Before training the $GAN1$ and $GAN2$, we pre-train the multi-modal feature learning model and intra-attention modal for both image and text on the training set, which is to prevent the instability of training $GAN1$ and $GAN2$. The minimax game is implemented by Adam [66].

Algorithm 1:Pseudocode of optimizing DA-GAN

1: Initialization: a training set $\mathcal{D}={\left\{⟨{\mathit{I}}_i,{\mathit{T}}_i,{\mathit{L}}_i⟩\right\}}_{i=1}^n$, mini-batch size m, the number of generative model training steps k, learning rate $\eta$. 2: pre-train $\mathit{ImgCNN}(·;{\mathbf{\theta }}_{Fea}^I)$ and ${\mathit{IntraAtt}}_I\left(·;{\mathbf{\theta }}_{Intra}^I\right)$; 3: pre-train $\mathit{TxtCNN}(·;{\mathbf{\theta }}_{Fea}^T)$ and ${\mathit{IntraAtt}}_T\left(·;{\mathbf{\theta }}_{Intra}^T\right)$; 4: repeat until convergence: 5: fork steps do 6:    Update the parameters of generator for image ${\mathbf{\theta }}_G^I$ by Equation (30); 7:    Update the parameters of generator for text ${\mathbf{\theta }}_G^T$ by Equation (31); 8:    Update the parameters of generators for both image and text ${\mathbf{\theta }}_G^I$ and ${\mathbf{\theta }}_G^T$ by Equation (32); 9: end for 10: Update the parameters of intra-modal discriminator ${\mathbf{\theta }}_D^I$ for image by Equation (26); 11: Update the parameters of intra-modal discriminator ${\mathbf{\theta }}_D^T$ for text by Equation (27); 12: Update the parameters of inter-modal discriminator for image ${\widehat{\mathbf{\theta }}}_D^I$ by Equation (28); 13: Update the parameters of inter-modal discriminator for text ${\widehat{\mathbf{\theta }}}_D^T$ by Equation (29); 14: Output: the optimized DA-GAN model.

4.4. Implementation Details

Multi-Modal Feature Learning Model. The image feature learning model is implemented by the AlexNet [64] pre-trained on ImageNet dataset. Each input is resized into $256\times 256$ without cropping and $227\times 227$ patches are extracted randomly from the inputs. The 4096-dimensional feature maps from the fc7 layer are treated as the outputs. To improve the learning performance, we fine-tune this model on the training dataset via squared loss. The mini-batch size of 128. the learning rate of the convolutional layer and fully-connected layer are set as 0.001 and 0.002, respectively. The momentum, weight decay, and drop-out rate are set to 0.9, 0.0005, and 0.5, respectively. The convolutional kernel size is set to $3\times 300$, following which is one layer fully-connected network. The drop-out rate is set to 0.5 to avoid over-fitting. The dimension of the last fully-connected layer is set to 4096. The Textual feature learning model includes a pre-trained word2vec model Skip-gram on Wikipedia corpus which contains over 1.8 billion words. This model outputs 300-dimensional word vectors from texts. The textual CNN contains a filter with a size of 3 × 300. The last fully-connected layer has 4096 dimensions and the learning rate is set to 0.01.

Encoder and Decoder. The two-channel encoder is implemented by a two-layer fully-connected network. For each channel, both of the fc layers are 1024-dimensional, and the weights of the second layer are shared over two branches to model the cross-modal joint distributions. Each branch of the decoder has two layers of fully-connected networks. The dimension of these two layers are 1024 and 4096, respectively.

Intra-modal and Inter-modal Discriminator. For intra-modal discriminator, each branch of it is constructed by one FC layer. To discriminate the convolutional representations and the reconstructed representations, the former is labeled by tag 1, and the latter is labeled by tag 0. For the inter-modal discriminator, both of the two channels are two-layer fully-connected networks. The 1ts layer has 1024 dimensions, and the 2nd layer with a sigmoid activation function calculates the predicted score for each input representation. The common representations of image modality are labeled by 1 and the representations of text modality are labeled by 0. For text modality, these two types of representations are labeled in the opposite way.

5. Experiments

5.1. Datasets

All the experiments are conducted on three widely-used benchmark datasets: Wikipedia [34], NUS-WIDE [67] and Pascal Sentences [68]. Some image and text samples of these three datasets are shown in Figure 3.

Figure 3. Some image and text samples of Wikipedia, NUS-WIDE and Pascal Sentences.

5.2. Competitors

We compare the proposed DA-GAN with 13 competitors, including 6 traditional cross-modal retrieval approaches, i.e., CCA [69], KCCA [11], MCCA [70], MvDA [71], MvDA-VC [72] and JRL [73], as well as 7 deep learning-based approaches, i.e., DCCA [42], DCCAE [24], CCL [74], CMDN [75], ACMR [57], DSCMR [44], CM-GANs [76]. The brief introductions of them are listed here.

• CCA [69] is a statistical method that is to learn linear correlations between samples of different modalities.
• KCCA [11] is a non-linear extension of CCA, which employs kernel function to improve the performance of common subspace learning.
• MCCA [70] is a generalization of CCA to more than two views, which is used to recognize similar patterns across multiple domains.
• MvDA [71] jointly learns multiple view-specific linear transforms so as to construct a common subspace for multiple views.
• MvDA-VC [72] is an extension of MvDA with with view consistency, which utilize the structure similarity of views corresponding to the same object.
• JRL [73] uses sparse projection matrix and semi-supervised regularization to explore correlations of labeled and unlabeled cross-modal samples.
• DCCA [42] is implemented by deep neural networks to learn non-linear correlation. It has two separated DNNs, one branch per modality.
• DCCAE [24] is a DCCA extension that integrates CCA model and autoencoder-based model to realize multi-view representation learning.
• CCL [74] realizes a hierarchical network to combine multi-grained fusion and cross-modal correlation exploiting. It includes two learning stages to realize representation learning and intrinsic relevance exploiting.
• CMDN [75] contains two learning stages to model the complementary separate representation of different modalities, and combines cross-modal representations to generate rich cross-media correlation.
• ACMR [57] is a adversarial learning-based method to construct a common subspace for different modalities by generating modality-invariant representations.
• DSCMR [44] exploits semantic discriminative features from both label space and common representation space by supervised learning, and minimizes modality invariance loss via weight-sharing to generate modality-invariant representation.
• CM-GANs [76] models cross-modal joint distributions by two parallel GANs to generate modality-invariance representations.

5.3. Performance Metrics

Two tasks are considered, i.e., (1) I2T retrieval and (2) T2I retrieval, both of which are defined in Definition 1. Besides, we utilize PR-curves and mAP score to measure the retrieval performance:

$$Pr=\frac{TP}{TP+FP},Re=\frac{TP}{TP+FN},$$

(33)

$$mAP=\frac{1}{\left|\right\{Q\left\}\right|}{\sum }_{i=1}^{\left|\right\{Q\left\}\right|}AP\left(Q\right)$$

(34)

5.4. Experimental Results

5.4.1. Results on Wikipedia Dataset

The mAP scores of DA-GAN and the 13 competitors on the Wikipedia dataset are reported in Table 2. For both I2T and T2I tasks, the proposed DA-GAN outperforms all these state-of-the-arts by 54.3% and 63.9% respectively, higher than the two best competitors, i.e., DSCMR [44] (I2T mAP = 52.1%) and CM-GANs [76] (T2I mAP = 62.1%). Besides, the average mAP of DA-GAN is the highest, which is 3% higher than CM-GANs. The main reason is that the combination of intra- and inter-modal attention captures more single-modal and cross-modal semantic correlations. Although both DSCMR and CM-GANs extract the semantic information by supervised learning, they do not learn the inter-modal semantic correlation effectively to realize cross-modal semantic augmentation. On the other hand, except for DCCA and DCCAE whose mAPs (I2T mAP = 44.4% and 43.5%, T2I mAP = 39.6% and 38.5%) are a bit lower than JRL (I2T mAP = 44.9%, T2I mAP = 41.8%).

Table 2. The comparison results (mAP@50 in %) with 13 competitors on Wikipedia dataset. The best performance values are in bold-font.

Traditional Method	I2T	T2I	Aver.
CCA [69]	13.4	13.3	13.4
KCCA [11]	19.8	18.6	19.2
MCCA [70]	34.1	30.7	32.4
MvDA [71]	33.7	30.8	32.3
MvDA-VC [72]	38.8	35.8	37.3
JRL [73]	44.9	41.8	43.4
Deep Learning-Based Method	I2T	T2I	Aver.
DCCA [42]	44.4	39.6	42.0
DCCAE [24]	43.5	38.5	41.0
CCL [74]	50.4	45.7	48.1
CMDN [75]	48.7	42.7	45.7
ACMR [57]	47.7	43.4	45.6
DSCMR [44]	52.1	47.8	49.9
CM-GANs [76]	50.0	62.1	56.1
The Proposed Method	I2T	T2I	Aver.
DA-GAN	54.3	63.9	59.1

Figure 4 and Figure 5 illustrate the I2T and T2I mAP scores of each category on Wikipedia dataset. The average mAP scores are shown in Figure 6. Obviously, for all these approaches, there are big differences between the retrieval precisions of different categories. Specifically, for both I2T and T2I tasks, the performances on “biology”, “geography & places”, “sport & recreation” and “warfare” are better than other categories. That is mainly because the samples in the above categories are semantically independent of other categories, and have more obvious distinguishing features than other categories. In contrast, the categories “art & architecture”, “history” and “royalty & nobility” are relative to each other in abstract semantics. The samples of the categories have more confusing features. From Figure 4 and Figure 5, it is clear that DA-GAN has better semantic recognition ability. For example, the highest I2T and T2I mAP scores of DA-GAN on “biology”, “sport & recreation” and “warfare” are near 83% and 85%, higher than the competitive rivals such as DSCMR (I2T mAP = 78%, T2I mAP = 73%), CCL (I2T mAP = 73%, T2I mAP = 69%) and CM-GANs (I2T mAP = 74%, T2I mAP = 82%).

Figure 4. The mAP of I2T task of each category on Wikipedia dataset for our method DA-GAN and the competitors.

Figure 5. The mAP of T2I task of each category on Wikipedia dataset for our method DA-GAN and the competitors.

Figure 6. The average mAP of each category on Wikipedia dataset for our method DA-GAN and the competitors.

Figure 7a,d show the I2T and T2I precisions of DA-GAN and the competitors on different recalls, respectively. For both I2T and T2I tasks, DA-GAN has the highest precision at all levels of recall, which exhibits the performance improvement by adversarial learning with a dual attention mechanism. DSCMR and CM-GANs are still the most two competitive rivals, but they cannot defeat DA-GAN at any recall value.

Figure 7. The PR curve of our method DA-GAN and the competitors on Wikipedia, NUS-WIDE and Pascal Sentences dataset. (a–c) show the PR curves of I2T task on Wikipedia, NUS-WIDE and Pascal Sentences; (d–f) report PR curves of I2T task.

5.4.2. Results on Nus-Wide Dataset

The mAP scores on NUS-WIDE of DA-GAN and competitors are reported in Table 3. Compared with the results on Wikipedia, the precision of all these methods are relatively higher. The proposed method performs well on this dataset, which defeats CM-GANs (I2T mAP = 78.1%, T2I mAP = 72.4%, Aver. mAP = 75.3%) and DSCMR (I2T mAP = 61.1%, T2I mAP = 61.5%, Aver. mAP = 61.3%) by I2T mAP = 79.7%, T2I mAP = 75.2%, Aver. mAP = 77.5%. It indicates that the dual attention mechanism can discover more important semantic features between different modalities to generate more discriminant representations. On the other hand, we observe that the performance of other traditional and deep learning-based approaches are far behind our method even though the precisions of them are obviously higher than the results on Wikipedia.

Table 3. The comparison results (mAP@50 in %) with 13 competitors on NUS-WIDE dataset. The best performance values are in bold-font.

Traditional Method	I2T	T2I	Aver.
CCA [69]	37.8	39.4	38.6
KCCA [11]	36.2	39.4	37.8
MCCA [70]	44.8	46.2	45.5
MvDA [71]	50.1	52.6	51.3
MvDA-VC [72]	52.6	55.7	54.2
JRL [73]	58.6	59.8	59.2
Deep Learning-Based Method	I2T	T2I	Aver.
DCCA [42]	53.2	54.9	54.0
DCCAE [24]	51.1	54.0	52.5
CCL [74]	50.6	53.5	52.1
CMDN [75]	49.2	51.5	50.4
ACMR [57]	58.8	59.9	59.3
DSCMR [44]	61.1	61.5	61.3
CM-GANs [76]	78.1	72.4	75.3
The Proposed Method	I2T	T2I	Aver.
DA-GAN	79.7	75.2	77.5

The PR curve of DA-GAN and the state-of-the-arts are presented in Figure 7b,c. We can find that the trends of the precisions on NUS-WIDE are different from the situations on Wikipedia. For the I2T task (shown in Figure 7b), the precision of DA-GAN and the competitors decline obviously in the interval $[0.0,0.2]$. After that, the downward trend tends to be gentle. When the recall is larger than 0.8, fast performance degradation occurs, except for three traditional methods, i.e., CCA, KCCA, and MCCA. At all levels of recall, the precision of DA-GAN is higher than all the rivals. For the T2I task (shown in Figure 7c), the performance of all these approaches shows a gradual downward trend. Although the precision of CM-GANs is slightly higher than our method in the interval $[0.1,0.2]$, it cannot defeat DA-GAN when the recall is larger than 0.2. The retrieval accuracies of other approaches, as expected, are much lower than DA-GAN.

5.4.3. Results on Pascal Sentences Dataset

The Comparison of mAP scores of DA-GAN and the 13 state-of-the-arts on Pascal Sentences dataset are shown in Table 4. Once again, DA-GAN is the winner in this contest, which achieves I2T mAP = 72.9%, T2I mAP = 73.5% and average mAP = 73.2%, defeats the runner-up DSCMR (I2T mAP = 71.0%, T2I mAP = 72.2%, average mAP = 71.6%) by 2.5%, 1.3% and 1.6%, respectively. Different from the above comparisons, CM-GANs (I2T mAP = 61.2%, T2I mAP = 61.0%, average mAP = 61.1%) performs worse than DA-GAN and DSCMR evidently. As analyzed above, the performance improvement mainly comes from the integration of intra- and inter-modal attention as well as adversarial learning.

Table 4. The comparison results (mAP@50 in %) with 13 competitors on Pascal Sentences dataset. The best performance values are in bold-font.

Traditional Method	I2T	T2I	Aver.
CCA [69]	22.5	22.7	22.6
KCCA [11]	43.3	39.8	41.6
MCCA [70]	66.4	48.9	55.45
MvDA [71]	59.4	62.6	61.0
MvDA-VC [72]	64.8	67.3	66.1
JRL [73]	52.7	53.4	53.1
Deep Learning-Based Method	I2T	T2I	Aver.
DCCA [42]	67.8	67.7	67.8
DCCAE [24]	68.0	67.1	67.5
CCL [74]	57.6	56.1	56.9
CMDN [75]	54.4	52.6	53.5
ACMR [57]	67.1	67.6	67.3
DSCMR [44]	71.0	72.2	71.6
CM-GANs [76]	61.2	61.0	61.1
The Proposed Method	I2T	T2I	Aver.
DA-GAN	72.9	73.5	73.2

Figure 8, Figure 9 and Figure 10 illustrate the I2T, T2I and average mAP scores of each approaches on 20 categories on Pascal Sentences dataset, respectively. For both I2T and T2I tasks, all these approaches have poor cross-modal retrieval performance in some categories, such as “bottle” and “chair”. It is mainly because the objects in these categories are relatively small. By contrast, the precisions on “aeroplane”, “bird”, “cat”, “horse”, “motorbike”, “sheep” and “train” are obviously higher since these samples contain much more discriminative semantic features. Specifically, for the I2T task, the mAP of DA-GAN reaches nearly 90%, 91% and 92% on “aeroplane”, “cat” and “train”, respectively. For the T2I task, it achieves nearly 92%, 93%, and 95% on these three categories. From Figure 10 we observe that the semantic recognition performance of DA-GAN is the best among these 14 approaches.

Figure 8. The mAP of I2T task of each category on Pascal Sentences dataset for our method DA-GAN and the competitors.

Figure 9. The mAP of T2I task of each category on Pascal Sentences dataset for our method DA-GAN and the competitors.

Figure 10. The average mAP of each category on Pascal Sentences dataset for the proposed method and the state-of-the-arts.

Figure 7c,f show the PR curves of DA-GAN and 13 state-of-the-arts for I2T and T2I tasks, respectively. On both tasks, it is clear that the changing of performance of DA-GAN and CM-GANs are very similar. Although CM-GANs show good performance, they cannot overcome our method. For the I2T task, the precision of DA-GAN declines slowly when the recall increases from 0.2 to 0.8. After that, it drops sharply. In contrast, the performance of our method shows a significant downward trend for the T2I task, but it is still the best.

6. Conclusions

We present a new deep adversarial model for cross-modal retrieval, called Dual Attention Generative Adversarial Network (DA-GAN). This method utilizes a novel dual attention mechanism to focus on important semantic details in a uni-modal manner and a cross-modal manner, which can effectively learn high-level semantic interaction across different modalities. Besides, a dual adversarial learning method that learns modality-consistent representation is proposed to reduce the heterogeneity gap. Comprehensive experiments on four commonly used multimedia datasets indicate the great performance of the proposed method.