Multi-Granularity Semantic Collaborative Reasoning Network for Visual Dialog

Abstract

A visual dialog task entails an agent engaging in a multiple round conversation about an image. Notably, one of the main issues is capturing the semantic associations of multiple inputs, such as the questions, dialog history, and image features. Many of the techniques use a token or a sentence granularity semantic representation of the question and dialog history to model semantic associations; however, they do not perform collaborative modeling, which limits their efficacy. To overcome this limitation, we propose a multi-granularity semantic collaborative reasoning network to properly support a visual dialog. It employs different granularity semantic representations of the question and dialog history to collaboratively identify the relevant information from multiple inputs based on attention mechanisms. Specifically, the proposed method collaboratively reasons the question-related information from the dialog history based on its granular semantic representations. Then, it collaboratively locates the question-related visual objects in the image by leveraging refined question representations. The experimental results conducted on the VisDial v.1.0 dataset verify the effectiveness of the proposed method, showing the improvements of the best normalized discounted cumulative gain score from 59.37 to 60.98 with a single model, from 60.92 to 62.25 with ensemble models, and from 63.15 to 64.13 with performing multitask learning.

1. Introduction

With the many advances gleaned from the intersection of vision and language domains, several vision–language tasks (e.g., image captioning [1], visual question answering (VQA) [2], referring expression comprehension [3], and visual dialog [4]) have been introduced, attracting massive attention from the computer-vision community. Specifically, a visual dialog [4] is used to train an artificial intelligence (AI) agent to answer questions based on an image and its dialog history. This method offers practical benefits to society, such as in aiding visually impaired users to enjoy and appreciate electronic imagery.

To answer a question correctly and accurately, the AI agent first browses the dialog history to identify passages related to the question. Then, it locates the specific visual objects that match the explicit semantic intent of the question. Figure 1 shows an example of the visual dialog task, to predict the correct answer for the current question Q3, “Is it next to the apple and orange?”, the agent traces the dialog history to obtain clues about the meaning of “it”. In this case, “it” refers to a “bag”, as mentioned in the Q2–A2 pairing. The agent must also determine if there are any mentions of the words “apple” and “orange”. Then, it grounds these entities in the image. For this to work, the agent must comprehensively understand the current question and the past question–answer pairs. In this case, the agent learns that the spatial relationship should be correctly inferred between “it” and the apple and orange fruits pictured.

To endow an AI agent with these capabilities, most methods use attention mechanisms to acquire clues from the dialog history and select the specific visual objects that are relevant to the question. Dual attention networks (DAN) [5] uses sentence-granularity semantic representations of the question and dialog history to compute the weight distributions of all of the question–answer rounds. Visual object identification then requires multi-head attention mechanisms. ReDAN [6] and dual-channel multi-hop reasoning model (DMRM) [7] use sentence-granularity semantic representations to iteratively capture information from the dialog history and the image via multiple reasoning steps. Using multi-head attention mechanisms, lightweight transformer for many inputs (LTMI) [8] and transformer-based modular co-attention (MCA) [9] apply token-granularity semantic representations to excavate token-to-token/visual object interactions and assign weight distributions. However, these methods only use single-granularity semantic representations of the question and dialog history to manage the multi-model intersections, which is intuitively insufficient to accurately answer the question. For example, when answering Q3 in Figure 1, the agent may infer incorrect answers if it only understands the semantic token information from the question while not understanding the question’s semantic information. To compensate for this deficiency, the multi-view attention network (MVAN) [10] aggregates question-relevant historical information to facilitate visual object grounding by considering both token- and sentence-granularity semantic representations. MVAN overcomes the single-granularity modeling limitations, but it fails to explicitly and collaboratively explore the impact of different granularity semantic representations to locate the target visual objects.

Nevertheless, most of the approaches above are of single-granularity approaches, which exhibit limited reasoning skills and ignore latent information about relationships among the question, the dialog history, and the image. To this end, we propose a novel multi-granularity semantic collaborative reasoning network (MGSCRN) for visual dialog tasks. The model collaboratively extracts question-related information from the dialog history and original visual objects via multi-granularity encoding. Figure 2 shows a diagram for the proposed method. We first encode the question and dialog history into token- and sentence-granularity semantic representations. Then, we collaboratively assess the question-related information from the dialog history by considering the granular semantic representations of the question and dialog history. In this process, refined question representations with different granularities are obtained. Next, we perform collaborative reasoning to form an accurate alignment between visual objects and these two refined question representations. Finally, the target visual object representation-fused question features are delivered to the decoder for predicting answers.

The major contributions of this work are summarized as follows. First, the joint use of the multi-granularity semantic representations of the new question and the image’s dialog history allows MGSCRN to successfully complete a visual dialog task. The collaborative reasoning of the coreference relations among multiple inputs capitalizes on the heterogeneous informative relationships (i.e., token- and sentence-granularity semantic information) in the history, including those of the various objects in the image. Second, our experiments with the VisDial v1.0 dataset demonstrate outstanding performance results that are superior to other state-of-the-art methods. Third, our qualitative analysis shows that MGSCRN infers both historical and visual question-related information more accurately than the other methods, which notably utilize single semantically granular representations.

This paper is structured in the following manner. The related works are shown in Section 2. Section 3 describes our proposed MGSCRN’s model. In Section 4, we introduce our experimental setup and the visual dialog task measures. In Section 5, we empirically evaluate our proposed model using VisDial v1.0, VisPro, and VisDialConv datasets. Then, we report our experimental results and analyze the findings. This paper is concluded in Section 6.

2. Related Work

Visual Dialog

With the advancement of computing sciences and artificial intelligence, many researchers seek solutions to achieve truly intelligent artificial systems for complex real-life applications. Various machine learning techniques have been developed to tackle challenging real-world problems, such as unstructured data classification in social networks [11], sentiment analysis of Chinese short financial texts [12], and even energy conservation in communications [13].

Recently, many works have paid attention to handle multimodal cognitive problems by analyzing vision and language from real life. Compared to the majority of vision–language tasks, such as image captioning [1] and VQA [2], visual dialog [4], which involves multi-turn visual-grounded conversations, is more engaging and challenging. Two separate teams of researchers simultaneously published two types of visual dialog datasets. One is the Guess-what dataset that De Vries et al. [14] gathered. It is a goal-driven visual dialog. The questioner must correctly predict the visual target by asking a series of questions. The Oracle provides yes/no/n.a. answers. The other is a large-scale free-form visual dialog dataset called VisDial [4], where the questioner asks open-ended questions based on the image captions and dialog history to assist itself to better comprehend the visual contents; the answerer responds depending on the image and dialog history. In this paper, the second setting was used.

Various technical vantage points were used to study visual dialog tasks. Three baselines were initially proposed to go with the VisDial dataset: the hierarchical recurrent encoder (HRE) [4] included a dialog-RNN sitting on top of a recurrent block, which was capable of selecting relevant history from previous rounds; the late fusion (LF) [4] directly concatenated individual representations of the image, dialog history, and question followed by a linear transformation; and the memory network (MN) [4] treated each prior question–answer pair as a fact and used a SoftMax to obtain probabilities over the stored facts.

Most of the later techniques utilized various attention mechanisms. In each step of the reasoning process, ReDAN [6] calculated attention maps to the dialog history and visual objects. To manage the visual and textual information in a parallel and adaptive way, heterogeneous excitation-and-squeeze network (HESNet) [15] excavated multi-modal attention. Recursive visual attention (RvA) [16] updated visual attention while recursively browsing the dialog history to tackle the problem of visual reference resolution. In order to use the historical information in a balanced way, the reciprocal question representation learning network (RQRLN) [17] utilized transformer-based attention to reciprocally learn a new question representation through the intersections of two types of question representations with and without dialog history.

Other approaches concentrated on combining neural networks with graphical structural representation to capture the semantic dependence between cross-modal information. A visual graph was built by aligning vision–language for graph inference (AVLGI) [18]. It was used to enhance visual features guided by textual features, and a scene graph was then created to incorporate external knowledge. To discover the partially relevant contexts, a context-aware graph (CAG) [19] constructed a dynamic graph structure, which could be iteratively updated by applying an adaptive top-K message transmission mechanism.

Pretrained models were used in recent studies, and they performed admirably. Wang et al. [20] initialized the encoder with BERT to fuse of dialog history and visual contents in order to better leverage the pretrained language representations. Murahari et al. [21] utilized two-stream vision-language pretrained models (i.e., ViLBERT) for this task to improve the interactions between vision and language.

The aforementioned techniques address the problem of visual dialog from several angles. However, these extant methods employ a token or a sentence semantic granularity representation of multiple inputs, so the output still suffers from poor accuracy. In this study, we employ collaborative reasoning for the visual dialog task to gather more pertinent information from the semantic representation of each granularity.

3. Methodology

3.1. Problem Definition

The problem is stated as follows. Given image $I$, dialog history $H_t=\left\{C, h_1,\dots ,h_{t-1}\right\}$, and current question $Q_t$, where $C$ is the caption describing the image, and $h_i=\left(Q_i,A_i^{gt}\right)$ is the $i^{th}$ question–answer pair after concatenating question $Q_i$ and the ground-truth answer $A_i^{gt}$. The goal of the dialog agent is to infer the best answer to $Q_t$ by discriminatively ranking a list of answer candidates: $A_t=\left\{A_1^t,\dots ,A_{100}^t\right\}$.

3.2. MGSCRN

In this section, we formally describe our proposed method. Figure 3 illustrates the overall structure of MGSCRN. It consists of three main modules. First, in the multi-granularity textual representation (MGTR) module, initial token- and sentence-granularity semantic representations are obtained by adopting self-attention mechanisms of transformers and convolutional neural networks (CNNs), respectively. Second, the question-aware attention-refer (QAAR) module leverages the transformer-based cross-attention. It aims to build intersections between the question and dialog history encoded using the same granular semantic representations, which are obtained by the MGTR module. In this process, two types of dialog history distributions are learned: one representing the attention weights of all of the tokens in the dialog history, and another representing the attention weights of all of the question–answer rounds. To more accurately identify the historical information associated with the question, one distribution is modified by the other during training. Consequently, even if one distribution deviates from the truth, the degree of deviation will be reduced via the corrections offered by the other distribution. Thus, the QAAR module collaboratively reasons the question-related information from the dialog history by leveraging different semantically granular representations of the current question and historical dialog. Third, the visual-aware attention alignment (VAAA) module learns the semantic alignments between the visual features and each representation of the question-fused history obtained from the previous module; hence, two types of distributions over visual objects are learned simultaneously. In the same way as the QAAR module, the VAAA module allows these two distributions to collaboratively assess the question-related visual objects. The following subsections discuss each module in detail.

3.2.1. MGTR

To collaboratively reason through the question and dialog history representations with different levels of granularity, the MGTR module adopts transformer-based attention mechanisms [22] and convolutional neural networks (CNN) to obtain the initial token- and sentence-granularity semantic representations. We first embed each token in the current question, $Q_t$, as $Q_t^G=\left[q_{t,1}^G,q_{t,2}^G,\dots ,q_{t,n_Q}^G\right]$, using a pretrained GloVe-embedding layer [23], where $n_Q$ denotes the number of tokens in $Q_t$. We then use a long short-term memory (LSTM) to encode $Q_t^G$ into a token sequence, $Q_t^{lstm}=\left[q_{t,1}^{lstm},q_{t,2}^{lstm},\dots ,q_{t,n_Q}^{lstm}\right]$. Next, we feed $Q_t^{lstm}$ into a standard transformer attentive module (i.e., $AM(·)$), which is carried out by a multi-head attention layer (i.e., $MutiHead(·)$) and a position-wise fully connected feed-forward layer (i.e., $FFN(·)$). Taking query matrix $Q$, key matrix $K$, and value matrix $V$ as input, $AM(·)$ is defined as follows:

$$AM\left(Q,K,V\right)=FFN\left(MultiHead\left(Q,K,V\right)\right),$$

(1)

$$MultiHead\left(Q,K,V\right)=Concat\left(hea{d}_1,hea{d}_2,\dots ,hea{d}_{N_{head}}\right)W^C,$$

(2)

$$hea{d}_k=Softmax\left(\frac{Q{W}_k^Q{\left(K{W}_k^K\right)}^T}{\sqrt{d}}\right)V{W}_k^V,$$

(3)

where $W^C$ and $W_k^{*}$ are learned parameter matrices. By setting inputs $Q$, $K,$ and $V$ as $Q_t^{lstm}$ in $AM(·)$, we obtain the token-granularity semantic representation of the question, $Q_t^{self,token}=\left[q_{t,1}^{self},q_{t,2}^{self},\dots ,q_{t,n_Q}^{self}\right]$, after stacking $L$ transformer blocks:

$$Q_t^{self,token}=AM\left(Q_t^{lstm},Q_t^{lstm},Q_t^{lstm}\right).$$

(4)

To more accurately obtain a sentence-granularity semantic representation of $Q_t^{self,s}$, we employ a CNN instead of average pooling the transformer or bidirectional encoder representations from transformers (BERT) token-embedding outputs, following the approach of [24]. We denote $Q_t^{self,s}$ as follows:

$$Q_t^{self,s}=CNN\left(Q_t^{self,token}\right).$$

(5)

Similarly, for the dialog history, $H_t$, we first concatenate the image caption and $t-1$ question–answer pairs, and then initialize them with the same embedding layer as in the question. Next, we utilize another LSTM and $AM(·)$ to acquire a token-granularity semantic representation of the dialog history, $H_t^{self,token}=\left[h_{0,1}^{self},\dots ,h_{0,n_H}^{self},h_{1,1}^{self},\dots ,h_{1,n_H}^{self},\dots ,h_{t-1,1}^{self},\dots ,h_{t-1,n_H}^{self}\right]$, where $n_H$ denotes the number of tokens in each turn of the dialog history, $H_t$. Likewise, the sentence-granularity semantic representation of the dialog history, $H_t^{self,s}=\left[h_0^{self,s},h_1^{self,s},\dots ,h_{t-1}^{self,s}\right]$, is obtained by utilizing Equation (5).

3.2.2. QAAR

Differing from most of the extant works [4,6,17,19] that rely only on single-granularity representations to construct relationships between the question and dialog history, QAAR uses multi-granularity representations to collaboratively capture relevant historical information. To achieve this goal, the question guides the dialog history using sentence- and token-granularity semantic representations to fetch the history features relevant to the question. Two attention-weight distributions over the dialog history are computed simultaneously: all tokens and all rounds. Then, the distributions correct each other to accurately align the features of the question and dialog history items. To this end, we formally describe the QAAR module, which adopts $AM(·)$ to learn the correlated history features while considering multi-granularity information. The equations are as follows:

$$Q_t^{his,s}=AM\left(Q_t^{self,s},H_t^{self,s},H_t^{self,s}\right),$$

(6)

$$Q_t^{his,token}=AM\left(Q_t^{self,token},H_t^{self,token},H_t^{self,token}\right),$$

(7)

where $Q_t^{his,s}$ and $Q_t^{his,token},$ respectively, represent the question with fused sentence- and token-granularity. When learning the representations of $Q_t^{his,s}$ and $Q_t^{his,token}$, the SoftMax function existing in $AM(·)$(i.e., Equations (6) and (7)) produce two types of distributions over the dialog history. Here, we take these out and denote them as $P_{his}^s=\left[p_0^s,p_1^s,\dots ,p_{t-1}^s\right]\in {ℛ}^t$ and $P_{his}^{token}={\left\{\left[p_{0,1}^{i,token},\dots ,p_{0,n_H}^{i,token},p_{1,1}^{i,token},\dots ,p_{1,n_H}^{i,token},\dots ,p_{t-1,1}^{i,token},\dots ,p_{t-1,n_H}^{i,token}\right]\right\}}_{i=1}^{n_Q}\in {ℛ}^{n_Q\times \left(t\times n_H\right)}$, respectively:

$$P_{his}^s=Softmax\left(\frac{Q_t^{self,s}{W}_k^{Q^s}{\left(H_t^{self,s}{W}_k^{H^s}\right)}^T}{\sqrt{d}}\right),$$

(8)

$$P_{his}^{token}=Softmax\left(\frac{Q_t^{his,token}{W}_k^{Q^{token}}{\left(H_t^{self,token}{W}_k^{H^{token}}\right)}^T}{\sqrt{d}}\right),$$

(9)

where $P_{his}^s$ represents the attention weights on $t$ rounds of dialog history, and the $i^{th}$ row of $P_{his}^{token}$ represents the attention weights on all of the tokens that possibly have dependencies to the $i^{th}$ token in the question.

To better explore the question-related tokens and sentences in the dialog history, we dynamically modify and update $P_{his}^s$ and $P_{his}^{token}$ under their mutual guidance during training. To make the deterministic distribution, $P_{his}^s$, more accurate, we first deal with the $i^{th}$ row of $P_{his}^{token}$ by separately adding the weight values of tokens in the different rounds of the dialog history, which is defined as:

$$p_{his}^{i,token}={\left\{{\displaystyle \sum }_{k=1}^{n_H}{p}_{j,k}^{i,token}\right\}}_{j=0}^{t-1}\in {ℛ}^t,$$

(10)

where $p_{his}^{i,token}$ represents the different contributions of $t$ rounds of dialog history to the $i^{th}$ token in the question. Thus, $P_{his}^{Q,token}={\left[p_{his}^{1,token},p_{his}^{2,token},\dots ,p_{his}^{n_Q,token}\right]}^T\in {ℛ}^{n_{Q\times }t}$ indicates the different contributions of $t$ rounds of dialog history to each token in the question. Then, we add the weight values of each column of $P_{his}^{Q,token}$ to obtain $P_{his}^Q$, followed by normalization:

$$P_{his}^Q=Norm\left({\displaystyle \sum }_{k=1}^{n_Q}{P}_{his,k}^{Q,token}\right)\in {ℛ}^t,$$

(11)

where $P_{his}^Q$ represents the contribution of $t$ rounds of dialog history to the whole question from the perspective of aggregating token-granularity semantic information. Next, as shown in Equation (12), we modify $P_{his}^s$ with the help of $P_{his}^Q$:

$$P^s=Norm\left(P_{his}^s\odot P_{his}^Q\right)\in {ℛ}^t$$

(12)

where $\odot$ denotes the elementwise product. To assist $P_{his}^{token}$ in accurately assigning attention weights over all tokens from the dialog history, we also modify $P_{his}^{token}$ under the constraint of $P_{his}^s$, which is calculated as follows:

$$P^{token}={\left\{Norm{\left[P_{his,r}^s·P_{r{n}_H:\left(r+1\right)n_H}^{i,token}\right]}_{r=0}^{t-1}\right\}}_{i=1}^{n_Q}\in {ℛ}^{n_Q\times \left(t\times n_H\right)},$$

(13)

where $[·]$ denotes the scalar multiplication of vectors, and $P_{r{n}_H:\left(r+1\right)n_H}^{i,token}$ is the subset of the $i^{th}$ row of $P_{his}^{token}$, which represents the attention weights assigned by the $i^{th}$ token of the question to the all tokens in the $r^{th}$ round of the dialog history.

Finally, during collaborative reasoning, $P_{his}^s$ and $P_{his}^{token}$ are replaced with $P^s$ and $P^{token}$, respectively, to obtain $Q_t^{his,s}$ and $Q_t^{his,token}$ via the interactive transmitting of information (i.e., $P_{his}^s$ and $P_{his}^{token}$) in Equations (12) and (13). After stacking $L$ transformer blocks as with the MGTR module, we obtain two types of question representations: $Q_t^{his,s}$ and $Q_t^{his,token}$, which are obtained by leveraging multi-granularity representations of the question and dialog history to collaboratively select question-aware historical information.

3.2.3. VAAA

Given the output representations of the QAAR module, $Q_t^{his,s}$ and $Q_t^{his,token}$, the VAAA module collaboratively aligns them with the visual features. Here, the initial features are extracted by employing the Faster-CNN [25], pretrained using Visual Genome [26], denoted as $V^0\in {ℛ}^{m\times d_v}$, where $m$ and $d_v$ represent the overall number of visual objects in image $I$ and the dimension size of each visual object, respectively. We use a linear projection to map the dimensions of the object embeddings to those of the token. Then, the visual features are denoted as $V\in {ℛ}^{m\times d}$. We first establish the latent connections among visual objects in image $I$ via the $AM(·)$:

$$V^{self}=AM\left(V,V,V\right)\in {ℛ}^{m\times d}.$$

(14)

Then, we apply the visual-aware attention mechanism to align visual objects with the visually related tokens from the question. The visual representation, $V^{self}$, queries the token-granularity semantic representation, $Q_t^{his,token}$, to obtain a new visual representation, $V^{Q,token}$, that fuses the attentive token-granularity question features. We utilize cross-attention to compute $V^{Q,token}$ as follows:

$$V^{Q,token}=AM\left(V^{self},Q_t^{his,token},Q_t^{his,token}\right)\in {ℛ}^{m\times d}.$$

(15)

However, the visual features, $V^{Q,token}$, are obtained by attending to the most relevant question-token features, which are not target visual features for adequately answering $Q_t$, then, an object-difference attention mechanism is used to locate the question-relevant visual objects in the corresponding image, formulated as:

$$P_V^{g,token}=Softmax{\left(\left[V^{Q,token}\odot \left(V^{Q,token}-V^{self}\right)\right]W_V^{token}\right)}^T\in {ℛ}^{g\times m},$$

(16)

where $V^{Q,token}\odot \left(V^{Q,token}-V^{self}\right)\in {ℛ}^{m\times d}$ is the difference between $V^{Q,token}$ and $V^{self}$ representations as guided by $V^{Q,token}$. $W_V^{token}\in {ℛ}^{d\times g}$ is the parameter matrix to be learned with $g$ glimpses. The SoftMax function transforms $g$ glimpse results into an attention weight over all visual objects. The final distribution on all of the visual objects is $P_V^{token}=1/g{{\displaystyle \sum }}_{i=1}^g{P}_V^{g,token}$.

$P_V^{token}$ is obtained by finding clues from question tokens and comparing visual object differences. Note that it may inaccurately identify relevant visual objects, depending on the token semantically granular representation of the question. Therefore, we also use the sentence-granularity semantic representation of $Q_t^{his,s}$ to align it with $V^{Q,token}$, which is implemented by the question-aware attention:

$$P_V^s=Softmax\left(\frac{Q_t^{his,s}{W}_k^{Q^{his,s}}{\left(V^{Q,token}{W}_k^V\right)}^T}{\sqrt{d}}\right).$$

(17)

The obtained $P_V^{token}$ and $P_V^s$ jointly decide an appropriate distribution, $P_V^{token+s}$, over the visual objects as follows:

$$P_V^{token+s}=Norm\left(P_V^{token}+P_V^s\right).$$

(18)

Based on distribution $P_V^{token+s}$, the weighted visual object representation is formulated as follows:

$$v^{Q,token}={\displaystyle \sum }_{i=1}^m{P}_{V,i}^{token+s}{V}_i^{Q,token}.$$

(19)

Finally, we fuse the filtered visual feature, $v^{Q,token}$, with $Q_t^{his,s}$ (expressing the complete semantic intent of the question) as the final multimodal fused representation:

$$C_t=tanh\left(W_f\left[v^{Q,token};Q_t^{his,s}\right]\right),$$

(20)

where $[;]$ denotes the concatenation, yielding context representation $C_t$, which is then fed to the discriminative decoder to predict the answer to the current question, $Q_t$.

3.3. Discriminative Decoder

We encode the candidate answers in the same way as the questions and the dialog history. Here, we utilize the last hidden states of the LSTMs as candidate answer representations, $A_t={\left\{u_t^s\right\}}_{s=1}^{100}\in {ℛ}^{100\times d}$. The discriminative decoder ranks these candidates using the dot product operations on $A_t$ and the context representation, $C_t$. A probability distribution over the candidates is then obtained using the SoftMax function, denoted as:

$$P_t^{Dis}=Softmax\left(C_t{\left(A_t\right)}^T\right).$$

(21)

Multi-class cross-entropy loss is employed as the discriminative objective function, which is formulated as follows:

$${ℒ}_{Dis}={\displaystyle \sum }_{i=1}^{100}{y}_i{P}_{t,i}^{Dis},$$

(22)

where $y_i$ is the one-hot encoded label vector of the ground-truth answer.

4. Experimental Setup and Metrics

4.1. Datasets

We conducted experiments on the VisDial v1.0 dataset [4] to verify our proposed model. The train, validation, and test splits contained 1.23 M, 20 K, and 44 K dialog rounds, respectively. For the train and validation splits, each image had a 10-round dialog, whereas in the test split, a random set of question–answer pairs and a current question were presented alongside each image. The train split included 123K images taken from the MS-COCO dataset [27], and 2 K and 8 K images from Flickr were used for the validation and test splits, respectively. Additionally, each question followed a list of 100 candidate answer options, one of which was the ground truth.

4.2. Evaluation Metrics

To evaluate our model, we used a number of retrieval metrics in the manner of Das et al. [4]. Specifically, $Recall@k \left(k\in \left\{1,5,10\right\}\right)$ was used to determine where the ground-truth answer was located among the top k ranked responses. The mean rank $\left(Mean\right)$ was the average rank of the ground-truth answer. It is defined as:

$$Mean=\frac{1}{\left|Q\right|}{{\displaystyle \sum }}_{i=1}^{\left|Q\right|}Ran{k}_i,$$

(23)

where $Q$ was the set of all the questions, and $Ran{k}_i$ was the rank of the ground-truth answer.

The mean reciprocal rank $\left(MRR\right)$ was the reciprocal rank of the ground-truth answer, which was computed as follows:

$$MRR=\frac{1}{\left|Q\right|}{{\displaystyle \sum }}_{i=1}^{\left|Q\right|}\frac{1}{Ran{k}_i}.$$

(24)

The normalized discounted cumulative gain ($NDCG)$ was introduced to penalize lower-ranked answers with high relevance. It should be note that the $NDCG$ accepts many answers that are comparable as right, whereas other metrics only take the rank of a single answer. $NDCG$ was defined as:

$$NDCG@k=\frac{DCG@k}{IDCG@k},$$

(25)

where $DCG@k={{\displaystyle \sum }}_{i=1}^k\frac{rel\left(i\right)}{log\left(2+i\right)}$ represents the submitted ranking, $IDCG@k={{\displaystyle \sum }}_{i=1}^k\frac{2^{rel\left(i\right)}-1}{log\left(2+i\right)}$ represents the ideal ranking, the value $k$ is the number of candidate answers with a relevance score greater than 0, and $rel\left(i\right)$ is the relevance score of $i^{th}$ candidate answer.

4.3. Training Details

The open-source code from Das et al. [4] was used to implement our model via PyTorch. The words that appeared at least five times in the training split were added to the vocabulary that we created. The question and dialog history were truncated to 20 and 40, respectively, or were padded. GloVe word vectors were used to initialize these words, and all LSTMs were set with a single layer and 512-dimensional hidden units. For the transformer blocks, we set the number of heads to eight, the hidden size to 512 dimensions, and the number of transformer layers to six by stacking the transformer blocks. On one single Titan RTX GPU, our model was trained using the Adam optimizer. The learning rate was increased from $1\times {10}^{-4}$ to $5\times {10}^{-4}$ during the first epoch, and then decreased by 0.2 at epochs 8 and 10. The batch size was set to 16.

5. Experimental Results and Analysis

5.1. Comparison with State-of-the-Art Methods

5.1.1. Results on Test-Standard v1.0 Split

Under the discriminative decoder setting using VisDial v1.0, we initially contrasted our proposed model with those of previously published results, such as LF [4], HRE [4], MN [4], co-reference neural module network (CorefNMN) [28], graph neural network (GNN) [29], factor graph attention (FGA) [30], dual visual attention network (DVAN) [31], RVA [16], dual encoding visual dialog (DualVD) [32], history-aware co-attention (HACAN) [33], synergistic network [34], DAN [5], textual-visual reference aware attention network (RAA-Net) [35], HESNet [15], CAG [19], LTMI-LG [36], VD-BERT [20], and MVAN [10].

Our MGSCRN model outperformed various techniques according to the NDCG metric, as shown in

Table 1. Performance comparison on the test-standard dataset of VisDial v1.0 under the discriminative decoder setting. Better performance is indicated by “$\uparrow$” when the value is higher and by “$\downarrow$” when the value is lower. The best results are shown in bold.

Model	NDCG↑	MRR↑	R@1↑	R@5↑	R@10↑	Mean↓
LF	45.31	55.42	40.95	72.45	82.83	5.95
HRE	45.46	54.16	39.93	70.45	81.50	6.41
MN	47.50	55.49	40.98	72.30	83.30	5.92
CorefNMN	54.70	61.50	47.55	78.10	88.80	4.40
GNN	52.82	61.37	47.33	77.98	87.83	4.57
FGA	52.10	63.70	49.58	80.97	88.55	4.51
DVAN	54.70	62.58	48.90	79.35	89.03	4.36
RVA	55.59	63.03	49.03	80.40	89.83	4.18
DualVD	56.32	63.23	49.25	80.23	89.70	4.11
HACAN	57.17	64.22	50.88	80.63	89.45	4.20
Synergistic	57.32	62.20	47.90	80.43	89.95	4.17
DAN	57.59	63.20	49.63	79.75	89.35	4.30
RAA-Net	55.42	62.86	49.05	79.65	88.85	4.35
HESNet	57.13	63.21	49.70	80.63	89.65	4.19
CAG	56.64	63.49	49.85	80.63	90.15	4.11
LTMI-LG	58.55	64.00	50.63	80.58	90.20	4.12
VD-BERT	59.96	65.44	51.63	82.23	90.68	3.90
MVAN	59.37	64.84	51.45	81.82	90.65	3.97
MGSCRN	60.98	65.40	51.92	81.94	90.89	3.88

. Specifically, when compared with VD-BERT, our model improved the NDCG from 59.96 to 60.98 (+1.02) and achieved results that were competitive with most non-NDCG metrics. Note that VD-BERT employs pretrained BERT language models and unified discriminative and generative training settings to improve performance. We also noticed that the MGSCRN model performed better than MVAN across various metrics (e.g., NDCG from 59.37 to 60.98 (+1.61) and MRR from 64.84 to 65.40 (+0.56)).

To compare the MGSCRN to more sophisticated works, we provided our ensemble model’s performance on the blind test-standard split of the VisDial v1.0 dataset in

Table 2. Performance comparison of different methods of performing ensemble models on the test-standard dataset of VisDial v1.0. Better performance is indicated by “$\uparrow$” when the value is higher and by “$\downarrow$” when the value is lower. The best results are shown in bold.

Model	NDCG↑	MRR↑	R@1↑	R@5↑	R@10↑	Mean↓
Synergistic	57.88	63.42	49.30	80.77	90.68	3.97
DAN	59.36	64.40	59.90	81.18	90.40	3.99
CDF	59.49	64.40	50.90	81.18	90.40	3.99
MVAN	60.92	66.38	53.20	82.45	91.85	3.68
MGSCRN	62.25	67.42	53.99	82.96	92.14	3.60

. For comparison, we chose the Synergistic [34], DAN [5], consensus dropout fusion (CDF) [37], and MVAN [10] models, because their ensemble performance results are available in the literature. For our ensemble model, we averaged the results ($P_t^{Dis}$) of the discriminative decoders after training six independent models with random initial seeds. Overall, our ensemble model outperformed all of the others (e.g., 62.25 NDCG compared to MVAN’s 60.92, and 67.42 MRR compared to MVAN’s 66.38). This further validates the effectiveness of our MGSCRN model as it focuses not only on predicting ground-truth answers (e.g., indicated by MRR) but also on selecting more plausible answers as correct ones (e.g., indicated by NDCG).

To improve task performance even more, we added multitask learning that combined the discriminative and generative decoders during training. For a fair comparison, we conducted multitask learning experiments (see

Table 3. Performance comparison of different methods of performing multitask learning on the test-standard dataset of VisDial v1.0. Better performance is indicated by “$\uparrow$” when the value is higher and by “$\downarrow$” when the value is lower. The best results are shown in bold.

Model	NDCG↑	MRR↑	R@1↑	R@5↑	R@10↑	Mean↓
LTMI	60.92	60.65	47.00	77.03	87.75	4.90
ReDAN	61.86	53.13	41.38	66.07	74.50	8.91
MVAN	63.15	63.02	49.43	79.48	89.40	4.38
MGSCRN	64.13	63.87	50.28	80.26	90.02	4.11

). Following the visual dialog task [4], the generative decoder generates the next token, depending on the current one from the ground-truth answer, and utilizes log-likelihood scores to determine the rank of the candidates. We averaged the ranking outputs of the discriminative and generative decoders for evaluation. As shown in Table 3, our proposed model was effective in a multitask setting, achieving the best results on all metrics. Note that ReDAN [6] and MVAN [10] adopt the same method as ours, but the LTMI [8] uses only the discriminative decoder output for evaluation.

5.1.2. Results on VisPro and VisDialConv Datasets

To examine our model’s performance even further, we report the performance results of our model on those of VisPro and VisDialConv datasets [9], in which humans must browse the dialog history to predict the answers for the current question. We contrasted our model’s outcomes with those of MCA-I-HGuidedQ, MCA-I-VGH, and MCA-I-H from the work of Agarwal et al. [9], who incorporated the dialog history in various ways using VisPro and VisDialConv datasets.

Table 4. Results comparison on VisPro and VisDialConv datasets. Better performance is indicated by “$\uparrow$” when the value is higher and by “$\downarrow$” when the value is lower. The best results are shown in bold.

Model	NDCG↑	MRR↑	R@1↑	R@5↑	R@10↑	Mean↓
VisPro dataset
MCA-I-HGuidedQ	61.35	60.13	47.11	75.26	86.18	5.23
MCA-I-VGH	61.68	59.33	46.18	75.35	86.17	5.07
MCA-I-H	61.72	59.62	45.92	77.11	86.45	4.85
MGSCRN	62.57	61.56	48.73	78.06	88.11	4.70
VisDialConv dataset
MCA-I-HGuidedQ	53.18	62.29	48.35	80.10	88.76	4.42
MCA-I-VGH	55.48	58.45	44.54	74.95	86.19	5.18
MCA-I-H	53.01	61.24	47.63	79.07	87.94	4.77
MGSCRN	57.13	61.87	48.84	80.44	88.98	4.22

shows that our model continues to yield much better results over all of the metrics and that it has powerful generalization and prediction abilities owing to the collaborative intersection of semantic information with different granularity.

However, the series of MCA methods mentioned above only modelled interactions at the token level of information granularity based on the question and dialog history, which may lead to yield lower scores in most metrics.

5.2. Ablation Study

To evaluate the influence of our model’s key components, we performed an ablation study using the VisDial v1.0 validation split and the same discriminative decoder for all variations:

• Sentence-granularity semantic representations (SgSR): This model used only sentence-granularity semantic representations of the question and dialog history to acquire clues via the attentive module and CNN (Equations (4)–(6)). Then, it located question-related visual objects from $V^{self}$ using the question-aware attention (Equation (17));
• Token-granularity semantic representations (TgSR): This model used only token-granularity semantic representations of the question and dialog history to acquire clues via the attentive module (Equations (4) and (7)). Then, it located question-related visual objects from $V^{Q,token}$: we first use the object-difference attention mechanism (Equation (16)) to obtain distribution on all of the visual objects, $P_V^{token}$; second, the final question-relevant visual representation is computed by $v^{Q,token}={\displaystyle \sum }_{i=1}^m{P}_{V,i}^{token}{V}_i^{Q,token}$. The $v^{Q,token}$-fused question features is fed to the discriminative decoder to predict answers;
• TgSR without object-difference attention (TgSR w/o OdA): In this model, we performed self-attention instead of object-difference attention on $V^{Q,token}$ to locate question-related visual objects. The other settings matched those of TgSR;
• SgSR+TgSR: This model further combined SgSR and TgSR. Specifically, in the discriminative decoder, two probability distributions over answer candidates from SgSR and TgSR were added after normalization to produce the final distribution. It notably did not include collaborative reasoning historical information (CRHI) and collaborative reasoning visual object (CRVO) operations;
• MGSCRN: This is our full semantic model, which combined SgSR and TgSR with CRHI and CRVO operations.

Table 5. Ablation study of MGSCRN using the VisDial v1.0 validation split. Better performance is indicated by “$\uparrow$” when the value is higher and by “$\downarrow$” when the value is lower. The best results are shown in bold.

Model	NDCG↑	MRR↑	R@1↑	R@5↑	R@10↑	Mean↓
SgSR	59.85	63.65	50.88	80.76	89.20	4.37
TgSR	60.49	64.22	51.59	81.46	90.01	4.19
TgSR w/o OdA	59.58	63.45	50.93	80.80	89.38	4.32
SgSR+TgSR	61.25	64.98	52.68	81.92	90.46	4.01
MGSCRN	62.28	66.44	53.42	82.80	91.21	3.83

shows that both SgSR and TgSR considered only single-granularity semantic representations of the question and dialog history to reason the relational information among the question, dialog history, and visual objects. They achieved lower performance on all of the evaluation metrics compared with SgSR+TgSR. This illustrates the effectiveness of the joint decision-making that uses multi-granularity semantic information. Due to the introduction of CRHI and CRVO operations, MGSCRN bridged the multi-granularity semantic information to collaboratively reason the important information from the dialog history and visual objects, and it performed best on all of the metrics. This validates the method’s provision of more accurate clues by leveraging a collaborative interaction strategy with multi-granularity semantic information. TgSR gained an extra boost over TgSR w/o OdA, which demonstrates the advantage of explicitly comparing different representations of the same visual object to identify those most related to the question.

5.3. Qualitative Analysis

To further demonstrate the good interpretability of our model, we visualized the attention weights generated by QAAR and VAAA modules through the examples illustrated in Figure 4. For the QAAR module, two types of attention weights were shown. The first was attention weights $P^s$ for $t$ rounds of dialog history, obtained by modifying $P_{his}^s$ with the assistance of $P_{his}^Q$ (as in Equation (12)). Doing so revealed the relevance of different dialog history rounds. The second was $P_{i,j}^{token}$ over all tokens in the $j^{th}$ (selected by $P^s$) round of dialog history based on their relationship to the current question, where $P_{i,j}^{token}=Norm\left(P^{token}\left[i,j{n}_H:\left(j+1\right)n_H\right]\right)$. That is, $P_{i,j}^{token}$ reveals the relevance of different tokens in the $j^{th}$ round of dialog history to the $i^{th}$ token in the current question. Importantly, for both types of attention, we fetched the average attention weight over all heads computed by the topmost stacked transformer block. For the VAAA module, we first visualized the object-difference attention ($P_V^{token}$ in Equation (16)) and question-aware attention ($P_V^s$ in Equation (17)), and visualized the collaborative reasoning attention ($P_V^{token+s}$in Equation (18)) computed by CRVO. Here, we display the top three ranked visual objects for each attention in the rightmost column of Figure 4.

Two QAAR visualization examples are presented, for which our model accurately captured question-aware historical information at both sentence- and token-granularities. In the top example, when answering the current question, “Is he wearing shorts?”, to determine what “he” refers to, the model assigned a maximum weight to the “H1” round of dialog history (i.e., “Is the young boy a toddler? No.”), in which the token, “boy”, gained a relatively large weight by visualizing the weight of each token in “H1”, given by the token “he” in the question. This observation validates that our model accurately obtains clues from historical information, thus clarifying the semantics of the question.

In the VAAA module’s visualization, we observed that the question-related visual objects were located more accurately, benefiting from collaboratively reasoning the distribution over all of the visual objects using multi-granularity semantic representations of the question. For instance, when answering “Is there anything around the laptop?”, the object-difference attention of the VAAA module grounded “cat,” “screen”, and “pens” as the top three target objects. However, “cat” and “screen” should not have been considered based on the complete semantics of the question. However, VAAA’s question-aware attention located all of the visual objects around the laptop, including “table lamp”, “keyboard”, and “pens.” Although the OdA did not produce an accurate distribution over the visual objects, the CRVO method that combined OdA and QaA determined a suitable distribution that still ranked “table lamp”, “keyboard”, and “pens” as the top three target objects. Similar phenomena were verified in the top example.

To verify the effectiveness of the proposed model compared with the single granularity model, Figure 5 shows two example results on three models: SgSR model, TgSR model, and MGSCRN model. As observed in Figure 5, the MGSCRN model had a larger number of relevant answers on its list when compared to the top 10 ranking lists of the SgSR model. In the SgSR model, there were four and five relevant answers for each of the two cases. In contrast, our approach boosted the number to six and seven, respectively, which resulted in a higher NDCG score. Note that our model also outperformed the TgSR model in terms of NDCG. Additionally, in both cases, the ground-truth (GT) answer is ranked first by our proposed model. Through the above analysis, compared with a single granularity semantic representation model (SgSR/ TgSR), we validated our model performed better on the visual dialog task due to collaborative reasoning, using two different granularity semantic representations.

5.4. Error Case Analysis

In this section, we illustrated some of the zero-scoring examples in terms of the R@10 metric. These can be categorized into three groups based on the reasons for the errors. The first is common-sense reasoning. That is, when faced with the question, “What material is the table?” or “How tall is the man?” (Figure 6a,b), common-sense reasoning is needed. However, our model was confused even after accurately locating the target visual objects. The second was related to the image not containing the target visual objects. Because “traffic lights” did not exist in the corresponding image (Figure 6c), our model picked the best candidate, which led to incorrect answers. The third type relates to keyword matching. When modeling the intersections between the question and dialog history, our model memorized the keywords from the dialog history (e.g., “eyes”), thus ranking answers with keywords at the top. For example, “No, his eyes are open” and “No, it’s eyes are open” (Figure 6d). However, these answers are sometimes inaccurate.

6. Conclusions

In this paper, we proposed the MGSCRN visual dialog system. The QAAR module performs collaborative reasoning while leveraging a variety of granular semantic representations of a new question and an image’s dialog history to capture and analyze historical information. The VAAA module performs collaborative reasoning to facilitate visual objects’ grounding. We empirically validated our novel method on VisDial v1.0, VisPro, and VisDialConv datasets, achieving outstanding prediction accuracy and task efficacy.

In terms of applications, the proposed model is a magnificent implementation of AI-supported knowledge management and plays a crucial role in human–machine interaction, such as dealing with robotics applications and AI assistants. In terms of the limitations of the proposed method, our method fed the entire dialog history into the model, disregarding the dialog history’s temporal relationship and discussed topic changes, when modeling the interactions between the current question and dialog history. By memorizing the topic changes, the model may capture the historical information more accurately as the conversation moves forward. Note that some of the questions can be answered by simply seeing the image without access to the history information. The proposed model may be unable to correctly answer these questions due to excessive memory of irrelevant history information. In addition, the accuracy of locating target objects may be decreased because our model does not consider the positional relationships between visual objects.

In the future, we will consider the temporal relationship between many rounds of the dialog history to determine how the discussed topic changes, so that the model can more precisely capture the background knowledge relevant to the current question. To answer questions that are unrelated to dialog history, how to use dialog history in a balanced manner is another challenge. Additionally, we expect to incorporate spatial relationships between visual objects by understanding the prior dialog history.