Visual Description Augmented Integration Network for Multimodal Entity and Relation Extraction

Abstract

Multimodal Named Entity Recognition (MNER) and multimodal Relationship Extraction (MRE) play an important role in processing multimodal data and understanding entity relationships across textual and visual domains. However, irrelevant image information may introduce noise that misleads the recognition of information. Additionally, visual and semantic features originate from different modalities, and modal disparity hinders semantic alignment. Therefore, this paper proposes the Visual Description Augmentation Integration Network (VDAIN), which introduces an image description generation technique that allows semantic features generated from image descriptions to be presented in the same modality as the semantic features of textual information. This not only reduces the modal gap but also captures more accurately the high-level semantic information and underlying visual structure in the images. To filter out the modal noise, we use VDAIN to adaptively fuse visual features, semantic features of image descriptions, and textual information, thus eliminating irrelevant modal noise. The F1 score of the proposed model in this paper reaches 75.8% and 87.78% for the MNER task and 82.54% for the MRE task on the three public data sets, respectively, which are significantly better than the baseline model. The experimental results demonstrate the effectiveness of the proposed method in solving the modal noise and modal gap problems.

1. Introduction

Named Entity Recognition (NER) and Relation Extraction (RE) are core tasks in social media information monitoring and evaluation. Given the rising influence of multimodal posts, multimodal information extraction technologies are more conducive to the supervision of public opinion on social media, such as Multimodal Named Entity Recognition (MNER) and Multimodal Relation Extraction (MRE). In recent years, food safety issues have received widespread attention on social media, particularly regarding the safety of staple crops such as rice and wheat [1]. The spread and attention of food safety incidents typically affect consumer confidence and purchasing decisions. Therefore, monitoring and analyzing food safety information on social media, especially the safety information of staple crops, has become increasingly important [2]. To more effectively mine this information, researchers have applied NER and RE techniques to the monitoring and evaluation of food safety incidents, as well as the analysis of food industry dynamics and food recall information [3,4]. On social media platforms, users comment, discuss, and share information about food safety issues, providing valuable data sources for researchers and regulatory authorities.

In early NER and RE research, the focus was mainly on text modality studies. Traditional approaches mainly have focused on constructing various efficient NER features, which were subsequently fed into different linear classifiers, such as the Support Vector Machines, Logistic Regression, and Conditional Random Fields (CRFs) [5,6,7,8,9]. With the development of deep learning techniques, the traditional one-hot representations have been replaced by word embedding methods, such as Word2Vec [10] and GloVe [11]. These methods capture semantic information of words in lower dimensions. In order to simplify the task of feature engineering, researchers have explored the combination of various deep neural networks with CRF layers to achieve word-level predictions—for instance, Recurrent Neural Network (RNN) [12] and Convolutional Neural Network (CNN) [13]. With the advent of the Transformer [14], many researchers adopted this model for NER and RE tasks [15,16,17]. Knowledge-enabled Bidirectional Encoder Representation from Transformers (K-bert) [18] and Enhanced Language Representation with Informative Entities (ERNIE) [19] combined pre-trained models with knowledge graphs to effectively learn additional entity knowledge.

Although the above models have achieved good performance in the text modality, they still face challenges when dealing with short, noisy data, such as Twitter datasets, especially encountering out-of-vocabulary words and complex abbreviations, as this type of text is often ambiguous. Many researchers have also proposed coping strategies. Von et al. [20] employed sentence-level features to enhance the recognition accuracy of named entities in short texts. Xie et al. [21] proposed a novel vocabulary substitution strategy based on constructing Entity Type Compatible (ETC) semantic spaces. Unknown words are replaced with ETC words found through Deep Metric Learning (DML) and Nearest Neighbor (NN) search. Soares et al. [22] fine-tuned pre-trained models with blank relationship data to automatically determine whether entities were shared between two relationship instances. Language Understanding with Knowledge-based Embeddings (LUKE) [23] introduced an entity-aware self-attention strategy and contextualized representation. Nevertheless, the aforementioned methods have not exploited the potential of visual information when dealing with multimodal data.

To compensate for the semantic context deficiencies in a single modality, Moon et al. [24] constructed a deep image network and a generic modality attention module that leverages images to enhance textual information and diminish irrelevant modalities. The model aimed to uncover correlations between multimodal data. Zhang et al. [25] designed an Adaptive Co-Attention Network (ACN), combining textual information with visual information processed by the Visual Geometry Group network (VGGnet) [26]. Subsequently, Arshad et al. [27] extended multidimensional self-attention techniques, simultaneously learning shared semantics between internal textual representations and visual features. Yu et al. [28] proposed an entity span detection technique to consider both word and visual representations of image and text information. To mitigate visual guidance biases introduced by images, Wu et al. [29] incorporated fine-grained images and designed a dense co-attention layer, introducing a co-attention mechanism between related image objects and entities. To further leverage image information, Chen et al. [30] proposed a novel neural network model that introduced image attributes and image knowledge. Zheng et al. [31] came up with the Adversarial Gated Bilinear Attention Neural network (AGBAN) which employs adversarial gating strategies to deal with the image-text alignment issue. To address the issue of image noise, Xu et al. [32] proposed a multimodal matching and alignment model to simultaneously handle modality differences and noise. Chen et al. [33] introduced a visual-prefix-based attention strategy (HVPNet) that progressively integrates visual features into each module of Bidirectional Encoder Representations from Transformers (BERT) for attention computation. Although these methods have improved the noise issue of multimodal information to some extent, there remains a significant modality gap between image features and text features originating from different modalities.

In order to bridge this gap and enhance the performance of multimodal models in various applications, it is crucial to examine the underlying structures and approaches of existing multimodal BERT models. Multimodal BERT models can be categorized into two main dimensions. (1) Structure—its main categories include single-stream and cross-stream. Single-stream multimodal models treat and fuse information from different modalities equally. Visual and textual inputs are processed by their respective encoders, resulting in visual and textual features that are subsequently combined into a unified sequence. The sequence serves as the input for the multimodal transformer, which autonomously learns the associations between cross-modal features. Representatives of this structure include VisualBERT [34], Universal Encoder for Vision and Language (Unicoder-VL) [35], and Object-Semantics Aligned Pre-training (Oscar) [36]. Cross-stream structures process visual and language information independently as two streams. Unimodal features were concatenated and fed into a deep feedforward neural network as input, with information exchange being achieved through cross-modal interaction transformation layers. Typical two-stream structure models include Contrastive Language Image Pre-training (CLIP) [37] and ViLBERT (short for Vision-and-Language BERT) [38]. (2) Pre-training tasks—pre-training tasks in vision-language models are key to interpreting and linking images as well as text within the artificial intelligence domain. However, most existing models are pre-trained based on image caption generation or conditional image generation, which emphasizes multimodal interaction, such as Dual-Level Collaborative Transformer (DLCT) [39], Graph Attention Networks (GAT) [40], Attentional Generative Adversarial Network (AttnGAN) [41], etc. Directly applying these models to multimodal named entity recognition (MNER) and multimodal relation extraction (MRE) tasks may not achieve ideal results because the core objectives of MNER and MRE are to utilize visual features for enhancing text information extraction, rather than predicting image content.

To address these challenges, many models have introduced image caption generation techniques. Accurately describing images is a challenging task, as it requires capturing and understanding various details within the image. Early image caption generation methods were primarily based on matching approaches, such as DFE [42] and VSE++ [43]. However, due to the limited generative capabilities of the description libraries, diversity is still difficult to achieve. Due to the ongoing advancements in deep learning and neural network technologies, numerous scholars have utilized these approaches in the area of image caption creation. Vinyals et al. [44] used CNN to encode images and LSTM as a text generator. Anderson et al. [45] employed Faster R-CNN [46] for staged object detection, improving the accuracy of image caption generation. Following the introduction of the Transformer model [14], Luo et al. [39] improved its attention mechanism by developing a Dual-Level Collaborative Transformer (DLCT) to enhance the complementarity between regional and grid features. Wang et al. [47] proposed the Geometry Attention Transformer (GAT) to further capture relevant geometric relationships of visual objects and extract spatial information within images. With the increase in large-scale models, inference speed has significantly decreased. Jin et al. [48] introduced an early exit strategy through layer-wise similarity to save computational resources while retaining state-of-the-art performance.

To effectively integrate visual features and semantic features generated from image captions, we propose a Visual Description Augmented Integration Network (VDAIN) in this paper. First, we input the raw image as well as the object detection image into the Vision Transformer (ViT) [49] and extract visual features from each ViT module as output. Furthermore, we apply image description generation techniques to the images and use A Lite BERT (ALBERT) [17] to extract semantic features from the generated image descriptions. Finally, we introduce the visual features and text vectors generated from image descriptions into the Visual Description Augmented Integration Network and predict entities and their relationships. In comparison to the shallow attention alignment method of MEGA, we have applied a BERT method fused with visual knowledge, which enhances the integration of visual and textual information. Compared with HVPNet, we adopt a visual description augmented integration mechanism to extract the textual semantic information from images, reducing the modality gap during the information fusion process. Furthermore, we employ ViT as the visual feature extractor, which is more similar to the BERT architecture, thereby improving the model’s ability to integrate cross-modal information to a certain extent. Therefore, our paper contributes to the existing research in three major aspects.

(1)

We employ an image description generation approach to extract key information and global summaries from visual content. The introduction of text features generated from image descriptions helps to reduce the modality gap, enhancing the robustness of MNER and MRE tasks.

(2)

We propose a Visual Description Augmented Integration Network for MNER and MRE. By adaptively integrating semantic features obtained from image descriptions with the visual features, the model aims to mitigate the error sensitivity caused by irrelevant features.

(3)

VDAIN achieves new state-of-the-art results on three public benchmark tests for MNER and MRE. Comprehensive evaluations confirm the practicality and efficacy of the proposed approach.

2. Materials and Methods

The model framework proposed in the text is shown in Figure 1 and consists of four main components: (1) image feature extraction; (2) image caption-based feature extraction; (3) visual description enhanced fusion; (4) integrated visual-text encoder. The model integrates image description generation techniques and introduces a visual description enhancement fusion network that cleverly fuses image features with semantic features extracted from the generated image descriptions to facilitate multimodal entity and relationship extraction. In the subsequent sections, we investigate methods to efficiently combine different features in multimodal datasets, aiming to minimize the interference of unrelated images and bridge the gap between modalities. This approach can multiply improve the performance of MNER and MRE.

2.1. Image Feature Extraction

Global image features primarily capture the comprehensive information within an image, including abstract concepts such as scenes, emotions, and themes, which can potentially weaken the learning signal strength. Conversely, local image features focus on capturing the distinct visual attributes of specific targets within the image, such as characters, objects, and shapes, thereby proving to be more beneficial for entity and relation recognition. Combining the two can fully exploit the potential value of global and local information. For the original image, we perform object detection and select images from the top n detection results with high confidence scores as input for subsequent local image feature processing.

Nowadays, ViT [49] is one of the most promising image neural network models, and its excellent fusion capabilities can lead to outstanding performance. To obtain high-quality visual features, we adopt ViT as the base network and extract image features by using the output of the last layer of each module. Specifically, we first resize the input image to 224 × 224 pixels and obtain a set of feature maps of different levels $\left[F_1,F_2,\dots ,F_c\right]$ after processing it through $ViT$.

$$[F_1,F_2,\dots ,F_c]=ViT\left(h_v\right)$$

(1)

where $h_v$ represents the input global image and corresponding object detection images, and $\left[F_1,F_2,\dots ,F_c\right]$ denotes the pyramid visual features. In order to map visual features to the same dimension to match the text encoder of the visual-semantic fusion, we apply a mapping function Ψ(·) to the features.

$$V_i={Conv}_{1\times 1}\left(MLP\right(P\left(F_i\right)\left)\right),i=\mathrm{1,2},\dots ,c$$

(2)

where $F_i$ denotes the visual features from different layers, and $P$ is the global average pooling layer that reduces the dimensions of the visual features through average pooling and linear transformation operations. A convolution layer with a 1 × 1 kernel size applies a linear transformation to each pixel without changing the feature map’s dimensions, aligning with the text encoder for visual-semantic fusion.

2.2. Image Caption-Based Feature Extraction

As shown in Figure 2, we incorporate image caption generation, which can assist in filtering out noise introduced during the image feature extraction process. Specifically, the semantic features of image caption generation and the textual information share the same modality. This approach not only reduces the modality gap but also more accurately captures the high-level semantic information and underlying visual structure of the image.

As one of the best-performing models in the image caption generation domain, we use OFA [48] pre-trained on the COCO dataset to carry out the image captioning task, converting image information into natural language. The input image is first encoded into feature maps using an encoder and then transformed into appropriate vectors and decoded by a decoder.

$$L=[L_0,L_1,L_2,\dots ,L_{n+1}],L_n\in {\mathbb{R}}^k$$

(3)

where $k$ represents the size of the vocabulary; $n$ denotes the length of the generated image caption; and $L_n$ denotes the word at position n in the sentence. In addition, we input the generated image caption into the pre-trained $Albert$ model:

$$C=Albert\left(L_n\right),C_i\in {\mathbb{R}}^d$$

(4)

where $C_i\in {\mathbb{R}}^d$ is the contextual representation generated by $L_n$. The generated vector is used as input for the subsequent fusion module facilitating semantic interaction between the multimodal features.

2.3. Visual Description Enhanced Fusion

Effectively reducing the noise introduced by visual features and narrowing the modality gap are significant challenges in current multimodal research. To address this issue, we design a Visual Description Augmented Integration Network that efficiently integrates visual features and semantic features generated by image captioning. Specifically, we incorporate a dynamic attention mechanism that computes multiple normalized vectors ${\alpha }_c^l$ and ${\beta }_k^l$ to determine the degree of integration for each visual feature and semantic feature generated by image captioning:

$${\alpha }_c^l=f(MLP(\frac{1}{c}\sum _{i=1}^{c}Pool\left(V_i\right)))$$

(5)

$${\beta }_k^l=f(\frac{1}{k}\sum _{i=1}^{k}MLP(C_i))$$

(6)

where $f$(·) denotes the activation function PReLU; $V_i$ is the visual feature of the different layers; $c$ is the number of layers; and $C_i$ is the semantic feature generated by image captioning, while $k$ is the text length. First, we reduce the visual feature $V_i$ to an appropriate dimension through average pooling and calculate the mean value. We further reduce the dimension of the visual feature using an $MLP$ layer and compute continuous values as path probabilities, implementing an adaptive visual feature gating strategy. Similarly, we apply a gating strategy for the semantic features generated by image captioning. Additionally, we use the activation function to process the normalized vectors ${\alpha }_c^l$ and ${\beta }_k^l$, obtaining the corresponding weight matrices ${\theta }_c^l$ and ${\eta }_k^l$:

$${\theta }_c^l=Softmax\left({\alpha }_c^l\right)$$

(7)

$${\eta }_k^l=Softmax\left({\beta }_k^l\right)$$

(8)

where $softmax(·)$ is the activation function. Finally, when integrating feature representations, we adopt an adaptive feature fusion approach. By using the calculated weight matrices ${\theta }_c^l$ and ${\eta }_k^l$, we perform a weighted summation of the corresponding layers’ visual and textual features to generate the fused feature representation:

$$V^l=\sum _{c=1}^C{\theta }_c^l{F}_c$$

(9)

$$T^l=\sum _{i=1}^k{\eta }_k^l{C}_i$$

(10)

where $V^l$ is the fused visual feature; $F_c$ is the visual feature from the c-th layer; $T^l$ is the fused semantic feature; and $C_i$ represents the $i$-th textual vector. Formally, for the $l$-th layer visual-semantic feature of BERT, we obtain it through the following concatenation operation:

$$D_f^l=[V_g^l;V_{o_1}^l;\dots ;V_{o_m}^l;T^l]$$

(11)

where $V_g^l$ and $V_{o_m}^l$ represent the visual features of the original image and the object detection image, respectively, and $T^l$ denotes the semantic features of the image.

2.4. Integrated Visual-Text Encoder

We perform multi-modal information fusion on visual features, semantic features generated from image descriptions, and textual semantic features, which are jointly fed into the encoder. Specifically, in the self-attention mechanism of BERT, text embeddings are first mapped to query (Q), key (K), and value (V) vectors. The process projects the contextual representation of text embeddings into a specific vector space:

$$Q=C^{l-1}{W}_q^1,K={C^{l-1}W}_k^1,V={C^{l-1}W}_v^1$$

(12)

where $C^{l-1}\in {\mathbb{R}}^{n\times d}$ represents the contextual representation of text embeddings. For that matter, a set of linear mappings $W_k^2,W_v^2\in {\mathbb{R}}^{d\times 2\times d}$ is applied to project the hierarchical visual features and semantic features generated by image description into the self-attention module, aligning them within the same embedding space as the textual representation.

$$K^{\prime }={D_f^{l}W}_k^2,V^{\prime }={D_f^{l}W}_v^2$$

(13)

where $D_f^l$ represents the concatenated matrix of visual features and semantic features generated from image descriptions, and $K^{\prime },V^{\prime }\in {\mathbb{R}}^{h\omega (m+1)\times d}$ are prefix vectors containing hierarchical visual features and semantic features generated from image descriptions. $h\omega (m+1)$ denotes the length of the visual features and semantic features generated from image descriptions, and $m$ refers to the number of input images and target detection images.

$${{Fused}_{Attention}}^l=softmax(\frac{Q[K^{\prime };K{]}^T}{\sqrt{d}})[V^{\prime };V{]}^T$$

(14)

2.5. Classifier

2.5.1. Named Entity Recognition

To enforce the structural correlation among labels during sequence decoding, we follow the methods presented in the research [24,27] and adopt a CRF decoder to accomplish the NER task, as illustrated in Figure 3a.

We utilize the output from BERT’s final layer as input, which undergoes processing through a feed-forward neural network layer before being processed in the CRF layer to obtain BIO tags for the MNER task. The probability formula between the label sequence $y$ and the MNER target is as follows:

$$p\left(y|H^L;W_{CRF}\right)=\frac{{\prod }_{i=1}^n{\psi }_i\left(y_{i-1},y_i,H^L\right)}{{\sum }_{y^{\prime }}{\prod }_{i=1}^n{\psi }_i\left(y_{i-1}^{\prime },y_i^{\prime },H^L\right)}$$

(15)

$$\mathcal{L}{\left(W_{CRF}\right)}_{ner}=-\sum _{I=1}^M\log (p(y^{\left(i\right)}|RE(X^{\left(i\right)},K^{\prime },V^{\prime });W))$$

(16)

where $H^L$ represents the final representation of the encoder, and $RE(·)$ denotes the fusion function between visual and textual features. Furthermore, $W_{CRF}$ represents a set of parameters defining the potential functions and weight vectors associated with label pairs for the task.

2.5.2. Relation Extraction

As illustrated in Figure 3b, we concatenate the identifiers of two entities and then classify the relationships between entities using a $softmax(·)$ classifier. The corresponding mathematical formula is as follows:

$$p\left(r|X\right)=Softmax(W{H}_{\left[CLS\right]}^L)$$

(17)

where $W$ represents the trainable parameters. With the help of the $\left[CLS\right]$ token, we can infer the probability distribution over the class set $Y$. For the MRE task, we employ the cross-entropy loss method:

$${\mathcal{L}}_{re}=-\sum _{I=1}^M\mathrm{l}\mathrm{o}\mathrm{g}(p(y^{\left(i\right)}|RE{(X}^{\left(i\right)},K^{\prime },V^{\prime })))$$

(18)

where $M$ represents the number of samples in the dataset, and $RE(·)$ denotes the fusion function between visual and textual features. In addition, $K^{\prime },V^{\prime }\in {\mathbb{R}}^{h\omega (m+1)\times d}$ are the prefix vectors containing hierarchical visual features and semantic features generated from image descriptions.

3. Results and Discussion

3.1. Data Set

We conduct experiments on the proposed model using three publicly available multimodal datasets: Twitter-2015, Twitter-2017, and MNRE. The two Twitter datasets cover four entity categories, including people, locations, organizations, and miscellaneous, as shown in

Table 1. The statistics of MNER datasets.

Entity Type	Twitter-2015			Twitter-2017
	Train	Dev	Test	Train	Dev	Test
Person	2217	552	1816	2943	626	621
Location	2091	522	1697	731	173	178
Organization	927	247	839	1674	375	395
Miscellaneous	931	220	720	701	150	157
Total	6166	1541	5072	6049	1324	1351
Num of sentence	4000	1000	3257	3273	723	723

. The distribution of samples in the training, validation, and test sets for these datasets is as follows: 4000/1000/3257 for Twitter-2015 and 3373/723/723 for Twitter-2017.

The MNRE dataset is expansive, encompassing diverse topics such as food safety, entertainment, and sports. It is characterized by 23 distinct relationship types. In addition, we have constructed a dataset related to the topic of food safety, named FOOD dataset. Further details about this data can be located in

Table 2. The statistics of MNRE and FOOD datasets.

Statistics	MNRE	FOOD
Number of Word	172,000	11,000
Number of Sentence	14,796	1525
Number of Instance	10,089	805
Number of Entity	20,178	1670
Number of Relation	31	7
Number of Image	10,089	928

. The distribution of samples in the training, validation, and test sets for these datasets is as follows: 12,247/1624/1614 for MNER and 1115/205/205 for FOOD datasets.

3.2. Experimental Environment

Our experiments are conducted on an Nvidia 3090 GPU. We select five different random seeds (1, 56, 3452, 4477, 1234) and perform each experiment five times, reporting the average performance. Then, we employ the BERT-base-uncased pre-trained model and the AdamW optimizer [50]. For learning, the rate of linear warm-up gradient updates is set to the maximum for the first 10% of the updates, while the weight decay is set to 0.01. Additionally, for the MNER task, we set the batch size to 8, the maximum sentence length to 128, and the learning rate to 3 × 10⁻⁵ and train the model for 15 epochs, with evaluation starting from the 10th epoch. In the MRE task, we set the batch size to 16, the maximum sentence length to 80, and the learning rate to 1 × 10⁻⁵ and train the model for 20 epochs, with evaluation starting from the 8th epoch. The assessment metrics employed in this research include accuracy, precision, recall rate, and F1 score [51].

3.3. Experimental Results of MNER and MRE Tasks

To validate the performance of our model, we conduct comparative analyses against several baseline models for both the MNER and MRE tasks.

We first compared our model with several classic pure text-based NER and RE models. The NER baselines contain (1) BiLSTM-CRF [12], which combines BiLSTM and CRF; (2) CNN-BiLSTM-CRF [13], which combines CNN and BiLSTM-CRF; (3) BERT [15], which employs a deep bidirectional Transformer architecture. The RE baselines involve (1) PCNN [52], which is a distant supervision relation extraction method; (2) MTB [22], which employs a training method maximizing cosine similarity; (3) PURE [53], which is a rule-based and text pattern-matching method.

Additionally, we compare our model with several competitive methods in the MNER and MRE domains. MNER baselines include (1) GVATT [54]; (2) AdaCAN [25]; (3) GVATT-BERT-CRF, which enhances the GVATT model by substituting the text encoder with BERT; (4) AdaCAN-BERT-CRF, which refines the AdaCAN model by replacing its text encoder with BERT; (5) UMT [28], which employs entity span detection techniques; (6) SMKG-Attention-CRF [30], which introduces image attribute modality; (7) HVPNet [33], which uses multiscale visual features as visual prefixes. MRE baselines include (1) VisualBERT [38], which inserts visual information into the input sequence of BERT; (2) BERT + SG [55], which uses two models to counter data bias; (3) UMT [28], which uses a unified multimodal transformer based on attention and gating mechanisms; (4) UMGF [56], which uses gating mechanisms and text-image alignment techniques; (5) MEGA [55], which uses a dynamic programming-based algorithm; and (6) HVPNet [33], which integrates multi-scale visual features into visual prefixes to alleviate modality noise.

As illustrated in Figure 4, we observe that during the training process, our models exhibit a stabilization of training and validation losses on both the Twitter-2015 and Twitter-2017 datasets after approximately 10 epochs. This observation implies that our models have attained a state of convergence on these datasets, yielding satisfactory performance. Similarly, in the case of the MNRE and FOOD datasets, the training and validation losses reached a stable state after around 15 epochs, suggesting that our models have also achieved laudable performance on this dataset.

In

Table 3. Overall MNER results.

Modality	Model	Twitter-2015			Twitter-2017
		P%	R%	F1%	P%	R%	F1%
Text	BiLSTM-CRF	68.14	61.09	64.42	79.42	73.43	76.31
	CNN-BiLSTM-CRF	66.24	68.09	67.15	80.00	78.76	79.37
	BERT	68.30	74.61	71.32	82.19	83.72	80.95
Text + Image	GVATT	73.96	67.90	70.80	83.41	80.38	81.87
	AdaCAN	72.75	68.74	70.69	84.16	80.24	82.15
	GVATT-BERT-CRF	69.15	74.46	71.70	83.64	84.38	84.01
	AdaCAN-BERT-CRF	69.87	74.59	72.15	85.13	83.20	84.10
	UMT	71.84	74.61	73.20	85.08	85.27	85.18
	SMKG-Attention-CRF	74.78	71.82	73.27	85.14	85.40	85.95
	HVPNet	73.87	76.82	75.32	85.84	87.93	86.87
	VDAIN	74.22	77.45	75.80	88.00	87.55	87.78

and

Table 4. Overall MNRE results.

Modality	Model	MNRE
		Acc%	P%	R%	F1%
Text	PCNN	72.67	62.85	49.69	55.49
	MTB	72.73	64.46	57.81	60.86
	PURE	73.05	64.50	58.19	61.22
Text + Image	VisualBERT	69.22	57.15	59.48	58.30
	BERT + SG	74.09	62.95	62.65	62.80
	UMT	74.49	62.93	63.88	63.46
	UMGF	76.31	64.38	66.23	65.29
	MEGA	76.15	64.51	68.44	66.41
	HVPNet	90.95	83.64	80.78	81.85
	VDAIN	93.92	82.44	81.25	82.54

, we evaluate the effectiveness of each model. Table 3 presents the evaluation results of the final performance of each model on the Twitter-2015 and Twitter-2017 datasets, while Table 4 shows the evaluation results of each model on the MNRE dataset. The evaluation results for the MNER task include the metrics of overall precision (P%), recall (R%), and micro-F1-score (F1%) for all models. For the MRE task, the evaluation results include the metrics of overall accuracy (Acc%), precision (P%), recall (R%), and micro-F1-score (F1%) for all models.

From the preceding data, the following can be inferred:

(1)

Adopting BERT as a text encoder significantly improves the performance of NER and RE tasks compared to traditional BiLSTM and CNN models. For instance, BERT achieves a notable increase in F1 scores on the Twitter datasets. In RE tasks, models with BERT as a text encoder (PURE and MTB) outperform other models (PCNN) with F1 value improvements of 5.37% and 5.73%, respectively. The improvements highlight the importance of pre-trained BERT in NER and RE tasks.

(2)

Incorporating image information significantly enhances the performance of NER and RE tasks. Multimodal methods generally perform better than pure text methods in these tasks. In NER tasks, GVATT employs image information and achieves a 6.38% and 5.56% increase in overall F1 values compared to BiLSTM-CRF on two datasets. Meanwhile, AdaCAN demonstrates improvements of 6.27% and 5.84%. In RE tasks, adding image modality with UMT and UMGT results in a 2.24% and 4.07% increase in F1 values compared to the single modality PURE. The improvements further confirm that integrating visual and textual information can generally improve the performance of NER and RE models, emphasizing the importance of combining visual and textual information.

(3)

By adopting a visual description augmented integration mechanism to reduce image noise and adaptively integrate visual features into BERT, the performance of the VDAIN method is significantly improved. In the two tables, our model outperforms methods such as UMT and MEGA, which employ shallow attention alignment, particularly achieving notable progress in the MNRE task. The finding emphasizes the importance of eliminating irrelevant visual noise and multimodal information fusion strategies. It may be the challenge of optimizing the integration of visual and textual information to enhance the performance of multimodal methods across various tasks.

(4)

Introducing image descriptions is more effective in extracting key image information and reducing modality gaps. We find that models using visual features combined with image descriptions to generate semantic features perform significantly better than HVPNet and MEGA, which only utilize visual features. VDAIN attains state-of-the-art results on all three datasets, showcasing the efficacy of our approach. Specifically, compared to the second-ranked HVPNet model, VDAIN shows a 0.48% increase in overall F1 scores on the Twitter-2015 dataset, a 0.91% increase on the Twitter-2017 dataset, and a 0.69% increase on the MNRE dataset. The experimental findings suggest that incorporating image description generation allows the model to capture essential image information and reduce modality gaps.

(5)

Figure 5 illustrates the performance of various models in terms of F1 score when solely utilizing datasets related to food safety. For this study, we selected data on the theme of food safety from three publicly available datasets and constructed a new dataset named FOOD datasets. We compared our model, VDAIN, with four other models: UMT, MEGA, and HVPNet. The results demonstrated that VDAIN outperforms the other multimodal models. This further validates the efficacy of our proposed method in entity recognition and relation extraction from food safety-related data on social media. Notably, despite the smaller volume of the FOOD datasets, our model still manages to enhance its performance under low-resource conditions.

Despite the semantic features generated from image descriptions and those from text information residing within the same modality and effectively narrowing modality gaps, there are instances where the interpretations between the two do not perfectly align. This disparity can be attributed to the inherent complexities associated with transposing visual context onto textual descriptors. Furthermore, although our visual description augmented integration mechanism proficiently mitigates image noise, it sometimes fails to fully filter out irrelevant or misleading visual cues. This can potentially introduce inaccuracies in entity recognition and relation extraction. As a result, refining the transformation and fusion mechanisms between these two modalities could be a constructive direction for optimizing our model’s overall performance.

3.4. Ablation Study

We conduct three sets of ablation experiments on the Twitter-2015, Twitter-2017, and MNRE datasets to validate the effectiveness of the image description generation strategy, the visual description augmented integration network, and the different levels of the image encoder.

3.4.1. Analysis of the Effectiveness of Image Caption Generation

We conduct ablation experiments to verify the effectiveness of image description generation on the overall performance. As shown in

Table 5. The effectiveness of image captioning.

Model	F1%
	Twitter-2015	Twitter-2017	MNRE
VDAIN	75.80	87.78	82.54
NoCap	75.28	86.42	79.84
NoObj	75.60	87.34	81.97

, we introduce two approaches for a more in-depth evaluation. Specifically, “NoCap” removes all image descriptions, and “NoObj” removes image descriptions of the object detection images while retaining the original image descriptions. We can observe that the introduction of image description generation achieved the best results, with the F1 scores on the three datasets being 0.52%, 1.36%, and 2.7% higher than “NoCap” and 0.2%, 0.44%, and 0.57% higher than “NoObj”. Therefore, image description generation is a key component of VDAIN’s effectiveness. In particular, image descriptions can summarize the key information as well as complex semantics of images and effectively prompt information extraction.

3.4.2. Analysis of the Effectiveness of Visual Description Augmented Integration Network

In this research, we present three approaches to demonstrate the efficiency of VDAIN. Specifically, the first method, “Direct”, aligns each layer of visual features from ViT and semantic features generated by image description directly with the corresponding layers in BERT. The second method, “Concatenate”, concatenates all visual features as well as the semantic features generated by image description and passes them through a linear layer. The third method, “Random”, randomly fuses each layer of visual features with the corresponding layers in BERT. Through these three methods, we can observe the effectiveness and flexibility of VDAIN in handling multimodal data.

As illustrated in

Table 6. The effectiveness of Visual Description Augmented Integration Network.

Model	F1%
	Twitter-2015	Twitter-2017	MNRE
VDAIN	75.80	87.78	82.54
Direct	74.75	85.87	79.38
Concatenate	74.92	85.91	78.75
Random	74.38	85.79	79.56

, we note that the F1 scores on the three datasets, when utilizing the VDAIN, are 1.05%, 1.91%, and 3.22% higher than the “Direct” method. It is also 0.88%, 1.87%, and 3.79% higher than the “Concatenate” method, and 1.42%, 1.99%, and 2.98% higher than the “Random” method. Therefore, the visual description augmented integration network is a core component in achieving outstanding performance. Specifically, the visual description-enhanced fusion mechanism can adaptively merge various image features with the semantic features generated by image descriptions and match them with each layer of the BERT model. The strategy helps improve the performance of multimodal data processing and reduce the modality gap, further enhancing the understanding of images and key information.

3.4.3. Analysis of the Effectiveness of Utilizing Various Layers of Image Encoder

To examine the impact of varying levels of image encoders, we carry out six ablation experiments. Specifically, the model Layer (1–12) uses 12 layers of the image encoder, and Layer (1–11) makes use of layers 1 to 11. Similarly, Layer (1–6) and Layer (6–12) apply layers 1 to 6 as well as layers 6 to 12. Furthermore, only the first layer and only the last layer are used for Layer (only1) and Layer (only12), respectively. As shown in Figure 6, without using the full-layer matching model, the F1 values of Layer (1–11), Layer (1–6), Layer (6–12), Layer (only1), and Layer (only12) decrease. The performance reduction is relatively small when using Layer (1–11), while it is more significant when using only the first or the last layer. This suggests that the visual features can provide support for semantic information in various levels of BERT, thereby enhancing the multimodal fusion effect. The F1 value decreases when only a portion of the layer is used. These experimental results further verify that visual information is distributed across the layers of the image encoder, and the visual knowledge contained in each layer is essential for improving the model’s performance during the multimodal fusion process. This also indicates that by analyzing and integrating different levels of visual features, semantic features generated from image descriptions, and semantic features of textual information, the relationship between images and text can be more effectively captured.

4. Conclusions

The MNER and MRE tasks play a vital role in accurately identifying and extracting entities and relationships related to food safety from a vast amount of social media text. This allows us to better monitor food safety opinion issues and provide early warning. Addressing the challenges posed by these tasks, we propose a model called VDAIN for MNER and MRE. The model employs visual information from a visually pre-trained model as an auxiliary, integrating image description generation and visual information through adaptive fusion. The approach aims to address modality noise and modality gap issues in MNER and MRE tasks. We conduct a large number of comparative experiments to validate the effectiveness of the proposed model and achieve optimal experimental results in three benchmark tests. This superior performance also extends to the FOOD datasets, where it showcases the best outcome. Moreover, we carry out several ablation experiments to demonstrate the effectiveness of each module of VDAIN. A key challenge in research is to explore how to more effectively alleviate the modality gap and filter crucial information in multimodal pre-trained models based on the Transformer framework. Despite our model’s success, there are limitations, such as occasional inaccuracies in entity recognition and relation extraction due to residual visual noise and modality translation challenges. Therefore, we leave these directions for future studies.