Modality Information Aggregation Graph Attention Network with Adversarial Training for Multi-Modal Knowledge Graph Completion

Abstract

Multi-modal knowledge graph completion (MMKGC) aims to complete knowledge graphs by integrating structural information with multi-modal (e.g., visual, textual, and numerical) features and leveraging cross-modal reasoning within a unified semantic space to infer and supplement missing factual knowledge. Current MMKGC methods have advanced in terms of integrating multi-modal information but have overlooked the imbalance in modality importance for target entities. Treating all modalities equally dilutes critical semantics and amplifies irrelevant information, which in turn limits the semantic understanding and predictive performance of the model. To address these limitations, we proposed a modality information aggregation graph attention network with adversarial training for multi-modal knowledge graph completion (MIAGAT-AT). MIAGAT-AT focuses on hierarchically modeling complex cross-modal interactions. By combining the multi-head attention mechanism with modality-specific projection methods, it precisely captures global semantic dependencies and dynamically adjusts the weight of modality embeddings according to the importance of each modality, thereby optimizing cross-modal information fusion capabilities. Moreover, through the use of random noise and multi-layer residual blocks, the adversarial training generates high-quality multi-modal feature representations, thereby effectively enhancing information from imbalanced modalities. Experimental results demonstrate that our approach significantly outperforms 18 existing baselines and establishes a strong performance baseline across three distinct datasets.

1. Introduction

Knowledge graphs (KGs) represent real-world knowledge in the form of structured fact triples (head entity, relation, and tail entity) and have profound research significance and broad application potential in fields such as question answering systems [1], recommendation systems [2], and intelligent search [3]. However, with the rapid development of artificial intelligence, the limitations of traditional uni-modal knowledge graphs [4] are becoming increasingly apparent when confronted with the growing complexity of multi-modal data and application scenarios in the modern digital landscape. Uni-modal knowledge graphs struggle to comprehensively represent information that includes various modalities such as textual and visual content. These multi-modal data often provide richer contextual cues and cognitive foundations for artificial intelligence systems. This challenge has prompted both academia and industry to actively explore and develop multi-modal knowledge graphs (MMKGs) [5]. MMKGs are an innovative knowledge representation tool that combines the structured knowledge of standard KGs with multi-modal data. By integrating diverse perceptual data, MMKGs provide a more comprehensive and multi-dimensional cognitive foundation for artificial intelligence systems, showcasing significant potential in multi-modal applications.

However, despite the considerable scale achieved by many publicly available MMKGs, the incompleteness of MMKGs remains a prominent challenge due to the slow accumulation of multi-modal corpora and the continuous emergence of complex relations and entities. This incompleteness directly limits the widespread application of MMKGs in intelligent systems, making it a critical issue that requires urgent attention. To address this problem, researchers have proposed the task of MMKGC [6]. This task incorporates multi-modal information into knowledge graph completion (KGC) [7] models to more comprehensively address the missing information in MMKGC. Specifically, multi-modal information can serve as supplementary data to enrich the representation of entity embeddings in the MMKGC task, thereby improving completion performance and resulting in more accurate predictions.

Current mainstream MMKGC methods, such as those in [8,9,10,11], typically embed multi-modal entity information into separate vector spaces and integrate these embeddings through operations like concatenation or averaging to enhance entity representation. However, despite performance improvements, these methods have significant limitations. First, they ignore the imbalance of modality contributions, as different modalities offer varying degrees of relevance and information density. Fixed fusion strategies, like concatenation or averaging, assume equal importance for all modalities, leading to suboptimal utilization of key modalities. Second, there is an underutilization of multi-modal knowledge, as quality disparities among data sources result in incomplete modal information, limiting the representation of multidimensional entity attributes. Furthermore, these methods primarily emphasize textual and visual modalities, neglecting the latent value of numerical data embedded within text, which is essential for improving reasoning and enhancing semantic representation.

To overcome the limitations discussed above, this paper proposes the modality information aggregation graph attention network with adversarial training for multi-modal knowledge graph completion (MIAGAT-AT). MIAGAT-AT consists of two core modules: the modality information aggregation (MIAGAT) module and the adversarial training (AT) module. The MIAGAT module dynamically adjusts the weight of each modality based on the correlation between modality embeddings and the target entity. The AT module enhances the representational capacity of modality embeddings by introducing a generative adversarial network (GAN) training strategy. Our main contributions are summarized as follows:

• We propose a novel modality information aggregation graph attention network (MIAGAT) designed to dynamically integrate four modalities (structural, textual, visual, and numerical features). By assessing the importance of each modality relative to the target entity, MIAGAT employs an adaptive attention mechanism to optimize weight allocation, producing integrated entity embeddings with deeply enriched multi- modal features.
• We propose an adversarial training (AT) module that generates synthetic adversarial entities as training examples, substantially enhancing the model robustness and generalization capability in complex multi-modal scenarios and concurrently strengthening its ability to comprehend and represent multi-modal knowledge with greater precision and effectiveness.
• We evaluated the performance of MIAGAT-AT through comprehensive experiments and analyses on link prediction tasks across three public benchmarks. The experimental results demonstrate that MIAGAT-AT significantly outperforms 18 existing baseline models, demonstrating its effectiveness.

2. Related Work

2.1. Multi-Modal Knowledge Graph Completion

Current research in MMKGC aims to improve the representation of entities by integrating various data modalities, such as visual, textual, and structural features. Unlike traditional knowledge graph embedding (KGE) methods, which rely solely on structural triples, MMKGC approaches seek to leverage the complementary strengths of multiple modalities to enhance predictive accuracy and the richness of entity representations. Over the years, several methods have been proposed to address these challenges, each introducing innovative strategies to better integrate and utilize multi-modal information. One of the pioneering works in this field is IKRL [11], which incorporated the visual modality into knowledge graph embeddings by using attention mechanisms to integrate entity visual features. TBKGC [9] further advanced this by combining visual and linguistic modalities within a translation-based framework, allowing for more comprehensive multi-modal representations. TransAE [12] unified structural and multi-modal information through a multi-modal autoencoder, improving the fusion of inter-modal features. RSME [10] introduced a relation-sensitive mechanism that dynamically filters irrelevant visual information, achieving a balance between structural and multi-modal contributions. OTKGE [13] applied optimal transport theory, modeling multi-modal embedding alignment as a Wasserstein distance minimization problem, thereby preserving spatial consistency across modalities. VBKGC [14] integrated the pre-trained Visual-BERT model for deep multi-modal fusion and introduced twin negative sampling, a strategy specifically designed for multi-modal contexts, significantly enhancing multi-modal embedding alignment and representation. To further boost the performance of MMKGC, several enhanced negative sampling strategies have been proposed. MMRNS [15] leverages relation aware multi-modal attention to generate diverse and challenging negative samples, thereby improving embedding quality and representation robustness. MANS [16] introduced a modality-aware negative sampling technique, which overcomes the limitations of traditional methods in multi-modal knowledge graph embeddings by effectively aligning structural and multi-modal embeddings.

2.2. Adversarial Training in Knowledge Graph Completion

Adversarial training is a key technique for enhancing the robustness of machine learning models by incorporating adversarial examples during training. Widely applied across various domains, it effectively mitigates adversarial attacks and improves model reliability under perturbations. In MMKGC, adversarial training addresses challenges such as modality imbalance and noisy data. By generating adversarial samples to refine cross-modal embeddings, it strengthens the alignment and integration of structural, visual, and textual information, thereby improving prediction accuracy and representation learning. For instance, PUDA [17] applies adversarial training within a positive-unlabeled mini-max framework to tackle data sparsity, generating informative adversarial samples to boost model robustness. AdaMF-MAT [18] extends this approach to an adaptive multi-modal fusion framework, addressing modality imbalance by generating adversarial samples to enhance modality integration. NATIVE [19] advances the concept by employing collaborative adversarial training within a relation-guided, dual-adaptive-fusion framework, enabling efficient and flexible multi-modal integration for MMKGC. These methods underscore the crucial role of adversarial training in overcoming key MMKGC challenges, offering robust solutions for modality imbalance, noisy data, and data sparsity across diverse scenarios.

3. Methodology

In this section, we provide a comprehensive introduction to the proposed MIAGAT-AT model, focusing on three key components: modal feature extraction, the modality information aggregation (MIAGAT) module, and the adversarial training (AT) Module. The complete framework of the model is illustrated in Figure 1.

3.1. Problem Definition

A knowledge graph is rigorously defined as a directed graph $G=\{E,R,T\}$, where E and R represent the sets of entities and relations, respectively. The set $T=\left\{\right(h,r,t)\mid h,t\in E,r\in R\}$ consists of triples in the graph, where $r\in R$ denotes the relation between the head entity h and the tail entity t. In the context of a MMKG, let M represent the set of modalities, encompassing various forms of modal information, such as texts, images, and numerical data. For each entity $e\in E$, its corresponding modality embedding is denoted as $e_m$, where $m\in M=\{o,i,n\}$. Consequently, the embeddings for the text, image, and numerical modalities are represented by $e_o$, $e_i$, and $e_n$, respectively. Furthermore, the graph structure itself is considered an intrinsic modality for each entity, with structural information embedded in the triples T. The structural embedding for an entity refers to its representation, which is derived through an embedding layer, and is denoted as $e_c$. MMKGC methods seek to learn a fused representation of entities across modalities, which are subsequently embedded into a continuous, low-dimensional vector space. These embeddings are then leveraged in link prediction tasks, where the objective is to predict the missing head entity h or tail entity t in a given query ($(h,r,?)$ or $(?,r,t)$). During the prediction phase, the entity set E serves as the candidate set for possible entities.

3.2. Modal Feature Extraction

To effectively leverage multi-modal information in MMKGC, encoding each modality and extracting its relevant features are essential. For the textual, visual, and numerical modalities, we employ pre-trained models as modality encoders to capture feature representations from each modality, which are then utilized for the fusion of multi-modal information. In the implementation of modality encoding, we adopt BEiT, which is based on the Vision Transformer (ViT) architecture, as the visual encoder. For the textual modality, we utilize SBERT (Sentence-BERT), which is built upon the BERT architecture, as the textual encoder. For the numerical modality, we apply BERT, leveraging its powerful contextualization capabilities, as the numerical encoder. The extraction of the raw feature $f_m$ for entity e in modality m can be formally expressed by the following general equation:

$$f_m={\displaystyle \frac{1}{\left|S_m\left(e\right)\right|}}\sum _{s_{m,i}\in S_m\left(e\right)}M{E}_m\left(s_{m,i}\right)$$

(1)

where $S_m\left(e\right)$ denotes the set of modality m corresponding to entity e, $s_{m,i}$ represents a specific element within the set $S_m\left(e\right)$, and $M{E}_m$ refers to the modality encoder associated with modality m. The general formulation of Equation (1) was adopted from NATIVE [19] because the role of its pre-trained models aligns with that in our work, serving to extract raw modal features.

3.3. Modality Information Aggregation Module

In MMKGC tasks, the imbalance of modality information necessitates a thorough analysis of each modality. Different modalities have varying informational values, making it essential to accurately evaluate their contributions to the target entity. Current methods often use fixed fusion strategies that treat all modalities equally, neglecting the differences in feature representations and their actual relevance to the target entity. To address the mentioned issues, we propose the modality information aggregation graph attention neural network. Serving as a modality information aggregation module, this network employs a multi-head attention mechanism to dynamically assign varying weights to each modality based on the correlation between the modalities and the target entity.

In this module, the initial embedding matrix of the entities is represented as $H\in {\mathbb{R}}^{N_e\times d}$, where the i-th row corresponds to the embedding of entity $e_i$, $N_e$ denotes the number of entities, and d represents the feature dimension of the entity embeddings. The multi-head attention mechanism of the MIAGAT processes this matrix, calculating the attention weights for each modality with respect to the target entity. After the weighted aggregation, the updated entity embedding matrix $H^{\prime }\in {\mathbb{R}}^{N_e\times d^{\prime }}$ is obtained.

First, as proposed by GAT [20], a learnable weight matrix W is used for linear transformations to map the target entity embedding $e_c$ and each modality embedding $e_m$ into a high-dimensional space. The high-dimensional representations of the target entity and modalities are then concatenated to form a joint feature vector:

$$e_{cm}=\left[W{e}_c,W{e}_m\right]$$

(2)

where $e_{cm}$ represents the importance of modality (m) features to the target entity c, $W\in {\mathbb{R}}^{d\times d^{\prime }}$, d is the dimension of the original features, and $d^{\prime }$ is the dimension of the transformed features.

The attention mechanism employed in our approach is a single-layer feedforward neural network that is parameterized by a linear transformation vector ($\overrightarrow{a}\in {\mathbb{R}}^{2d^{\prime }}$), with a LeakyReLU activation function applied to introduce non-linearity:

$${\beta }_{cm}={\alpha }_m·LeakyReLU\left(\overrightarrow{a}\left(e_{cm}\right)\right)$$

(3)

where ${\beta }_{cm}$ represents the absolute attention coefficient of each modality m, while ${\alpha }_m$ is a learnable scaling factor used to dynamically adjust the contribution of each modality to the target entity.

To obtain the relative attention value of each modality, ${\beta }_{cm}$ is normalized using the Softmax function:

$${\alpha }_{cm}={\displaystyle \frac{exp({\alpha }_m·LeakyReLU\left({\overrightarrow{a}}^{\top }\left[W{e}_{c}W{e}_m\right]\right))}{{\sum }_{z\in M}exp({\alpha }_z·LeakyReLU\left({\overrightarrow{a}}^{\top }\left[W{e}_{c}W{e}_z\right]\right))}}$$

(4)

where ${·}^{\top }$ denotes the transpose operation, $\left|·\right|$ represents feature concatenation, and z is a specific element in the modality set M.

After normalizing the modality attention coefficients, we weight and combine the modality features through a linear transformation to produce the final output for each node, where

$$\tilde{e}=\sigma \left(\sum _{m\in M}{\alpha }_{cm}W{e}_m\right)$$

(5)

To enhance the learning stability of the self-attention mechanism, a multi-head attention mechanism is introduced. Therefore, the joint modality embedding of an entity can be represented as

$${\tilde{e}}^{\prime }=\sigma \left({\displaystyle \frac{1}{K}}\sum _{k=1}^K\sum _{m\in M}{\alpha }_{cm}^k{W}^k{e}_m\right)$$

(6)

where $\sigma$ denotes the non-linear activation function, ${\displaystyle \frac{1}{K}}$ represents the averaging operation, ${\alpha }_{cm}^k$ refers to the attention distribution computed by the k-th attention head, and $W^k$ represents the linear transformation matrix of the k-th attention head.

To strengthen the target entity’s representation and retain structural information, its structural embedding is combined with the multi-head attention output through a residual connection:

$$H^{″}=W_{T}H+H^{\prime }$$

(7)

where H is the initial structural modality embedding, the weight matrix $W_T\in {\mathbb{R}}^{E_i\times E_f}$$E_i$ is the dimension of the initial entity embedding, and $E_f$ is the dimension of the final entity embedding.

To evaluate triplet validity, we use the RotatE model [21] as the scoring function, which rotates the embeddings of entities and relations in complex space to model symmetric, antisymmetric, and cyclic relations:

$$\mathcal{F}(h,r,t)=h\circ r-t$$

(8)

where ∘ denotes the Hadamard product. During training, we use negative sampling to optimize model parameters, enhancing efficiency and distinguishing samples:

$$\begin{array}{cc}\hfill {\mathcal{L}}_{kgc}=& \sum _{(h,r,t)\in T}-log\sigma \left(\gamma +\mathcal{F}(h,r,t)\right)\hfill \\ \hfill & -\sum _{i=1}^{N}p(h_i^{\prime },r_i^{\prime },t_i^{\prime })log\sigma \left(-\mathcal{F}(h_i^{\prime },r_i^{\prime },t_i^{\prime })-\gamma \right)\hfill \end{array}$$

(9)

where $\sigma$ denotes the sigmoid function, $\gamma$ is a fixed margin hyperparameter, and $(h_i^{\prime },r_i^{\prime },t_i^{\prime })\in T^{\prime }$ with $i=1,2,\dots ,N$ represents the N negative samples of $(h,r,t)$. $p(h_i^{\prime },r_i^{\prime },t_i^{\prime })$ is the self-contradiction weight in the RotatE model, which is used to assess the reliability of adversarially generated triplets. Its formula is given by

$$p(h_i^{\prime },r_i^{\prime },t_i^{\prime })={\displaystyle \frac{exp\left(\tau \mathcal{F}(h_i^{\prime },r_i^{\prime },t_i^{\prime })\right)}{{\sum }_{j=1}^{k}exp\left(\tau \mathcal{F}(h_j^{\prime },r_j^{\prime },t_j^{\prime })\right)}}$$

(10)

where $\tau$ is a temperature parameter used to control the weight of negative triplets, balancing their impact on the training process.

3.4. Adversarial Training Module

The previous modality aggregation module generates the final target entity embeddings by dynamically adjusting and fusing modal embeddings. However, it only captures basic modality features and fails to enrich their expressive power with deeper semantic information. Inspired by generative adversarial networks, we propose a novel adversarial training (AT) module for MMKGC to address this limitation. The AT module consists of two components: the generator (G) and the discriminator (D).

3.4.1. Generator Design

The proposed generator G architecture consists of stacked residual blocks, each with two linear layers and a non-linear activation function. This design alleviates the vanishing gradient problem through skip connections, enhancing information flow. The input includes a noise vector for randomness and real multi-modal structured features for semantic diversity:

$$e_{gen}=G(z,e_{real})=R{B}_n\left(R{B}_{n-1}\left(\dots \left(R{B}_1(z,e_{real})\right)\right)\right)$$

(11)

where $e_{gen}$ represents the generated modality-specific entity embedding, which has the same feature shape as the real entity embedding $e_{real}$; $z\sim N(0,1)$ represents random noise; n is the number of residual blocks; and $R{B}_i$ represents the operation of the i-th residual block. The computation of each residual block can be uniformly represented as

$$RB\left(x\right)=SiLU\left(W_2·SiLU\left(W_1·x+b_1\right)+b_2+x\right)$$

(12)

where x represents the input features, $W_1$ and $W_2$ represent the weight matrices for the linear transformations, $b_1$ and $b_2$ are the corresponding bias terms, and $SiLU$ represents the non-linear activation function applied to each layer.

3.4.2. Discriminator Choice

In multi-modal knowledge graph completion, approaches such as AdamF-MAT [18] and NativE [19] utilize score functions as discriminators. In these methods, the score function not only serves as a tool for assessing the credibility of triples but also as a core mechanism for adversarial enhancement. Following this strategy, our model adopts the score function from the RotatE model as the discriminator.

In our model, the discriminator evaluates the authenticity of samples by computing the scores of generated triples, determining whether they align with the inherent relations in the knowledge graph. For a given triple $(h,r,t)$, the generator G produces entity embeddings $h^{\ast }$ and $t^{\ast }$, generating a set of triples, which forms the generated triple set $\mathcal{P}(h,r,t)$. Thus, in the adversarial training framework, the loss function is defined as

$${\mathcal{L}}_{at}=\sum _{(h,r,t)\in T}\left(-\mathcal{F}\left(h,r,t\right)+{\displaystyle \frac{1}{\left|\mathcal{P}\right|}}{\sum }_{(h^{\ast },r,t^{\ast })\in \mathcal{P}(h,r,t)}\mathcal{F}\left(h^{\ast },r,t^{\ast }\right)\right)$$

(13)

3.4.3. Optimization Objective

The optimization objective of the generator is to generate adversarial examples in such a way that the discriminator cannot accurately distinguish between the generated samples and the real samples. Specifically, the generator aims to minimize the discriminator’s probability of correctly classifying the generated samples while maximizing the probability that the discriminator incorrectly classifies the generated samples as real:

$$\underset{{\mathcal{L}}_G}{min}=-{\mathbb{E}}_{z\sim p_Z\left(z\right)}\left[logD\left(G\right(z\left)\right)\right]$$

(14)

where $G\left(z\right)$ denotes the generated sample by the generator G given the input noise z, $D(·)$ represents the discriminator function, $p_Z\left(z\right)$ denotes the noise distribution of the input to the generator, $\mathbb{E}$ represents the expectation operator, and z refers to the noise vector.

The optimization objective of the discriminator is to continuously improve its ability to distinguish between real samples and generated samples, thereby providing effective feedback signals to the generator. Specifically, the discriminator aims to maximize the probability of correctly classifying real samples while minimizing the probability of misclassifying generated samples:

$$\underset{{\mathcal{L}}_D}{max}=-{\mathbb{E}}_{x\sim p_{real}\left(x\right)}\left[logD\left(x\right)\right]-{\mathbb{E}}_{z\sim p_Z\left(z\right)}\left[log(1-D(G\left(z\right)\left)\right)\right]$$

(15)

where x denotes an entity sampled from the real data distribution $p_{real}\left(x\right)$, $D\left(G\right(z\left)\right)$ represents the probability that the discriminator classifies the generated sample $G\left(z\right)$, and z refers to the noise vector sampled from the latent space $p_Z\left(z\right)$.

4. Experimental Setup

4.1. Datasets

To evaluate the model’s performance on the multi-modal knowledge graph completion task, three widely used benchmark datasets were selected for the experiment: DB15K [22], MKG-W [15], and MKG-Y [15]. DB15K is a multi-modal knowledge graph constructed based on DBpedia [23], which leverages the rich structured triples from DBpedia and integrates multi-modal data such as visual, textual, and numerical information, providing a more comprehensive and multi-dimensional representation of real-world entities. MKG-W and MKG-Y are derived from Wikidata [24] and YAGO [25], respectively. Detailed information about these datasets is provided in

Table 1. Statistical information of the datasets.

Datasets	Entities	Relations	Train	Valid	Test
MKG-W	15,000	169	34,196	4276	4274
MKG-Y	15,000	28	21,310	2665	2663
DB15K	12,842	279	79,222	9902	9904

.

4.2. Baselines

To validate the effectiveness of the proposed model, a comparative analysis was conducted with current mainstream baseline models for KGC. The baseline methods include KGC models that focus solely on structural information, such as TransE [26], TransD [27], DistMult [28], ComplEx [29], and RotatE [21], as well as MMKGC models that integrate structural information with additional modality data, such as IKRL [11], TBKGC [9], TransAE [12], MMKRL [30], RSME [10], VBKGC [14], OTKGE [13], IMF [8], VISTA [31], and AdaMF-MAT [18]. Additionally, KGC models based on negative sampling, such as KBGAN [32], MANS [16], and MMRNS [15], were also considered. The experimental results for the uni-modal KGC models and MMRNS are adopted from the original MMRNS paper, while the data for IMF and VISTA in the MMKGC models are obtained from NATIVE [19]. For the remaining models, the data are sourced from AdamF-MAT [18].

4.3. Implemention Details

For each MMKG dataset, hyperparameter optimization was performed using a grid search strategy to determine the optimal configuration. The search space encompassed the following ranges: batch sizes $\left\{128,512,1024\right\}$, the number of training epochs $\left\{250,500,1000,1500\right\}$, the number of attention heads $\left\{1,2,3\right\}$, the number of residual blocks $\left\{2,4,6,8\right\}$, the number of negative samples $\left\{32,64,128\right\}$, and embedding dimensions $\left\{100,150,200,250,300\right\}$. In the final implementation, the embedding dimension was set to 250, the batch size to 1024, and the number of negative samples to 128. Both the learning rate and the regularization learning rate were fixed at 10⁻⁴, with a regularization coefficient of 0.0001. The model was trained for 1000 epochs with a missing modality rate of 0.8. The margin parameter was set to 12 for the DB15K dataset and to 4 for the MKG-W and MKG-Y datasets.

5. Experimental Results

5.1. Comparison with Existing Methods

Table 2. Comparison of link prediction performance, where MIAGAT-AT* leverages structure, image, and text modalities, while MIAGAT-AT utilizes structure, image, numerical, and text modalities.

Baselines	DB15K				MKG-W				MKG-Y
	MRR	H@1	H@3	H@10	MRR	H@1	H@3	H@10	MRR	H@1	H@3	H@10
TransE [26]	24.9	12.8	31.5	47.1	29.2	21.1	33.2	44.2	30.7	23.5	35.2	43.4
TransD [27]	21.5	8.3	29.9	44.2	25.6	15.9	33.0	40.2	26.4	17.0	33.6	40.3
DistMult [28]	23.0	14.8	26.3	39.6	21.0	15.9	22.3	30.9	25.0	19.3	27.8	36.0
ComplEx [29]	27.5	18.4	31.6	45.4	24.9	19.1	26.7	36.7	28.7	22.3	32.1	40.9
RotatE [21]	29.3	17.9	36.1	49.7	33.7	26.8	36.7	46.7	35.0	29.1	38.4	45.3
IKRL [11]	26.8	14.1	34.9	49.1	32.4	26.1	34.8	44.1	33.2	30.4	34.3	38.3
TBKGC [9]	28.4	15.6	37.0	49.9	31.5	25.3	34.0	43.2	34.0	30.5	35.3	40.1
TransAE [12]	28.1	21.3	31.2	41.2	30.0	21.2	34.9	44.7	28.1	25.3	29.1	33.0
MMKRL [30]	26.8	13.9	35.1	49.4	30.1	22.2	34.1	44.7	36.8	31.7	39.8	45.3
RSME [10]	29.8	24.2	32.1	40.3	29.2	23.4	32.0	40.4	34.4	31.8	36.1	39.1
VBKGC [14]	30.6	19.8	37.2	49.4	30.6	24.9	33.0	40.9	37.0	33.8	38.8	42.3
OTKGE [13]	23.9	18.5	25.9	34.2	34.4	28.9	36.3	44.9	35.5	32.0	37.2	41.4
IMF [8]	32.3	24.2	36.0	48.2	34.5	28.8	36.6	45.4	35.8	33.0	37.1	40.6
VISTA [31]	30.4	22.5	33.6	45.9	32.9	26.1	35.4	45.6	30.5	24.9	32.4	41.5
AdaMF [18]	35.1	25.3	41.1	52.9	35.9	29.0	39.0	48.4	38.6	34.3	40.6	45.8
KBGAN [32]	25.7	9.9	37.0	51.9	29.5	22.2	34.9	40.6	29.7	22.8	34.9	40.2
MANS [16]	28.8	16.9	36.6	49.3	30.9	24.9	33.6	41.8	29.0	25.3	31.4	34.5
MMRNS [15]	29.7	17.9	36.7	51.0	34.1	27.4	37.5	46.8	35.9	30.5	39.1	45.5
MIAGAT-AT*	35.7	26.0	41.4	53.2	36.7	30.0	39.8	48.9	38.8	34.7	40.7	45.9
MIAGAT-AT	37.0	28.5	41.9	53.0	36.7	30.0	39.8	48.9	38.8	34.7	40.7	45.9

summarizes the comparative results between the proposed MIAGAT-AT framework and existing KGC baselines in terms of MRR and Hits@N metrics, where MRR (mean reciprocal rank) denotes the average reciprocal rank of the correct entities and Hits@N represents the proportion of correct entities ranked among the top N predictions. Baseline results are sourced as detailed in Section 4.2. Experimental results demonstrate that MIAGAT-AT consistently outperforms all baseline methods across all evaluation metrics on the three datasets. Compared to the conventional structure-based KGC model RotatE [21], MIAGAT-AT achieves the most significant improvements on the DB15K dataset, with gains of 7.7% in MRR, 10.6% in Hits@1, 5.8% in Hits@3, and 3.3% in Hits@10. The experimental results demonstrate that incorporating multi-modal information with conventional structural representations significantly enhances the model’s semantic comprehension of knowledge, leading to substantial performance improvements in KGC tasks.

Against the state-of-the-art multi-modal model AdaMF-MAT [18], MIAGAT-AT achieves further improvements on DB15K, especially in MRR and Hits@1, with improvements of 1.9% and 3.2%, respectively. These results demonstrate MIAGAT-AT’s superior capability in accurate link prediction. Even when processing the commonly employed modalities (structure, visual, and text modalities), MIAGAT-AT consistently outperforms all baseline models, further underscoring its robust capability in multi-modal integration. Furthermore, comparisons with adversarial training models such as MMKRL [30], KBGAN [32], and MMRNS [15] validate the effectiveness of the proposed adversarial module in enhancing cross-modal fusion and representation learning.

5.2. Generalization Experiment

We introduce an adversarial training (AT) module to enhance the expressiveness of modality embeddings. To evaluate the generalization capability of this module, we apply it to existing methods such as TBKGC [9], AdaMF [18], and NATIVE [19]. In the configuration without adversarial training, the model proceeds directly to the decoder for scoring and ranking to accomplish knowledge graph completion. The initial experimental data for these methods, prior to the incorporation of the adversarial training module, originates from our reproduction results. The experimental results are presented in Figure 2.

The experimental results demonstrate that incorporating the adversarial training module leads to significant performance improvements across all metrics for the three evaluated multi-modal knowledge graph completion models, with particularly notable gains in Hits@1 and Hits@3. The adversarial training module enhances model performance through a generator–discriminator adversarial game mechanism: the generator continuously produces high-quality adversarial samples, while the discriminator improves its ability to distinguish real samples from generated samples. This iterative adversarial process enables the generated samples to progressively approximate the real data distribution while substantially improving the model’s robustness against perturbations.

5.3. Case Study

To evaluate the performance of MIAGAT-AT in multi-modal information fusion tasks, we conducted missing triple prediction experiments on the MKG-Y dataset and compared it with mean modal aggregation methods. The experimental results demonstrate that MIAGAT-AT achieves superior prediction performance, generating higher triple prediction scores and effectively prioritizing semantically relevant candidate entities. Specifically, the top three predictions of MIAGAT-AT show strong alignment with the head entities and relations in the missing triples.

As shown in

Table 3. Missing Triple Case 1.

Missing Triple: (Matt Damon, actedIn, ?)
Rank	MIAGAT-AT	Score
1	The Good Shepherd	2.21
2	Rounders	2.03
3	Behind the Candelabra	1.90
Rank	Mean	Score
1	The Good Shepherd	1.99
2	Rounders	1.74
3	J.K.Simmons	1.73

, all candidate entities in Case 1 are of the type “movie,” while in

Table 4. Missing Triple Case 2.

Missing Triple: (German Iran, isLocatedIn, ?)
Rank	MIAGAT-AT	Score
1	Kharqan Rural District	3.14
2	Bastam District	3.06
3	Kalat-e Hay-ye Gharbi	1.44
Rank	Mean	Score
1	Kharqan Rural District	2.94
2	Bastam District	2.84
3	Kalat-e Hay-ye Gharbi	1.08

, Case 2 consists entirely of “geographical location” entities. In contrast, the mean modal aggregation method exhibits comparatively weaker prediction performance. For the same set of candidate entities, this method yields lower triple prediction scores than MIAGAT-AT and includes irrelevant entities in its top-three predictions, such as “J.K. Simmons” (a person) in Case 1. Although both methods successfully predict the correct answers for the missing triples in Cases 1 and 2, MIAGAT-AT achieves higher prediction scores, which represent distance metrics between positive and negative samples. This enhanced discriminative capability enables more accurate identification of missing triple entities.

5.4. Ablation Study

To evaluate the contribution of each component in the multi-modal information aggregation module, we conducted a comprehensive ablation study by systematically removing (1) individual modality inputs (visual, textual, and numerical inputs) and (2) Modal Scaling Coefficients. Our experiments revealed that the absence of any of these components led to a consistent degradation in link prediction performance.The experimental results are shown in

Table 5. Ablation study results of the MIAGAT-AT model.

Module	Settings	MRR	Hits@1	Hits@3	Hits@10
MIAGAT	w/o Visual	31.3	18.7	39.2	53.1
	w/o Textual	31.4	19.3	39.1	52.9
	w/o Numerical	31.1	18.3	39.2	53.0
	w/o MS	31.5	19.0	39.3	53.1
	w/o AT	32.5	20.1	40.3	53.1
AT	Residual-CNN	36.2	27.1	41.4	53.1
	DenseResidual-CNN	36.0	26.9	41.3	53.0
	DenseResidual-MLP	36.9	28.4	41.9	52.8
	Residual-MLP	37.0	28.5	41.9	53.0

. The introduction of numerical modality provides additional constraints that significantly improve the model’s prediction accuracy. Experimental results demonstrate that removing this modality leads to noticeable performance degradation, particularly with MRR decreasing by 1.4% and Hits@1 dropping by 1.8%. These findings confirm that the numerical modality effectively narrows the prediction space, guiding the model to make more precise inferences within reasonable bounds and consequently enhancing its link prediction capability.

In the adversarial training module, we evaluated the impact of different generator architectures on link prediction performance by (1) replacing standard residual blocks with dense residual connections and (2) substituting multi-layer perceptrons (MLPs) with convolutional neural networks (CNNs). The experimental results demonstrated that the combination of residual blocks and MLPs achieved the best performance. Specifically, the multi-hop connections in dense residual structures introduced additional computational complexity and redundant information, making it difficult for the model to maintain efficient gradient propagation and parameter optimization during training, thereby limiting performance improvements. While CNNs excel at capturing local features and spatial relationships, their ability to model cross-modal feature interactions as generators was suboptimal, as they failed to fully leverage their potential in extracting global information from multiple modalities.

6. Conclusions

In this paper, we propose an improved MMKGC model called MIAGAT-AT, which addresses two key limitations in existing multi-modal knowledge graph completion approaches: (1) the neglect of modality imbalance among entities and (2) insufficient utilization of multi-modal information. The MIAGAT-AT framework comprises two core components: the modality information aggregation (MIAGAT) module and the adversarial training (AT) module. The MIAGAT module dynamically allocates attention weights across different modalities and integrates multi-modal information to generate joint embeddings. The AT module enhances modality representation learning through a generative adversarial network-based training strategy. To comprehensively evaluate the proposed model, we conducted experiments on three distinct MMKGC datasets: MKG-Y, MKG-W, and DB15K. The results demonstrate that our model outperforms a range of baseline methods, including traditional structured knowledge graph completion models and multi-modal knowledge graph completion models, thereby validating its effectiveness and robustness.