Fusion-Optimized Multimodal Entity Alignment with Textual Descriptions

Abstract

Multimodal knowledge graph entity alignment is a key basic task of knowledge fusion and integration, which is used to identify entities with semantic equivalent but different representation forms in different knowledge graphs. Previous entity alignment research has mostly focused on encoding and utilizing basic features such as entity names and attributes; however, it is difficult to comprehensively capture the rich semantic information of entities by solely relying on these basic features. To effectively overcome this limitation, this paper proposes a fusion-optimized multimodal entity alignment method, FMEA-TD. Compared with previous work, this method makes full use of the textual description information in the knowledge graph to provide rich supplements for entity features, thereby better capturing the entity semantics and solving the problems faced by relying solely on the entity’s own features. FMEA-TD is able to effectively fuse the entity’s own information and text description information through multimodal cooperation confidence, establish the interaction mechanism between them, and thus promote mutual collaboration between different modalities, which enhances the model’s ability to understand the semantic text. Experimentally validated, FMEA-TD outperforms current state-of-the-art baseline methods on public knowledge graph datasets.

1. Introduction

With the ongoing development of knowledge graph research, an increasing number of multimodal knowledge graphs, such as DBpedia [1] and YAGO [2], have been constructed and applied. These graphs integrate information from diverse modalities and languages, thereby establishing a crucial foundation for cross-source information sharing and advanced applications including knowledge retrieval, question answering, and machine translation [3,4,5]. Recent advances in incorporating multimodal data (e.g., images and audio) into knowledge graphs have spurred innovative approaches [6,7,8] in cross-modal knowledge representation and reasoning [9,10], ultimately demonstrating that multimodal knowledge graphs can significantly enhance the performance of various applications.

Since real-world knowledge is complex and extensive, a single knowledge graph often has missing information, making it difficult to comprehensively cover all types of knowledge. Indeed, we take advantage of the complementary nature of knowledge graphs from different sources in terms of their content to align entities in different knowledge graphs, thus enriching the coverage of knowledge graphs and optimizing the quality of knowledge graphs.

In previous studies, numerous multimodal entity alignment [11,12,13] approaches have been proposed, leveraging information from different modalities to construct more comprehensive and enriched knowledge graphs. For instance, EVA [11] presents a fully unsupervised framework that integrates visual cues with auxiliary information, providing strong visual signals for cross-knowledge graph entity alignment. Similarly, MCLEA [12] employs intra-modal contrastive learning and inter-modal alignment learning to encode entity representations using multiple modalities, thereby improving alignment performance. However, existing multimodal entity alignment methods still exhibit certain limitations. These methods mainly focus on the interactive relationship between different modalities, such as the association between visual and textual information, without making full use of the detailed textual description information provided by the text. Such descriptions often contain rich contextual information, which is essential for accurately understanding entities, their relationships, and their attributes. To further illustrate this point, we propose a multimodal entity alignment example, as shown in Figure 1. This example demonstrates the process of entity alignment of knowledge graphs in different languages of multimodal information. In this example, the text and visual information commonly seen in traditional alignment methods is included, and textual description information is also introduced to supplement the fine-grained semantics in the knowledge graph, thereby further improving the accuracy of entity alignment.

In order to make full use of and integrate textual description information, we propose a Fusion-Optimized Multimodal Entity Alignment with Textual Descriptions (FMEA-TD). While prior methods such as MCLEA and MEAFE have explored multimodal entity alignment through contrastive learning and cross-modal representation techniques, they primarily rely on high-level features from modalities like images and names. These models often overlook the fine-grained semantic cues embedded in textual descriptions, which are crucial for disambiguating entities—especially in cases with sparse structural or visual information. Our proposed model, FMEA-TD, addresses this limitation by fully integrating textual descriptions into the alignment process through a novel confidence-based fusion mechanism and a joint optimization framework, making it a substantial advancement over existing models. Some entities in the knowledge graph may lack sufficient structured information (such as name, attributes) for entity alignment. By introducing description information, the model can cover more knowledge about these entities, thereby improving the comprehensiveness and completeness of alignment. Since the attention mechanism used in previous multimodal feature fusion approaches focuses more on the correlation between modalities (such as the degree of match between text and image semantics), it assigns weights by calculating the similarity between different modal features. For example, an erroneous description may accidentally match a certain image, causing attention to mistakenly assign it a high weight, which will introduce noise. In contrast, we propose a new multimodal fusion strategy based on the confidence mechanism that pays more attention to the reliability of the modality itself (i.e., the probability of availability of the current modal information), thereby ensuring that the fusion result relies on more reliable information. For example, if the image modality is clear and complete, its confidence can be close to 1, which dominates the modal fusion. Our contributions are as follows:

• The FMEA-TD method focuses on leveraging the rich semantic information embedded in the textual descriptions of entities, including their attributes, relationships, and contexts. This textual information often provides valuable contextual support for other modalities, such as visual and speech data, helping the model to more accurately understand and represent the entity’s overall semantic characteristics.
• For information fusion in the multimodal knowledge graph, we employ a confidence predictor to calculate the confidence level of each modality. The collaborative confidence level across modalities not only reflects the quality of information from each individual modality but also considers the influence of all modalities, dynamically adjusting their weights to enhance the model’s robustness and generalization capabilities.
• FMEA-TD established a joint optimization of loss function (JOF), which improved the Inter-modal Alignment Loss (IAL). It not only considers the correlation between different modalities but also makes full use of the correlation between the rich semantic features contained in the entity textual description information and other modalities. By optimizing the loss function, FMEA-TD can effectively learn the deep association between the entity textual description information and other modal information, which helps the model to more accurately understand and represent the overall semantic features of the entity.

2. Problem Definition

In this section, we will detail our work on multimodal knowledge representation learning and multimodal entity alignment. We will give a comprehensive account of the proposed approach and its innovations from the perspective of problem definition and notation description. In the multimodal entity alignment work, entities refer to nodes of specific things (people, places, organizations, etc.) in the knowledge graph, and multimodal entity alignment refers to matching and associating the information of the same entity in different modalities (text, images, etc.) to establish the correspondence between them.

The mathematical form of the multimodal knowledge graph is denoted as $G=(E,N,A,R,V,S)$, where E stands for the set of entities in the knowledge graph, and $N,A,R,V,S$ are the name, attribute, relationship, visual, and description sets of entities, respectively. The different multimodal knowledge graphs are defined as $G_i=(E_i,N_i,A_i,R_i,V_i,S_i)$. Let us consider a cross-language knowledge graph as an example. If there exist multimodal knowledge graphs of different languages (Chinese and English, for example) $G_{zh}$ and $G_{en}$, We map the information of different modalities into the same semantic space by extracting the embedding, such that $I=\left\{(e_{zh}^i,e_{en}^j)|e_{zh}^i\equiv e_{en}^j,e_{zh}^i\in E_{zh},e_{en}^j\in E_{en}\right\}$, at which point $e_{zh}^i$ and $e_{en}^j$ are a pair of aligned entities. $\overrightarrow{E^g},\overrightarrow{E^n},\overrightarrow{E^a},\overrightarrow{E^r},\overrightarrow{E^i},\overrightarrow{E^s}$ in Figure 2 represent graph structure embeddings, embeddings of names, attributes, and relationships, visual embeddings, and textual description embeddings, respectively, which adjust the weights between modalities through dynamic confidence to generate a fusion embedding $\overrightarrow{E^f}$ of information from multiple modalities.

3. Related Work

Research on entity alignment can be categorized into unsupervised entity alignment and supervised entity alignment.

Unsupervised entity alignment does not require manually labeled seeds; SEU [14] is pre-mapped into a unified semantic space through a machine translation system or cross-linguistic word embeddings, and its textual characterization of entities retains the basic graph convolution operation for feature propagation while abandoning complex neural networks, significantly improving efficiency and interpretability. PRASE [15] learns KG embeddings through a probabilistic reasoning system in a probabilistic system called PARIS via entity mappings to learn KG embeddings, and it feeds the generated entity mappings and embeddings back to PARIS for augmentation. EVA [11] combines visual information with other semantic information to provide a completely unsupervised solution by utilizing the visual similarity of entities to create an initial seed dictionary (visual pivot). UDCEA [16] utilizes a deep learning multilingual encoder combined with a machine translator to encode the knowledge graph text to reduce the dependence on labeled data. It treats the alignment task as a dichotomous matching problem, then it employs the re-exchange idea to complete the alignment.

Compared with unsupervised methods, supervised entity alignment methods can utilize labeled samples to train the model, which improves the accuracy and reliability of the alignment. EAST [17] proposes an iterative representation learning process that can increase the size of the training set and generate higher-quality structural representations using confident EA pairs selected from each round. Furthermore, it points out that structural and textual representations are essentially two complementary viewpoints of EA, suggesting that they should be integrated in order to convey a more comprehensive signal. MMEA [18] is a multimodal knowledge embedding method used to generate entity representations of relational, visual, and numerical knowledge, respectively. After obtaining these knowledge representations, the multiple representations of the different types of knowledge are integrated. BERT-INT [19] is an interaction model that utilizes only side information. It does not aggregate neighbors but instead computes interactions between neighbors so that fine-grained matching of neighbors can be captured. MCLEA [12] not only aligns entities within modalities but also takes into account inter-modal interactions by designing two different losses that together model both intra-modal and inter-modal relationships. While previous approaches have considered many different types of data such as graph neighborhoods, text, and images, they have rarely considered textual description information. However, textual description information can often provide rich contextual information and complement this multimodal information. The FMEA-TD approach proposed in this paper makes full use of this textual description information and effectively fuses these descriptions with the underlying data information, enabling the model to better characterize the textual semantic features of entities.

4. Methodology

We propose a multimodal entity alignment method with textual descriptions, namely Fusion-Optimized Multimodal Entity Alignment (FMEA-TD). This method achieves alignment and matching of the same entity across different information sources by rationally utilizing the entity’s multimodal information, including text, images, and descriptions, and by employing a reasonable multimodal feature fusion system.

To help readers better understand the overall workflow of FMEA-TD, we provide a schematic summary of the methodology (Figure 2). The model operates in three main stages: (a) multimodal features—including graph structures, raw textual information, textual descriptions, and images—are extracted from multimodal knowledge graphs and encoded into vector representations; (b) these embeddings are fused using a collaborative confidence mechanism that dynamically weighs each modality based on its reliability; and (c) the fused representation is optimized using a joint loss function composed of cross-modal alignment loss and similarity loss, which guides the model to align semantically equivalent entities across graphs.

4.1. Knowledge Embedding

Knowledge information from different modalities in multiple languages is mapped to a shared semantic space, including graph structure information, underlying textual information of the knowledge graph, textual description information, and visual features of entities. This heterogeneous information is fused [20] to better capture semantic associations among entities and improve the accuracy of cross-language and cross-modality entity alignment.

4.1.1. Graph Structure Embedding

Node embedding techniques based on Graph Attention Networks (GATs) [21,22] are applied to knowledge graphs to better capture the complex topology of relationships between entities. Since the importance of information embedded in the neighboring nodes of an entity varies, different weights must be assigned to them. The core idea of GAT is to use an attention mechanism to adaptively compute the importance weights between a node and its neighboring nodes, thereby enhancing the encoding of graph structural information in the node representation.

Specifically, GAT calculates an attention weight $a_{ij}$ for each node i to indicate the importance of the neighboring node j to the central node i.

$$a_{ij}=softma{x}_j(e_{ij})$$

(1)

$$softma{x}_j(e_{ij})={\displaystyle \frac{exp(e_{ij})}{{\sum }_{k\in N_i}exp(e_{ik})}}$$

(2)

where $e_{ij}=a(W\overrightarrow{h_i},W\overrightarrow{h_j})$.

$\overrightarrow{h_i}$ and $\overrightarrow{h_j}$ are the feature vectors of the master node and the neighbor nodes, respectively, W is the attention weight parameter to be learned, and a is a scoring function based on the attention mechanism of a single-layer forward-propagation neural network. The attention score $e_{ij}$ is normalized to obtain the final attention weight $a_{ij}$.

After obtaining all the required attentional weights $a_{ij}$, a weighted average is taken to obtain the embedding representation of the next layer under the center node i, where $N_i$ denotes the set of neighboring nodes of node i, which includes all nodes directly connected to node i, and $\sigma$ denotes a nonlinear activation function such as $ReLU$:

$$\overrightarrow{h_i^{\prime }}=\sigma ({\sum }_{j\in N_i}{a}_{ij}W\overrightarrow{h_j})$$

(3)

In order to obtain richer and more accurate neighborhood structure information, we use two-layer GAT (K = 2) to capture the multi-level topological information in the graph structure:

$$\overrightarrow{E_i^g}=\sum _{k=1}^K\sigma ({\sum }_{j\in N_i}{a}_{ij}^{k}W\overrightarrow{h_j})$$

(4)

While the first layer of GAT aggregates the information of the direct neighbor nodes, the second layer of GAT aggregates the neighborhood features in the two-hop range to obtain a wider range of structural information. Ultimately, the output of the second layer is used as a structural embedding of the neighborhood of the node.

4.1.2. Text Embedding

In addition to graph structure information, text information is also an important source of information. This cross-language textual information includes entity names, attributes, and relationships, which can be used to enhance the representation of entities since vanilla GAT operates on unlabeled graphs and the knowledge graph contains a large amount of textual information. We use the pre-trained GloVe [23] model to obtain textual features by averaging strings over entity names and attributes, which can provide important semantic information about the entities, compensating for the lack of using only graph structure information.

The text features obtained through the GloVe pre-trained model are further fed into a simple feed-forward neural network layer to obtain the final text embedding representation:

$$\overrightarrow{E_i^t}=W_t·u_i^t+b_t,t\in \left\{n,a,r\right\}$$

(5)

where $\overrightarrow{E_i^t}$ denotes the embedding vector of entity i under modality t; $W_t$ is a weight matrix for mapping the input vector $u_i^t$ to the output vector $\overrightarrow{E_i^t}$; and $b_t$ is a bias vector that provides an additional adjustment term for each modality.

4.1.3. Visual Embedding

The pre-trained model ResNet-152 [24] is used as a visual coder to extract the visual feature vectors of an image. We input image $v_i$ corresponding to entity $e_i$ into the pre-trained model. The output of the penultimate layer of the model is used as the feature vectors of that image, then these vectors are inputted into a feed-forward neural network that completes the mapping from the original image to the visual embedding.

$$\overrightarrow{E_i^v}=W_v·PVM(v_i)+b_v$$

(6)

4.1.4. Textual Description Embedding

In order to extract rich semantic information from the text description of entities, we use the Sentence-BERT model [25] to generate sentence-level semantic embeddings. The original BERT model is mainly oriented to word-level tasks, and its output is not directly suitable for calculating the similarity between sentences. SBERT introduces a pooling operation based on the pre-trained BERT, and it fine-tunes it on the sentence comparison task so that it can directly generate embedding representations suitable for sentence-level semantic comparison. In the entity alignment task, we input the text descriptions of entities in different languages into the pre-trained SBERT model, obtain their corresponding semantic embedding representations, and calculate the similarity between the descriptions to assist in determining whether the entities refer to the same real-world object, as shown in Figure 3.

Before being encoded by the SBERT model, the textual descriptions are normalized using a preprocessing pipeline. This includes lowercasing all characters, removing special symbols, and collapsing excessive whitespace. These normalized sentences are then tokenized by the WordPiece tokenizer embedded within SBERT to ensure consistent subword representations across languages. SBERT applies mean pooling on the output of BERT [26]; that is, taking the average of the representations of all tokens in the last layer and aggregating multiple token representations of a sentence into a vector, represented by $\overrightarrow{E_i^s}$. It can capture sentence information well, not just the [CLS] token. The sentence vector generated in this way can more accurately capture the overall characteristics of the sentence, which is expressed in the semantic space as follows: similar sentences correspond to closer vectors, while the distance between unrelated sentences is farther. The BERT representations (u and v) of the two input sentences (Sentence A and Sentence B) and the absolute difference between them ($\left|u-v\right|$) are concatenated together as the input of the Softmax classifier. The Softmax classifier is the final output layer, which obtains the probability matching output of each entity, and it selects the candidate entity pair with the highest probability as the output. The output of the Softmax classifier is probability $q_s\left(u,v\right)$, which represents the probability of entity alignment in the current output:

$$q_s\left(u,v\right)=Softmax\left(W·\left(u,v,\left|u-v\right|\right)+b\right)$$

(7)

4.2. Feature Dynamic Fusion

In the past, multimodal feature fusion often used the attention mechanism to weigh and sum the different modal embeddings and automatically learn the importance of each modality in the entity representation to obtain effective fusion embeddings. However, the attention mechanism focuses on inter-modal correlation, and it assigns the weights by calculating the feature similarity, which is prone to introducing noise due to some accidental matches, thus reducing the accuracy of modal fusion. On the other hand, the confidence mechanism focuses on the reliability of the modalities themselves, assigning higher weights to modalities with high confidence to ensure that the results of modal fusion can depend on reliable information. Therefore, we propose a multimodal confidence collaborator method that can dynamically adjust the weights of feature fusion (see Figure 4). The method not only considers the confidence of the current modality but also comprehensively captures the relationship between entities and contextual information, thus enhancing the effectiveness of feature fusion.

4.2.1. Uni-Confidence

To perform the uni-confidence calculation, we use a confidence predictor to obtain the uni-confidence, where we use the probabilities of the true class labels instead of the weights in the attention mechanism given the original uni-confidence $C_m$, which gives us the normalized confidence.

$$Uni-Confidenc{e}^m={\displaystyle \frac{e^{\frac{C_m}{T}}}{{\sum }_{j\in M}^M{e}^{\frac{C_j}{T}}}}$$

(8)

Here, T is the temperature parameter and M is the set of different modes.

4.2.2. Mul-Confidence

Since different uni-modal confidence levels interact across modalities to generate multimodal confidence levels that are interconnected with other modalities, we use the sum of the weights of the other modalities to construct the weights, which enable the different modalities to collaborate with each other.

$$Mul-Confidenc{e}^m={\displaystyle \frac{{\sum }_{i\ne m}^{M}logUni-Confidenc{e}^i}{{\sum }_{j\in M}^{M}logUni-Confidenc{e}^j}}$$

(9)

As a result, we obtain a multimodal confidence collaborator, which fuses uni-confidence and mul-confidence into a final weight. The fusion method is $Final-Con{f}^m=Uni-Confidenc{e}^m+Mul-Confidenc{e}^m$. Knowledge from different modalities often provides complementary information. A multimodal confidence collaborator can capture and utilize this complementary information more comprehensively by integrating the confidence of each modality, enhancing the interaction between different modalities. Ultimately, we obtain multimodal fusion embedding as follows:

$$\overrightarrow{E_i^f}={\oplus }_{m\in M}^M({\displaystyle \frac{exp(e_m)}{{\sum }_{j\in M}exp(e_j)}}·Final-Con{f}_i^m·x_i^m)$$

(10)

where $x_i^m$ is the original representation (embedding) of entity x in modality m.

4.3. Joint Optimization of Loss Functions

4.3.1. Multimodal Entity Contrastive Loss

We use Contrastive Loss (CL) [27] (see Figure 2) to perform similarity judgment of entity pairs in different knowledge graphs using the known 1-to-1 entity alignment constraints. Except for the aligned entity pairs $I^i=\left\{(e_1^i,e_2^i)|e_1^i\equiv e_2^i,e_1^i\in E_1,e_2^i\in E_2\right\}$, the negative pairs are classified into two types, which are the unaligned pairs in the same knowledge graph $N_1^i=\left\{e_1^j|\forall e_1^j\in E_1,j\ne i\right\}$ and unaligned pairs $N_2^i=\left\{e_2^j|\forall e_2^j\in E_2,j\ne i\right\}$ in different knowledge graphs. These are two pairs of unaligned entities constrained to the joint embedding space to keep their embedding representations away from each other, thus improving the similarity of the aligned entities.

By encoding the inputs u or v into feature vectors, ${\tau }_m$ is a temperature coefficient used to adjust the distribution of similarity. ${\delta }_m(u,v)$ denotes the similarity score between two different inputs under mode m, defined as follows:

$${\delta }_m(u,v)=Softmax(f_m{(u)}^T{f}_m(v)/{\tau }_1)$$

(11)

The probability distribution of positive entity pairs in different modes is as follows:

$$q_m(I^i)={\displaystyle \frac{{\delta }_m(I^i)}{{\delta }_m(I^i)+{\sum }_{e_1^j\in N_1^i}{\delta }_m(e_1^i,e_1^j)+{\sum }_{e_2^j\in N_2^i}{\delta }_m(e_1^i,e_2^j)}}$$

(12)

CL can be expressed as follows:

$$L_m^{CL}=-E_{i\in B}log\left[{\displaystyle \frac{1}{2}}(q_m(I_1^i)+q_m(I_1^i))\right]$$

(13)

where $I_1^i=\left\{(e_1^i,e_2^i)|e_1^i\equiv e_2^i,e_1^i\in E_1,e_2^i\in E_2\right\}$ and $I_2^i=\left\{(e_2^i,e_1^i)|e_2^i\equiv e_1^i,e_1^i\in E_1,e_2^i\in E_2\right\}$.

Specifically, for a pair of already aligned entities, we include both orientations of the entity pair in the set of positive examples.

There are also contrast losses for text description embeddings and contrast losses for joint embeddings:

$$L_s^{CL}=-E_{i\in B}log\left[{\displaystyle \frac{1}{2}}(q_s(I_1^i)+q_s(I_1^i))\right]$$

(14)

$$L_f^{CL}=-E_{i\in B}log\left[{\displaystyle \frac{1}{2}}(q_f(I_1^i)+q_f(I_1^i))\right]$$

(15)

$q_s(·)$ and $q_f(·)$ are positive entity pair probability distributions for textual descriptions and joint features, respectively. This multimodal entity comparison loss can effectively use the existing entity alignment constraints to improve the entity alignment performance.

4.3.2. Cross-Modal Similarity Loss

In order to make the similarity between modalities increase, the embedded representations of different modalities are compared in terms of similarity, and the associations and correspondences between different modalities are learned to bring the knowledge of each modality in the knowledge graph closer (see Figure 2). We use Minkowski’s loss. Different types of data, such as text and images, have very different features and representations. These heterogeneous data often have a significant ‘modal gap’, which makes it difficult to compare and align them directly. Moreover, Minkowski’s loss is a very general metric that can be used to compare the similarity between different modalities of the embedded representations. Minkowski’s loss is a very versatile metric that can be used to align semantic representations of entities in different modalities into a common potential space, which not only takes into account the directional information of vectors but also the magnitude of the vector’s paradigm, better capturing the semantic correlation between heterogeneous data. It can not only enhance the consistency of the cross-modal knowledge representations but also reduce the semantic gap between different modalities, effectively by minimizing the loss function. We define the Minkowski loss as follows:

$$L_s^{ML}={({\sum }_{i\in B}{|\frac{1}{2}\left[v_s(I_1^i)-v_m(I_1^i)\right]+\frac{1}{2}\left[v_s(I_2^i)-v_m(I_2^i)\right]|}^p)}^{\frac{1}{p}}$$

(16)

$$L_f^{ML}={({\sum }_{i\in B}{\left|\frac{1}{2}\left[v_f(I_1^i)-v_m(I_1^i)\right]+\frac{1}{2}\left[v_f(I_2^i)-v_m(I_2^i)\right]\right|}^p)}^{\frac{1}{p}}$$

(17)

$$L_m^{ML}=L_s^{ML}+L_f^{ML}$$

(18)

where $v_s(·)$ denotes the embedding vector of the textual description, and $v_f(·)$ denotes embedding vectors for multimodal fusion. These capture the semantic information of the entity, including the contextual knowledge of entity attributes and relationships. This approach provides important information for entity representation, reduces the gap between the textual description and other modal features, and improves the consistency of the representation of the entity in different modalities. p is the Minkowski distance of the order ($p=1$ corresponds to the Manhattan distance and $p=2$ corresponds to the Euclidean distance).

The overall loss of the model is:

$$L=L_s^{CL}+L_f^{CL}+{\sum }_{m\in M}{\alpha }_m{L}_m^{CL}+{\sum }_{m\in M}{\beta }_m{L}_m^{ML}$$

(19)

where $M=\left\{g,n,a,r,v\right\}$, ${\alpha }_m$ and ${\beta }_m$ are used to balance the loss importance.

5. Experiments

5.1. Datasets

We evaluated our method on two benchmark datasets: the multimodal cross-lingual dataset DBP15K and the multimodal dataset FB15K-DB15K. DBP15K consists of three subsets: zh-en (Chinese–English), ja-en (Japanese–English), and fr-en (French–English). FB15K in FB15K-DB15K is sourced from Freebase, while DB15K is derived from the English portion of the DBP15K dataset. To assess the effectiveness of our model, we compared it with several state-of-the-art models on these two datasets. Unlike previous datasets, we enhanced these two datasets by incorporating textual description information to improve the performance of entity alignment tasks. The text description data in FB15K-DB15K is constructed based on public knowledge graphs (Wikidata and DBpedia). For instance, Wikidata provides comprehensive entity information that is used to obtain textual descriptions for entities in FB15K. We retrieve entity descriptions by making API requests based on entity IDs in FB15K. However, due to the absence of original textual description data, not all entities can be fully described. As a result, during the entity alignment process, some entities may lack their corresponding textual descriptions. Since we use two different datasets to verify the experimental results, namely DBP15K and FB15K-DB15K, the data structures in these two datasets are different, so we use different proportions of entity pairs as alignment seeds. Specifically, DBP15K is a dataset for cross-language knowledge graph alignment, and its entity alignment task mainly relies on the alignment of cross-language entities. Therefore, we only use $30\%$ of the entity pairs as seed data to ensure effective training under limited alignment information while avoiding excessive dependence of model training on aligned entity pairs due to too many seed pairs. For FB15K-DB15K, it is based on a multimodal monolingual knowledge graph, which contains entity pairs from different fields, so we use $20\%$, $50\%$, and $80\%$ of the entity pairs as seed data to measure the performance of the model under different amounts of alignment information.

5.2. Evaluation Metrics

In this paper, the evaluation metric $Hits@1$ represents a measure of the proportion of target entities appearing in the first position in the retrieval results, i.e., the accuracy rate, and $Hits@10$ represents a measure of the proportion of target entities appearing in the top 10 positions in the retrieval results, which represent the model’s ability to correctly identify the target entities. $MRR$ is the Mean Reverse Rank, calculated as the inverse rank of the first correct result for each retrieval query mean value. It reflects the speed at which the model returns the correct answer. The higher the $MRR$ value, the more the model tends to return the correct answer in the first few positions. The $MRR$ is calculated using the following formula:

$$MRR={\displaystyle \frac{1}{Q}}\sum _{i=1}^Q{\displaystyle \frac{q}{ran{k}_i}}$$

(20)

where Q represents the total number of queries, and $ran{k}_i$ represents the ranking position of the first relevant result of query i. We set random seeds to ensure the reproducibility of the experiments. Our model consists of three hidden layers, each containing 300 units. We employ a two-layer Graph Attention Network (GAT) with two attention heads per layer to compute the CL and ML losses across different modalities. We adopt a confidence-based modality fusion strategy to automatically assign varying weight coefficients to the different modalities. To prevent overfitting, we incorporate an early stopping mechanism that halts the training process if the experimental results do not improve significantly over ten consecutive rounds. These hyperparameter settings are carefully selected to build a well-optimized model and ensure its generalization performance through regularization techniques.

5.3. Competing Methods

To demonstrate the advanced nature of the model, we conducted comparative experiments on our FMEA-TD model with other models on two different datasets. The experimental results show that our model has the best performance.

Table 1. Comparison of performance with previous models on the multilingual dataset DBP15K.

Model	zh-en			ja-en			fr-en
	H@1	H@10	MRR	H@1	H@10	MRR	H@1	H@10	MRR
MTransE	0.308	0.614	0.364	0.279	0.575	0.349	0.244	0.996	0.335
BootEA	0.629	0.848	0.703	0.622	0.854	0.701	0.653	0.874	0.731
KEGC	0.478	0.835	0.598	0.490	0.844	0.610	0.486	0.851	0.610
HMAN	0.871	0.987	-	0.935	0.994	-	0.973	0.998	-
EVA	0.761	0.907	0.814	0.762	0.913	0.817	0.793	0.942	0.847
HGCN	0.720	0.857	0.768	0.766	0.897	0.813	0.892	0.961	0.917
BERT-INT	0.968	0.990	0.977	0.964	0.991	0.975	0.992	0.998	0.995
MCLEA	0.972	0.996	0.981	0.986	0.999	0.991	0.997	1.000	0.998
MEAFE	0.973	0.997	0.982	0.987	0.999	0.992	0.997	1.000	0.998
FMEA-TD (ours)	0.979	0.998	0.986	0.991	1.000	0.995	0.997	1.000	0.998

compares the performance of our model with previous models on multilingual datasets. The improvement in performance achieved by our model with respect to the best baseline MEAFE validates the significance of multilingual text description information and cross-modal similarity loss in improving multimodal entity alignment.

The compared models include the use of a graph structure through TransE variants (MTransE [28], BootEA [29]), the use of a graph structure using GCN (KECG [30], HMAN [31], HGCN [32]), GCN combined with multimodal features (EVA [11]), GCN combined with auxiliary information (BERT-INT [19]), and GAT combined with multimodal features (MCLEA [12], MEAFE [33]). Based on the comparison, our model has the most advanced baseline. Based on the experimental results, it can be concluded that our model uses richer textual descriptions; compared to MCLEA, the textual description information contains features such as entity names, attributes, etc., and it provides more detailed descriptions of these basic features, which can help the model to better understand and represent the semantic information of entities. Compared with other uni-modal alignment models, such as HMAN, the visual information can respond to the visual features of the entity, and the fusion of the information for the two modes can describe the entity more accurately and overcome the limitations of single-modal information. Compared to the BERT-INT model, our model uses only are only 1/8 of the parameters, and our model is able to better learn the semantic representation of text by fusing multilingual text information with cross-modal similarity loss without relying on specific entity description data. In contrast, the BERT-INT model relies heavily on fine-tuning the BERT model using characteristic entity description information to enhance the entity representations.

We progressively incorporated $20\%$, $40\%$, $60\%$, and $80\%$ of textual description information to supplement the entity alignment process. As shown in the entity alignment accuracy bar chart (see Figure 5), the results clearly indicate that increasing the amount of textual description information significantly enhances the alignment accuracy.

Table 2. Comparison of performance with previous models on the cross-knowledge graph multimodal dataset FB15K-DB15K.

Model		20%			50%			80%
		H@1	H@10	MRR	H@1	H@10	MRR	H@1	H@10	MRR
	20%	0.326	0.573	0.402	0.594	0.704	0.665	0.763	0.893	0.801
MCLEA	40%	0.458	0.649	0.502	0.647	0.766	0.701	0.818	0.912	0.842
	60%	0.595	0.735	0.634	0.738	0.84	0.783	0.863	0.937	0.902
	80%	0.728	0.874	0.787	0.864	0.913	0.887	0.918	0.956	0.93
	20%	0.297	0.552	0.388	0.436	0.654	0.585	0.625	0.864	0.766
MMEA	40%	0.412	0.636	0.498	0.59	0.754	0.634	0.766	0.886	0.803
	60%	0.547	0.628	0.586	0.678	0.831	0.747	0.843	0.898	0.855
	80%	0.687	0.814	0.756	0.802	0.869	0.825	0.882	0.923	0.902
	20%	0.359	0.627	0.451	0.641	0.801	0.713	0.823	0.933	0.865
FMEA-TD	40%	0.507	0.727	0.582	0.722	0.87	0.769	0.875	0.954	0.899
	60%	0.648	0.822	0.708	0.812	0.918	0.851	0.905	0.966	0.927
	80%	0.773	0.903	0.82	0.906	0.966	0.927	0.948	0.984	0.963

presents the performance of three models—MCLEA, MMEA, and FMEA-TD (ours)—in the entity alignment task, with evaluation metrics including Hit@1, Hit@10, and MRR for four different percentages of textual description data ($20\%$, $40\%$, $60\%$, $80\%$). Taking the scenario with a $20\%$ training seed ratio as an example, the best baseline model MCLEA achieves a Hit@1 value of $32.6\%$ with $20\%$ textual description data, while our model achieves a Hit@1 value of $35.9\%$ under the same conditions, representing a $10.12\%$ performance improvement. When the proportion of textual description data increases to $80\%$, the Hit@1 value of our model rises to $77.3\%$. Such substantial growth highlights the critical role of textual description data in enhancing the accuracy of entity alignment.

Compared with the performance on the DBP15K dataset, the accuracy improvement of entity alignment by adding textual description information is more significant on multimodal monolingual datasets such as FB15K-DB15K. The remarkable enhancement in model accuracy indicates that textual descriptions play a more prominent positive role in entity alignment tasks under multimodal monolingual environments.

The reasons for the differences in results between the two different experimental datasets are as follows. The differences in performance between the two datasets stem from their inherent characteristics. In DBP15K (cross-lingual), knowledge graphs have similar structures and clearly aligned entity names, which already provide strong cues for alignment. Thus, adding textual descriptions offers limited additional benefit, as the structural and name information is usually sufficient for accurate matching. In contrast, FB15K-DB15K (cross-knowledge source) contains graphs with less structural similarity and more ambiguous or varied entity names. Here, basic features alone are often insufficient, making textual descriptions crucial for capturing rich semantic details and improving alignment. This explains why removing descriptions leads to a larger drop in performance on this dataset, highlighting the need to adapt fusion strategies based on dataset characteristics.

The above experiments clearly demonstrate that the positive impact of textual description information on the entity alignment task is significant and cannot be overlooked. Particularly in scenarios where data is scarce and alignment is challenging, textual descriptions provide a valuable supplement and support for improving entity alignment.

5.4. Effects of Parameters

We investigated the number of layers K of the graph attention network (GAT), different graph structure embedding models (GCN, GAT, and GAT2V [34])), and the effects of different temperature parameters on the experimental results during the multimodal knowledge fusion process (see Figure 6). These parameters had a significant impact on FMEA-TD. We used the AdamW optimizer with a learning rate of $5\times {10}^{-4}$ and a batch size of 512 to update the model parameters. To prevent overfitting, we applied an early stopping strategy, where training was terminated if the validation performance (Hits@1) did not improve for 10 consecutive evaluations.

When choosing the number of layers of the GAT network, we considered both the accuracy of the model and the resource consumption. With similar experimental results, we chose $K=2$. During the selection process of graph neural networks, we found that GAT significantly outperformed GCN and GAT2V in the multimodal entity alignment task. GAT adaptively assigned weights to different neighbors through the attentional mechanism and was able to dynamically adjust the information propagation according to the importance of the neighboring nodes, thus capturing the alignment information more efficiently. In contrast, GCN uses fixed neighbor weights, which may not fully utilize the graph structure information. In addition, GAT is more capable of handling heterogeneous graphs (different types of nodes and edges), while GAT2V is not as efficient as GAT when faced with diversity and complexity. The temperature parameter is typically used in multimodal collaboration to regulate the smoothness of model outputs. When the temperature value is low, it sharpens the output distribution and increases the probability of high-confidence categories. Conversely, higher temperature values flatten the output distribution, thereby reducing the probability of high-confidence categories. In the end, we chose a temperature coefficient of 0.5.

5.5. Ablation Experiments

We conducted an exhaustive analysis of the ablation experiments for the contributions of the different components for graph structural features, visual features, underlying textual information, textual description information, cross-modal similarity loss, and multimodal confidence collaborator, as shown in

Table 3. Experimental results of ablation on multilingual dataset in different directions.

	zh-en		en-zh		ja-en		en-ja		fr-en		en-fr
	H@1	MRR	H@1	MRR	H@1	MRR	H@1	MRR	H@1	MRR	H@1	MRR
w/o Gph	0.920	0.943	0.916	0.941	0.960	0.974	0.958	0.971	0.991	0.999	0.992	0.995
w/o Img	0.970	0.983	0.969	0.993	0.981	0.991	0.983	0.991	0.994	0.999	0.994	0.998
w/o Text	0.864	0.891	0.861	0.888	0.903	0.914	0.907	0.918	0.962	0.978	0.958	0.973
w/o Desc	0.971	0.980	0.970	0.979	0.981	0.990	0.980	0.988	0.994	0.993	0.991	0.995
w/o ML	0.976	0.987	0.972	0.983	0.988	0.993	0.988	0.993	0.997	0.999	0.997	0.999
w/o Fus	0.973	0.982	0.971	0.981	0.983	0.991	0.984	0.991	0.994	0.996	0.993	0.997
ours	0.979	0.986	0.976	0.985	0.991	0.995	0.989	0.994	0.997	0.999	0.997	0.999

.

After removing the visual information, text description information, and cross-modal similarity loss, the model performance declined. However, this change had a limited impact on the experimental results. When the base text information is removed, the model performance drops significantly. This indicates that the base text makes a greater contribution to the model. Different components all play a role in the model’s performance, but the role of text information is particularly prominent. It provides the model with the most fundamental and crucial semantic representation. In future experiments, we will consider making more comprehensive use of description information and adopt knowledge graph completion methods to enhance the model’s ability to align multimodal knowledge graphs.

6. Conclusions

Most recent mainstream alignment studies typically adopt deep learning-based methods to embed knowledge for different modalities, but these methods often fail to fully consider richer descriptive information. Therefore, this paper proposes an entity alignment method in the cross-lingual multimodal knowledge graph FMEA-TD based on textual descriptions. The main contributions are as follows: (1) Textual descriptions are used to provide rich complementary features for multimodal data, thereby capturing more comprehensive semantic information. (2) A multimodal confidence collaborator is introduced to assign dynamic weights for multimodal feature fusion through inter-modal collaboration, making better use of complementary information. (3) A joint optimization loss function is established to simultaneously consider the associations between each modality embedding, text description embedding, and fusion embedding, enabling the model to more accurately understand and represent the overall semantic features of entities. The strong performance of FMEA-TD demonstrates its potential in improving multilingual knowledge retrieval by accurately aligning semantically equivalent entities across different languages and modalities. In applications such as multilingual search and question answering, correct alignment ensures that user queries are matched with the most relevant and complete information, regardless of the language or modality of the underlying data. Furthermore, by leveraging textual descriptions and a confidence-aware fusion strategy, FMEA-TD is able to handle incomplete or noisy data more effectively, making it a robust solution for real-world dynamic and heterogeneous knowledge environments. In future work, we plan to explore methods for supplementing unstructured data such as images and textual descriptions in knowledge graphs, further improving the content of knowledge graphs and enhancing the coverage and accuracy of the graphs. This is expected to further improve the alignment performance, especially in handling multilingual and multimodal alignment tasks to achieve better results.