GC4MRec: Generative-Contrastive for Multimodal Recommendation

Abstract

The rapid growth of information technology has led to an explosion of data, posing a significant challenge for data processing. Recommendation systems aim to address this by providing personalized content recommendations to users from vast datasets. Recently, multimodal recommendation systems have gained considerable attention due to their ability to leverage diverse data modalities (e.g., images and text) for more accurate recommendations. However, effectively fusing these modalities to accurately represent user preferences remains a challenging task, despite progress made by existing multimodal recommendation approaches. To address this challenge, we propose a novel method which we call GC4MRec (Generative-Contrastive for Multimodal Recommendation). On the one hand, we design a bilateral information flow module using two graph convolutional networks (GCNs). This module captures modal features from two distinct perspectives—standard and generatively augmented—to extract latent preferences. On the other hand, we introduce a novel modality fusion module that dynamically represents user multimodal fusion preferences, enabling the construction of accurate user preference profiles. Finally, we evaluate our proposed method, GC4MRec, on three public real-world datasets and demonstrate its effectiveness compared to the state-of-the-art methods.

1. Introduction

Recommender systems are widely applied in various domains of the internet, such as e-commerce, social media, and entertainment, aiming to personalize recommendations by modeling collected data and establishing latent mappings between users and items. Content-based recommendation models extract features representing item characteristics and construct user profiles reflecting past preferences [1,2]. Collaborative filtering (CF) identifies users with similar preferences based on their ratings or behaviors and then recommends items liked by similar users but not yet experienced by the target user [3,4]. CF techniques may utilize a variety of characteristics as the basis for establishing similarity [5,6]. The traditional recommendation algorithms build relatively simple and shallow mappings between users and recommendation targets. These methods struggle to discern user preferences in scenarios where user interaction data are sparse and are susceptible to data biases.

In recent years, deep learning-based recommendation algorithms have made significant progress,. Graph convolutional networks (GCNs) are increasingly employed in recommender systems to model user–item relationships [7,8]. GCNs operate on a user–item interaction graph, propagating information through graph convolutional layers [9]. They capture higher-order connectivity between nodes by aggregating and propagating information across nodes with interactions, enabling the recommendation of items of interest to users [10,11]. However, conventional deep learning-based recommendation systems fail to leverage the modality diversity of user interaction data. In real-world scenarios, user preferences can be reflected by data from different modalities, which exhibit user data complementarity. The proper fusion of these modalities can construct a comprehensive user profile, enabling the more accurate capture of user needs compared to single-modal recommendation systems.

Existing works have combined multimodal data with GCNs, constructing independent channels to convolute and propagate user preference information for different modalities [12,13]. DGVAE [14] enhances the interpretability of multimodal recommendations by unifying the mapping the data of image and text modalities into text, performing graph encoding and reconstruction separately. Deconstructing graph nodes to incorporate auxiliary information can enhance the performance of recommender systems [15,16]. GCNs are particularly well suited for processing multimodal data due to their inherent ability to model complex relationships [17,18]. For instance, MMGCN [12] offers a novel approach to personalized micro-video recommendation. Its strength lies in explicitly modeling user preferences across modalities, leveraging graph convolutional networks for representation refinement through information exchange on modality-aware user–item graphs. FREEDOM [19] addresses multimodal recommendation by freezing the item–item graph and denoising the user–item graph via degree-sensitive pruning. This approach achieves state-of-the-art performance with significantly reduced computational cost. LATTICE [13] addresses multimedia recommendation by learning latent item–item structures from multimodal features using a novel modality-aware structure learning layer and graph convolutions. However, some methods employ early-stage modality fusion techniques, such as simple weighted fusion or concatenation, lacking the ability to deeply explore the underlying preferences of user modalities. When the number of convolutional layers in GCNs increases, over-smoothing issues arise. Some methods lack optimization for this drawback, while others utilize different approaches to mitigate the problem; for instance, LightGCN [8] mitigates over-smoothing by linearly reducing the contribution factor of higher-order graph convolution propagation.

Recently, contrastive learning (a form of self-supervised learning) has been widely applied in recommender systems [20,21]. It enhances model representation capabilities by maximizing the similarity between positive samples and minimizing the similarity between negative samples [22,23]. However, the sparsity of samples and the design of pretext tasks significantly impact the training effectiveness of contrastive learning.

To address the aforementioned challenges, we propose a novel multimodal recommendation learning framework. Regarding data sparsity and pretext task setting in contrastive learning, we first project the graph features using a linear layer, followed by graph reconstruction using a Variational Graph Autoencoder (VGAE) [24], resulting in original graph G and augmented graph G’. The original graph and the augmented graph are treated as a pair of positive samples, while others are treated as negative samples. The InfoNCE loss function is employed to optimize the representation learning ability of the linear layer. The introduction of VGAE enhances the model’s ability to explore latent user features and narrows the distance in feature space between nodes with similar preferences. To mitigate over-smoothing, we employ cross-residual connections between the input and output layers of the two GCNs. This counteracts the tendency of individual GCNs to produce homogenized representations. Due to the generative capability of VGAE, it can mitigate the over-smoothing effect of GCN.

Regarding multimodal fusion, we hypothesize that user preferences exhibit a long-tail effect, meaning that users pay significant attention to a small portion of item features. Therefore, we propose a metadata slicing generation method: we segment the features of users and items into chunks of varying lengths, where each chunk is mapped to a specific attribute entity (color, size, etc.), referred to as “metadata”. The length of the metadata reflects the intensity of preference for a particular attribute. We hypothesize that users are typically interested in only a subset of attributes; therefore, we determine the length of metadata by sampling from a Gaussian distribution. This ensures that a small portion of metadata captures the majority of the feature length, thereby effectively identifying the users’ primary points of interest. The channel attention mechanism [25], originally used for convolutional graph attention calculation in image processing, is introduced into GC4MRec for fusing different modal data. In this paper, each metadata point is treated as a channel. First, we pool the metadata, cross-add the modalities, and perform channel-wise convolution to obtain channel weights. The channel weights are multiplied by the original metadata channel to obtain the final metadata value. Finally, the different modal data are summed and averaged to obtain the final modality fusion result. In summary, our contributions are as follows:

• A novel multimodal learning paradigm employing enhanced self-supervised learning is introduced to uncover latent user preferences and enable effective multimodal fusion.
• Latent user preferences are captured more effectively via a dual spatio-temporal cross-residual module, which integrates information from both the original and enhanced graph at various temporal stages.
• A metadata-aware fusion method provides a flexible mechanism for fusing multimodal data.
• Experimental validation is performed on several public datasets.

Following this introductory section, Section 2 reviews the relevant literature on multimodal recommendation, contrastive learning, and generative models, identifying limitations in existing approaches. Section 3 details the GC4MRec model, including graph information enhancement, subgraph generation, reconstruction, contrastive learning, and multimodal fusion. Section 4 presents a comprehensive experimental evaluation on three datasets, comparing GC4MRec against baselines and assessing component contributions. Finally, Section 5 concludes by summarizing findings, discussing implications, and outlining future research directions.

2. Related Work

2.1. Multimodal Recommendation

The recent surge in computational power and the explosive growth of data have led to a growing number of studies exploring the integration of multimodal techniques into recommender systems. Multimodal recommender systems leverage data from various modalities, such as texts and images, to provide a more comprehensive and complementary representation of rich semantic information. This approach facilitates a more accurate and nuanced understanding of user preferences, ultimately leading to improved recommendation performance.

VBPR [26] pioneered the incorporation of image modality semantic information with ID-based user interaction data to generate user preference rankings for recommended items. Based on this foundation, researchers have conducted further exploratory work with the aim of capturing fine-grained user–item interaction preferences. To this end, researchers have incorporated attention mechanisms into multimodal recommendation systems [27,28,29], effectively fusing information across different modalities, leveraging their complementary aspects, and enhancing the accuracy of content recommendations. High-order connectivity information is crucial for the performance of recommendation systems.To this end, numerous studies have leveraged graph neural networks to propagate latent interaction information among users or items, enabling more comprehensive and accurate preference recommendations for users [12,13,29].

The key distinction between multimodal recommendation and unimodal recommendation lies in effectively handling data from diverse modalities and seamlessly fusing information represented across these modalities. MMSSL [30] leverages adversarial self-supervised learning to enhance modality-aware graphs for extracting modal data and employs cross-modal attention mechanisms for modality fusion. MMGCN [12] establishes interaction graphs for different modalities, extracting unimodal information individually before fusing them through a stacking operation.

In contrast to conventional approaches, our fusion method acknowledges the specific characteristics of recommendation tasks. We posit that user attention exhibits a long-tail distribution, implying that users prioritize specific features. Therefore, our method focuses on capturing and assigning higher weights to these user-emphasized features.

2.2. Contrastive Learning

Contrastive learning, a self-supervised learning approach, has gained significant traction in recent years. It excels at learning effective embedding representations by maximizing the similarity between positive sample pairs and minimizing the similarity between negative pairs within an embedding space. This method has proven particularly valuable in scenarios with limited labeled data. CLIP [31] constructs a multimodal embedding space and leverages contrastive learning to explore the semantic relationships between unlabeled text–image pairs. This approach results in a pre-trained model with a strong understanding of both textual and visual semantics, inspiring us to explore the integration of contrastive learning within multimodal recommender systems.

SGL [32] employs node dropout, edge dropout, and random walk operations to generate contrastive views, maximizing the agreement between representations of the same node across different views. This self-discrimination approach effectively learns the consistency of individual nodes under diverse perspectives, ultimately leading to enhanced node representation capabilities. CL4SRec [33] introduces a contrastive learning approach for sequence recommendation that leverages self-supervision signals captured through data augmentation. Specifically, it employs three augmentation strategies: item cropping, item masking, and item reordering, to generate diverse views of the input data. DHCN [34] proposes a contrastive learning framework based on hypergraph session views for sequence recommendation. This method constructs hypergraphs to represent user sessions and applies contrastive loss to encourage similar sessions to have closer representations in the embedding space. EGLN [35] focuses on graph-based recommendation and performs graph augmentation by maximizing the mutual information between local and global features of the graph. This approach aims to enhance the representation learning by capturing both fine-grained and high-level structural information. NCL [36] enhances the agreement between nodes with connectivity within the neighborhood while reducing the agreement between nodes from different neighborhoods. This method emphasizes the local structure of the graph and improves the node representation learning.

2.3. Generative Models in Recommendation

Generative models have emerged as a transformative force across various domains in recent years. By learning the underlying distribution of data, these models can generate novel data samples. This remarkable generative capability has injected fresh vitality into recommendation systems. DreamRec [37] generates an ideal item by learning user interaction sequences and recommends items within the sample space that exhibit the highest similarity to the ideal item. ACVAE [38] enhances the accuracy and robustness of Variational Autoencoders (VAEs) by incorporating an adversarial component. CLLM4Rec [39] adapts large language models (LLMs) to collaborative recommendation tasks through prompting techniques. BERT4Rec [40] leverages the powerful sequence modeling capabilities of transformers to model user interaction sequences and predict subsequent user behavior. Meta-SGCL [41] optimizes the sequence representation ability of recommendation models by combining generative and contrastive learning approaches.

3. Methodology

In this section, we detail the proposed GC4MRec. We begin by introducing the fundamental notations and definitions used throughout this paper. Figure 1 shows the overall architecture of GC4MRec. We model different modalities of data separately, resulting in modality-aware graphs. During GCN propagation, we design two distinct GCNs to capture high-order connectivity. Through cross-residual connections, we integrate cross-temporal and cross-modal information, mitigating over-smoothing. The GCN outputs are then fed into a Variational Graph Autoencoder (VGAE), where subgraphs are sampled to obtain augmented graph data. These augmented data are utilized in a contrastive learning framework to train the model and capture latent user preferences. In the multimodal fusion stage, we introduce the concept of metadata to model user preferences at different granularities. Channel attention is employed to compute attention weights for metadata from different modalities thereby dynamically fusing multimodal information based on specific user preferences.

3.1. Preliminaries

Let us define $\mathcal{U}=\{u_1,u_2,\dots ,u_{\left|\mathcal{U}\right|}\}$ representing the user set, and $\mathcal{I}=\{i_1,i_2,\dots ,i_{\left|\mathcal{I}\right|}\}$ representing the item set. $A\in {\mathbb{R}}^{\left|\mathcal{U}\right|\times \left|\mathcal{I}\right|}$ denotes the user–item adjacency matrix, capturing the interactions between users and items. $\mathcal{G}=(\mathcal{V},\mathcal{E})$ denotes the user–item interaction graph, where $\mathcal{V}=\mathcal{U}\cup \mathcal{I}$ represents the set of all nodes (users and items) and $\mathcal{E}\subseteq \mathcal{V}\times \mathcal{V}$ represents the set of edges. $\mathcal{M}=\{t,i\}$ denotes the set of modalities, where t represents the text modality and i represents the image modality. We denote with ${\mathbf{f}}_i^t\in {\mathbb{R}}^{d_t}$ the textual feature embedding for item $i\in \mathcal{I}$, where $d_t$ represents the dimensionality of the textual feature space. ${\mathbf{e}}_i\in {\mathbb{R}}^d$ denotes the overall embedding representation of item $i\in \mathcal{I}$, where d is the embedding dimension. ${\mathbf{e}}_u\in {\mathbb{R}}^d$ denotes the overall embedding representation of user $u\in \mathcal{U}$, where d is the embedding dimension.

3.2. Graph Information Enhancement

To obtain contrastive positive samples with both rich semantic representations and smooth contextual information, we employ a graph information enhancement approach.

3.2.1. Modality-Aware Subgraph Generation

Specifically, we utilize a Variational Graph Autoencoder (VGAE) to re-encode subgraphs, mapping them into latent representations. For each modality $m\in \mathcal{M}$, we construct a modality-specific subgraph ${\mathcal{G}}_c^m=({\mathcal{V}}_c,{\mathcal{E}}_c,{\mathbf{X}}_c^m)$ centered at node c. Here, $\mathbf{X}{c}^m\in {\mathbb{R}}^{N\times d_m}$ represents the feature matrix of the subgraph nodes for modality m, where the i-th row corresponds to the feature vector ${\mathbf{f}}_{v_i}^m$ of node $v_i\in {\mathcal{V}}_c$. The VGAE model learns to encode the structural information and node features of ${\mathcal{G}}_c^m$ into a lower-dimensional latent representation ${\mathbf{z}}_c^m\in {\mathbb{R}}^{d^{\prime }}$, where $d^{\prime }$ is the dimension of the latent space. This encoding process aims to capture the essential characteristics of the subgraph from the perspective of modality m.

3.2.2. Graph Reconstruction with Resample

To obtain enhanced samples, we assume that each sample $\mathbf{x}$ has a latent representation ${\mathbf{x}}^{\prime }$. This latent distribution is derived as follows:

$$q\left({\widehat{\mathbf{X}}}^{\mathbf{m}}\right|{\mathbf{X}}^{\mathbf{m}},\mathbf{A})=\prod _{i=1}^{N}q\left({\widehat{\mathbf{x}}}_{\mathbf{i}}^{\mathbf{m}}\right|{\widehat{\mathbf{X}}}^{\mathbf{m}},\mathbf{A}).$$

(1)

where ${\widehat{\mathbf{X}}}_m$ and ${\mathbf{X}}_m$ represent the latent features and raw features of modality m, respectively, and $\mathbf{A}$ is the adjacency matrix. ${\widehat{\mathbf{x}}}_i^m$ represents the latent representation distribution of a user or item. We define this latent distribution as:

$$q\left({\widehat{\mathbf{x}}}_i^m\right|{\widehat{\mathbf{X}}}^m,\mathbf{A})=\mathcal{N}({\widehat{\mathbf{x}}}_i^m\mid {\mu }_i^m,\mathrm{diag}{\left({\sigma }_i^m\right)}^2).$$

(2)

where $\mathrm{diag}{\left({\sigma }_i^m\right)}^2$ denotes the diagonal covariance matrix of the latent distribution for generating ${\widehat{\mathbf{X}}}^{\mathbf{m}}$ where ${\sigma }_i^m$ represents standard deviations for the i-th node in modality m. This diagonal structure assumes independence among latent dimensions, balancing computational efficiency and uncertainty modeling. It enables VGAE to generate graph nodes that capture users’ latent preferences, improving recommendation performance by accurately representing user–item interactions. Here, ${\mu }_i^m$ and ${\sigma }_i^m$ are defined as follows:

$${\mu }_i^m={\mathrm{GCN}}_{\mu }({\mathbf{X}}_i^m{\mathbf{W}}^{\mu }+{\mathbf{b}}^{\mu })+ϵ$$

(3)

$${\sigma }_i^m={\mathrm{GCN}}_{\sigma }({\mathbf{X}}_i^m{\mathbf{W}}^{\sigma }+{\mathbf{b}}^{\sigma })+ϵ$$

(4)

where ${\mathbf{W}}^{\mu }$, ${\mathbf{b}}^{\mu }$, ${\mathbf{W}}^{\sigma }$, and ${\mathbf{b}}^{\sigma }$ are learnable parameters, and $ϵ$ is a noise signal sampled from a standard Gaussian distribution.

3.2.3. Optimization

During latent variable distribution reconstruction, the adjacency matrix $\mathbf{A}$ plays a crucial role as a supervisory signal. We construct the modality-aware posterior probability distribution of the adjacency matrix derived as follows:

$$p\left(\mathbf{A}\right|{\widehat{\mathbf{X}}}^m)=\prod _{i=1}^N\prod _{j=1}^{N}p\left(A_{ij}\right|{\widehat{\mathbf{x}}}_i^m,{\widehat{\mathbf{x}}}_j^m)$$

(5)

The non-probabilistic interaction edge between nodes i and j in the modality-aware adjacency matrix ${\widehat{A}}^m$ is calculated as follows:

$${\widehat{A}}_{ij}^m=\left\{\begin{array}{cc}1,\hfill & \mathrm{if}{\displaystyle \frac{{\left({\widehat{\mathbf{x}}}_i^m\right)}^{\top }{\widehat{\mathbf{x}}}_j^m}{|{\widehat{\mathbf{x}}}_i^m\left|\right|{\widehat{\mathbf{x}}}_j^m|}}>\theta ,\hfill \\ 0,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(6)

Here, $\theta$ is a learnable parameter acting as a pre-defined threshold. Leveraging the foundational architecture of our approach, the contrastive learning module seeks to utilize high-quality samples. Due to its enhanced generalization and rapid generation capabilities relative to alternatives like diffusion models, we employ VGAE. This strategy seeks to generate samples that accurately reflect underlying user preferences. Only then can contrastive learning yield precise and generalizable representations, bolstering subsequent prediction accuracy. The variational lower bound, a common optimization objective for generative models, is instrumental in achieving high performance. The underlying principle is learning a latent representation of the input graph while ensuring the generated graph structure closely resembles the original. More specifically, our training target is to optimize the parameters of this variational model. We maximize its variational lower bound, which encourages the distribution of $\widehat{X}$ to match the distribution of X and penalizes its divergence from a standard Gaussian distribution. The lower bound is defined as:

$$\begin{array}{c}\hfill {\mathcal{L}}_{vgae}=\mathrm{KL}\left[q(\widehat{\mathbf{X}}\mid \mathbf{X},\mathbf{A})\mid \mid p\left(\mathbf{Z}\right)\right]-\mathbb{E}q\left(\widehat{\mathbf{X}}\right|\mathbf{X},\mathbf{A})\left[logp\left(\mathbf{A}\mid \widehat{\mathbf{X}}\right)\right]\end{array}$$

(7)

$\mathrm{KL}$ denotes the Kullback–Leibler divergence. $p\left(\mathbf{Z}\right)={\prod }_i\mathcal{N}\left({\mathbf{z}}_{\mathbf{i}}\right|0,\mathbf{I})$ represents the Gaussian prior, with ${\mathbf{z}}_{\mathbf{i}}$ being the latent representation of node i.

3.3. Graph Contrastive Learning

To further excavate user preferences and capture intricate high-order connectivity patterns between nodes, we leverage the self-supervised nature of contrastive learning. By encouraging similar nodes to be closer in the embedding space and dissimilar nodes to be further apart, the model aims to enhance the discriminative power of node representations. This design facilitates more accurate node classification and crucially improves the recommendation performance, as the final prediction stage of the model relies on computing the similarity between users and items to determine whether an item should be recommended to a user. Therefore, the model’s ability to effectively discriminate between nodes is essential for ensuring high-quality recommendations. This approach encourages the model to learn representations where similar nodes are closer in the embedding space, while dissimilar nodes are pushed further apart.

Designing a reasonable contrastive pretext task can fully leverage the self-supervised signals in the corpus. We first sample the original graph G to obtain multiple subgraphs: $G^{sub}=\{g_1^{sub},g_2^{sub},\dots ,g_N^{sub}\}$. Then, we generate an augmented copy of each subgraph using VGAE, which is derived as follows:

$$G^{\widetilde{sub}}=VGAE(MLP\left(G^{sub}\right),A^{sub})$$

(8)

where $A^{sub}$ is the adjacency matrix corresponding to $G^{sub}$. We construct N pairs of positive samples, where each pair consists of $(g^{sub},g^{\widetilde{sub}})$. All other pairs are considered negative samples. We use infoNCE to optimize the parameters and the contrastive loss is derived as follows:

$${\mathcal{L}}_{cl}=-\sum _{m\in M}\sum _{i=1}^{N}log{\displaystyle \frac{exp({\left({\mathbf{x}}_i^m\right)}^T\left(\widetilde{{\mathbf{x}}_i^m}\right)/\tau )}{{\sum }_{k=1}^{N}exp({\left({\mathbf{x}}_i^m\right)}^T\left(\widetilde{{\mathbf{x}}_k^m}\right)/\tau )}}$$

(9)

where ${\mathbf{x}}_i$ and ${\tilde{\mathbf{x}}}_i$ represent the embedding vectors of the corresponding nodes in the original subgraph and the augmented subgraph, respectively, for the i-th positive sample pair, and $\tau$ is a temperature parameter that controls the sharpness of the contrastive loss.

3.4. Dual Spatio-Temporal Cross-Residual Module

To effectively capture users’ latent preferences, we developed a spatio-temporal cross-residual module. This module employs a residual connection between an original and an enhanced GCN to integrate graph propagation information from different temporal steps. The derivation of the architecture is presented below:

$${\mathbf{e}}^m=\sum _{i=1}^N\sum _{j=1}^N\left[{\mathrm{GCN}}_{\mathrm{normal}}\left({\mathbf{e}}_i^m\right)+{\mathrm{GCN}}_{\mathrm{enhanced}}\left({\mathbf{e}}_{N-j+1}^m\right)\right]$$

(10)

$e^m$ represents the embedding for modality m, N denotes the total number of GCN propagation layers, i is the index of the $GC{N}_{normal}$ layer, and j is the index of the $GC{N}_{enhanced}$ layer.

$$\begin{array}{cc}x\hfill & =GC{N}_{enhanced}\left(e^m\right)\hfill \\ x\hfill & =x+VGAE(G^{sub\left(x\right)},A^{sub\left(x\right)})\hfill \end{array}$$

(11)

For simplicity, we let x = $GC{N}_{enhanced}\left(e^m\right)$ and then generate a set of subgraphs and adjacency matrices for x. We use VGAE to generate enhanced subgraphs, which are then residual-connected with $GC{N}_{enhanced}\left(e^m\right)$.

3.5. Multimodal Fusion

Fusing embeddings from different modalities is a crucial step in multimodal content processing. As illustrated in Figure 2, modeling user preferences based on single-modal information is often insufficient. When users exhibit heightened sensitivity to a particular modality, incorporating multiple modalities enables a more comprehensive modeling of user preferences, significantly enhancing the accuracy of recommendations, a simple model can intuitively demonstrate how different modalities influence user decisions. Therefore, we aim to capture the complex complementary dependencies inherent in multimodal data through a more fine-grained attention mechanism. This approach enables the establishment of clearer and more explicit boundaries for preference semantics. To this end, we propose a modality-channel multi-head attention mechanism to flexibly scale the influence of different modality features on the final representation.

3.5.1. Metadata and Meta-Preference Modeling

We focus on the importance of metadata within user preferences or item attributes. As shown in Figure 3, we define $h^u$ and $h^i$ as the metadata chunk features for users and items, respectively, aiming to model atomic user preferences through these metadata blocks, such as material attributes, color characteristics, price features, and more. For instance, this might include a user’s preferred color, their favorite product category, or an item’s type and size. Naturally, each metadata element warrants varying degrees of attention. User preferences exhibit diverse focuses, while item attributes, representing their selling points, are similarly subject to different levels of consideration. Therefore, we employ a dynamic weighted summation of metadata to capture the latent representations embedded within these interactions. This process is defined as follows:

$$\begin{array}{cc}\hfill & H=\{h_1,h_2,\dots ,h_{\left|H\right|}\},\hfill \\ \hfill & h_i={\displaystyle \frac{s_i}{{\sum }_{j=1}^{\left|H\right|}{s}_j}}·d,\hfill \\ \hfill & s_i\sim \mathcal{N}(0,1),i=1,2,\dots ,\left|H\right|.\hfill \end{array}$$

(12)

$s_i$ represents a randomly sampled value from a normal Gaussian distribution with a mean of 0 and a variance of 1. $\left|H\right|$ signifies the number of metadata, which corresponds to the number of samples. $h_i$ represents the length of each metadata point. To isolate these distinct attribute features within each sample and map them to separate vector spaces for processing, we employ multi-head attention. For each metadata index $i\in \left|H\right|$, the corresponding feature $e_i^m$ with length $h_i$ is linearly projected into $\left|H\right|$ different subspaces, resulting in $Q_i^{\left|H\right|},K_i^{\left|H\right|},V_i^{\left|H\right|}$ as follows:

$$\begin{array}{cc}\hfill & Q_i^{\left|H\right|}=W_Q^{\left|H\right|}{e}_i^m,\hfill \\ \hfill & K_i^{\left|H\right|}=W_K^{\left|H\right|}{e}_i^m,\hfill \\ \hfill & V_i^{\left|H\right|}=W_V^{\left|H\right|}{e}_i^m.\hfill \end{array}$$

(13)

where $W_Q^{\left|H\right|},W_K^{\left|H\right|},W_V^{\left|H\right|}$ are learnable weight matrices, and $\left|H\right|$ is the number of attention heads. We calculate the attention weights between each metadata point: for each head h, we calculate the attention weight between each metadata point i and other metadata points j:

$$Attention{(Q_i,K_j,V_j)}_h=softmax(Q_i^h{K_j^h}^T/\sqrt{d_k})V_j^h,$$

(14)

where $d_k$ is the dimension of $K_j^h$, and the softmax function is used to normalize the attention weights. We perform a weighted summation of the output for each metadata point: for each head h, we perform a weighted summation of the output $Attention{(Q_i,K_j,V_j)}_h$ for all metadata i, resulting in the final output for each metadata point $Attention{(Q_i,K,V)}_h$:

$$Attention(Q_i,K,V)h=\sum _{j=1}^{\left|H\right|}Attention{(Q_i,K_j,V_j)}_h.$$

(15)

We integrate the outputs of all metadata, concatenate the outputs of all heads h, and perform a linear projection using a learnable weight matrix $W^O$, resulting in the final output:

$$e_i^m=W^O[Attention{(Q_i,K,V)}_1,\dots ,Attention{(Q_i,K,V)}_h].$$

(16)

$$e^m=\left[\begin{array}{c}{e}_1^m,e_2^m,\dots ,e_{\left|H\right|}^m\end{array}\right]$$

(17)

$e_i^m$ represents the output of the i-th metadata point calculated across $\left|H\right|$ attention heads, while $e^m$ denotes the final output after processing all metadata.

3.5.2. Channel Cross-Fusion Convolution (CCFC)

Channel attention mechanisms have achieved remarkable success in convolutional operations for image processing [25,42]. Inspired by this, we introduce a novel multimodal dynamic fusion approach. We treat different modalities as distinct channels and initially cross-integrate the data from these channels, treating the result as a single modality. Subsequently, we employ a 1D deformable convolutional kernel to perform a convolution aggregation operation on the k nearest metadata entries to the target metadata. The aggregated result then serves as the new representation of the target metadata, defined as:

$$\begin{array}{c}{p}_i^m=mean\left(e_i^m\right),\\ P=p_i^{text}\oplus p_i^{image},\\ W_{fusion}=\sigma (W_c\ast P),\\ e=W_{fusion}\ast e.\end{array}$$

(18)

We begin by applying average pooling to the embedding $e_i^m$ of the specified modality and metadata. The pooled results of the metadata across different modalities are then concatenated element-wise. Subsequently, we map the concatenated vector P to attention scores using a learnable matrix $W_c$. Finally, we multiply these attention scores with the original features to update the features with the integrated weights. More specifically, the features of both text and image samples are divided into multiple metadata units, with the set of H = {$h_1$, $h_2$, …, $h_{\left|H\right|}$}. The size of each metadata unit is determined by sampling from a Gaussian distribution, as described in Section 3.5.1. Subsequently, pooling operations are applied separately to the metadata of the image and text modalities. The pooled text metadata and image metadata are then concatenated in a staggered manner, as illustrated in the modality fusion module of Figure 1. Following this, channel attention is employed to compute the attention scores for each metadata unit. These scores are used as weights to scale the original metadata, thereby adjusting the corresponding modality features. The scaled features of both modalities are then restored to their original forms and combined through a weighted averaging process to produce the final fused result.

3.6. Prediction and Optimization

The final GCN outputs user and item representations, denoted as $h_u$ and $h_i$, respectively. The user–item interaction score, $z_{u,i}$, can be computed using the following formula:

$$y_{u,i}=\sigma (h_u^T\ast h_i)$$

(19)

To optimize the ranking quality, we utilize the Bayesian Personalized Ranking (BPR) loss function. This approach involves constructing a triplet $(u,i,j)$, where u represents the user, i is an item that the user has interacted with (positive example), and j is an item that the user has not interacted with (negative example). Based on these triplets, we calculate the corresponding positive example score, $y_{u,i}$, and negative example score, $y_{u,j}$. Our objective is to minimize the following BPR loss function:

$${\mathcal{L}}_{bpr}=-\sum _{(u,i,j)\in D}log(y_{u,i}-y_{u,j})+{\lambda \left|\right|\theta \left|\right|}^2$$

(20)

To prevent model overfitting caused by excessive model parameters, we introduce a regularization term. Specifically, we employ L2 regularization, which is defined as ${\lambda \left|\right|\theta \left|\right|}^2$, where $\lambda$ is the regularization strength, a positive hyperparameter that controls the amount of regularization applied. Figure 4 illustrates the sensitivity of the regularization hyperparameter $\lambda$. $\theta$ represents the set of model parameters. The overall loss function of our model is defined as follows:

$${\mathcal{L}}_{rec}={\mathcal{L}}_{vgae}+{\mathcal{L}}_{cl}+{\mathcal{L}}_{bpr}$$

(21)

4. Experiment

In this section, we conduct comprehensive experiments to evaluate the performance of our proposed GC4MRec on three widely used real-world datasets.

4.1. Experimental Settings

4.1.1. Datasets

To evaluate our proposed GC4MRec in the top-N item recommendation task, we conduct extensive experiments on three categories of the widely used Amazon dataset: Sports, Baby, and Clothing. These datasets offer items from distinct categories and vary in the number of users, items, and interactions. The statistics of these datasets are presented in

Table 1. Statistics of the datasets.

Dataset 1	# Users	# Items	# Interactions	Density
Clothing	39,387	23,033	237,488	0.00026
Sports	35,598	18,357	256,308	0.00039
Baby	19,445	7050	139,110	0.00101

¹ Datasets can be accessed at https://cseweb.ucsd.edu/~jmcauley/datasets/amazon/links.html (accessed on 18 January 2025).

. Each dataset comprises both textual and visual modalities. Textual features are extracted as 384-dimensional embeddings using pre-trained sentence transformers. For the visual modality, we utilize publicly available 4,096-dimensional visual features.

4.1.2. Baselines

To demonstrate the effectiveness of our proposed GC4MRec, we compare it with the following state-of-the-art recommendation methods, spanning both general collaborative filtering and multimedia recommendation approaches.

General Collaborative Filtering:

* MF-BPR [43]: This method employs matrix factorization (MF) as its core model and optimizes the user and item latent representations using Bayesian Personalized Ranking (BPR) loss.

* LightGCN [8]: This method streamlines the conventional graph convolutional network (GCN) by eliminating non-linear activation and feature transformation layers, focusing on efficient neighborhood aggregation for recommendation.

Multimedia Recommendation:

* VBPR [26]: This method incorporates visual features as side information to enhance MF-BPR, improving recommendation performance by leveraging item visual characteristics. For a fair comparison, we extend VBPR to include textual modality in our experiments.

* MMGCN [12]: This method constructs modality-specific graphs and employs a GCN for each modality. The final prediction is obtained by fusing the representations learned from each modality’s GCN.

* LATTICE [13]: This method leverages multimodal features to construct an item semantic graph, capturing latent semantic correlations between items and improving recommendation quality. As our proposed method builds upon and improves LATTICE, we highlight these improvements in the Results section.

* DGVAE [14]: This method maps multimodal data into textual representations and employs VGAE for modeling and reconstructing the item–item graph, thereby enhancing the accuracy of the model’s recommendations.

* FREEDOM [19]: This method refines the user–item graph used in LATTICE by removing potential false-positive edges and freezes the item–item graph to boost recommendation performance.

* SLMRec [44]: This method integrates self-supervised learning into multimedia recommendation, utilizing data augmentations to uncover multimodal patterns and improve contrastive learning.

* MMSSL [30]: This method introduces a modality-aware interactive structure learning paradigm using adversarial perturbations and employs cross-modal contrastive learning to disentangle shared and modality-specific features.

4.1.3. Evaluation Protocols

To evaluate the performance fairly, we adopt Recall@k and NDCG@k as our evaluation metrics, where Recall@k measures the proportion of relevant items (e.g., user-interacted items) retrieved within the top k recommendations, and NDCG@k evaluates both the relevance and positional importance of ranked lists. Specifically, we employ Recall@10 (R@10), Recall@20 (R@20), NDCG@10 (N@10), and NDCG@20 (N@20). We report the metrics of all users in the test dataset. We follow the popular evaluation setting, splitting the data randomly into 8:1:1 for training, validation, and testing.

4.1.4. Implementation Details

Our proposed model is implemented using PyTorch [45]. We use an embedding dimension of 64 for both users and items across all models for consistency. The embedding parameters are initialized using Xavier initialization and optimized using the Adam optimizer [46] with a mini-batch size of 1024. The number of cross-residual layers is set to 6 and the first layer for all modalities is dropped. Other hyperparameters specific to our method, such as weighting factors for different modalities or graph construction parameters, are tuned empirically on the validation set. We train for a maximum of 500 epochs and employ early stopping based on the performance (e.g., Recall@20) on the validation set, stopping if performance does not improve for 20 consecutive epochs.

4.2. Performance Comparison

Table 2. Performance comparison: models ranked by metric (with improvement).

R@10 Ranking				R@20 Ranking
Model	Sports	Baby	Clothing	Model	Sports	Baby	Clothing
GC4MRec (Ours)	0.0784	0.0677	0.0645	GC4MRec (Ours)	0.1178	0.1074	0.0959
DGVAE	0.0753 4.12%	0.0636 6.45%	0.0619 4.20%	DGVAE	0.1127 4.53%	0.1009 6.44%	0.0917 4.58%
FREEDOM	0.0717 9.34%	0.0627 7.97%	0.0629 2.54%	FREEDOM	0.1089 8.17%	0.0992 8.27%	0.0941 1.91%
SLMRec	0.0671 16.84%	0.0504 34.33%	0.0447 44.30%	SLMRec	0.0998 18.04%	0.0768 39.84%	0.0662 44.86%
MMSSL	0.0652 20.25%	0.0620 9.19%	0.0492 31.10%	MMSSL	0.0981 20.08%	0.0969 10.84%	0.0783 22.48%
LATTICE	0.0588 33.33%	0.0528 28.22%	0.0483 33.54%	LATTICE	0.0926 27.21%	0.0842 27.55%	0.0726 32.09%
VBPR	0.0542 44.65%	0.0450 50.44%	0.0277 132.85%	VBPR	0.0849 38.75%	0.0668 60.78%	0.0405 136.79%
LightGCN	0.0527 48.77%	0.0503 34.59%	0.0371 73.85%	LightGCN	0.0832 41.59%	0.0766 40.21%	0.0547 75.32%
MMGCN	0.0401 95.51%	0.0390 73.59%	0.0220 193.18%	MMGCN	0.0632 86.39%	0.0624 72.12%	0.0347 176.37%
MF-BPR	0.0451 73.84%	0.0346 95.66%	0.0177 264.41%	MF-BPR	0.0662 77.95%	0.0566 89.75%	0.0252 280.56%
N@10 Ranking				N@20 Ranking
Model	Sports	Baby	Clothing	Model	Sports	Baby	Clothing
GC4MRec (Ours)	0.0421	0.0360	0.0349	GC4MRec (Ours)	0.0523	0.0462	0.0429
DGVAE	0.0410 2.68%	0.0340 5.88%	0.0336 3.87%	DGVAE	0.0506 3.36%	0.0436 5.96%	0.0412 4.13%
FREEDOM	0.0385 9.35%	0.0330 9.09%	0.0341 2.35%	FREEDOM	0.0481 8.73%	0.0424 8.96%	0.0420 2.14%
MMSSL	0.0369 14.09%	0.0333 8.11%	0.0271 28.78%	MMSSL	0.0462 13.20%	0.0427 8.20%	0.0345 24.35%
SLMRec	0.0354 18.93%	0.0299 20.40%	0.0231 51.08%	SLMRec	0.0445 17.53%	0.0367 25.89%	0.0299 43.48%
LATTICE	0.0311 35.37%	0.0281 28.11%	0.0251 39.04%	LATTICE	0.0399 31.08%	0.0352 31.25%	0.0311 37.94%
VBPR	0.0310 35.81%	0.0236 52.54%	0.0155 125.16%	VBPR	0.0401 30.42%	0.0299 54.52%	0.0189 126.98%
LightGCN	0.0291 44.67%	0.0250 44.00%	0.0192 81.77%	LightGCN	0.0354 47.74%	0.0335 37.91%	0.0248 72.98%
MMGCN	0.0189 122.75%	0.0207 73.91%	0.0115 203.48%	MMGCN	0.0244 114.34%	0.0284 62.68%	0.0153 180.39%
MF-BPR	0.0244 72.54%	0.0205 75.61%	0.0100 249.00%	MF-BPR	0.0301 73.75%	0.0250 84.80%	0.0119 260.50%

Note: Improvement is calculated as (GC4MRec Value − Model Value)/Model Value × 100.

presents the performance comparison of our proposed GC4MRec method against several state-of-the-art baselines on three datasets. We observe the following key trends:

1. Effectiveness of Graph-Based and Multimodal Methods: GCN-based methods consistently outperform MF-based models (MF-BPR and VBPR) across all datasets and metrics. This highlights the effectiveness of modeling user–item interactions as a graph and leveraging graph convolutional operations. Furthermore, incorporating multimodal information generally leads to improved performance. Multimodal methods like VBPR, LATTICE, FREEDOM, SLMRec, and MMSSL outperform MF-BPR in most cases. This demonstrates the value of leveraging visual and textual features for enriching item representations and improving recommendation quality.

2. Superiority of GC4MRec: Our proposed GC4MRec method achieves the best performance across all datasets and metrics. It surpasses the strongest baseline of DGVAE, with improvements on Sports, Baby, and Clothing. GC4MRec achieves relative improvements of 4.12%, 6.45% and 4.20%, respectively, in terms of R@10 over DGVAE.

These gains demonstrate the effectiveness of GC4MRec.

3. Impact of Multimodal Features: While multimodal methods generally perform better, the effectiveness of multimodal features can vary across datasets. For example, DualGNN significantly improves upon LightGCN on Clothing, where visual and textual information likely plays a more crucial role in characterizing items, but shows less pronounced gains on Sports and Baby. This suggests that the informativeness of multimodal features can be dataset-dependent.

4. Benefits of Graph Structure Refinement: Methods that refine or augment the graph structure, such as FREEDOM (with its denoising and freezing components) and LATTICE (with its latent item–item graph), tend to perform well. FREEDOM consistently achieves strong results, highlighting the importance of mitigating noise in user–item interactions and leveraging robust item relationships.

5. Comparison with Self-Supervised Methods: Self-supervised methods like SLMRec, BM3, and MMSSL demonstrate competitive performance, showcasing the potential of leveraging self-supervision for learning better representations. However, GC4MRec still outperforms these methods. Through extensive experiments on three publicly available datasets, the proposed GC4MRec model demonstrably surpasses state-of-the-art models, highlighting its effectiveness compared to existing self-supervised methods.

Our work introduces a novel approach for enhancing subgraph representation through generative contrastive learning. By generating positive sample pairs from augmented subgraphs, the model effectively amplifies the representation of latent features within the data. Furthermore, a novel dual spatio-temporal cross-residual module is proposed, effectively fusing information from the original and augmented subgraphs, enabling the capture of dynamic user preference shifts across different temporal contexts. A key innovation lies in the channel cross fusion convolution strategy, dynamically learning weights for metadata across different modalities. This enables effective multimodal information fusion, leading to more accurate preference recommendations for users.

These gains demonstrate the effectiveness of GC4MRec. Its superiority is further corroborated by the performance curves depicted in Figure 5. GC4MRec continues to improve with increasing epochs, eventually converging to significantly higher values for both metrics, indicating that GC4MRec’s advantage becomes more pronounced with extended training. This suggests that GC4MRec is better able to leverage the available data and learn more nuanced representations over time. This visual analysis further underscores the effectiveness of our proposed bilateral information flow module, generative augmentation via VGAE, and metadata preference modeling in capturing latent user preferences for improving recommendation accuracy. The improvement across datasets and metrics, as observed in Table 2, provides compelling evidence for the robustness and generalizability of GC4MRec.

4.3. Ablation Study

To thoroughly examine the impact of individual components within our proposed GC4MRec model, as shown in Figure 6, a comprehensive ablation study was conducted across three distinct real-world datasets. This analysis aims to isolate and quantify the contributions of each component to the overall model performance. The following GC4MRec variants were systematically evaluated to achieve this objective:

* GC4MRec-GCN-ONLY: This variant removes all modules from GC4MRec, using only GCN. This helps analyze the impact of the layer-wise convolution approach. We can see that GC4MRec always outperforms GC4MRec-GCN-ONLY.

* GC4MRec-NoModalityFusion: This variant removes the multimodal fusion strategy from GC4MRec and retains other modules. This assesses the impact of multimodal fusion on performance. The ablation shows that the performance drops by 3.5%, 3.5%, and 5.5% respectively, compared with GC4MRec, indicating that capturing information from multiple modalities fusion is essential for accurate recommendations.

* GC4MRec-NoGraphRefinement: This variant ablates the graph refinement process. We observe a performance decrease of 3%, 3%, and 5.4% when graph refinement is removed, showing that refining the graph structure is crucial for improving recommendation accuracy.

These ablation results demonstrate that each component of GC4MRec contributes to its overall performance. Removing any of these components leads to a noticeable decrease in recommendation accuracy, validating the design choices and effectiveness of our proposed method.

4.4. Hyperparameter Combination Effectiveness

We evaluate the performance of GC4MRec under different combinations of learning rate and regularization weight. The learning rate is varied from 1 × 10⁻¹ to 1 × ${10}^{-6}$, and the regularization weight from 1 × 10⁻⁵ to 1 × ${10}^{-1}$. Figure 4 shows the R@20 and N@20 scores achieved by GC4MRec on the Clothing, Baby, and Sports datasets, respectively. The results reveal that the performance of GC4MRec varies across datasets and is sensitive to the choice of both learning rate and regularization weight. For instance, on the Clothing dataset, a higher learning rate generally leads to better R@20 performance, while for the Baby dataset, a moderate learning rate coupled with a smaller regularization weight yields the best results. Further analysis reveals that the optimal hyperparameter configuration appears to be dataset-specific. Further investigation is needed to understand the interplay between these hyperparameters and the characteristics of each dataset.

5. Conclusions

In this work, we proposed GC4MRec, a novel self-supervised learning framework for multimodal recommendation. GC4MRec leverages contrastive learning with a Variational Graph Autoencoder (VGAE) for generating augmented subgraphs, enabling the accurate learning of latent features and high-order user–item connectivity. This approach addresses a key limitation of existing contrastive learning methods in recommendation systems: the difficulty in defining effective pretext tasks and the sparsity of interaction data. The augmented subgraphs provide rich positive sample pairs, enhancing the model’s ability to capture latent user preferences. Furthermore, a novel bidirectional cross-temporal residual module effectively fuses information from both original and augmented subgraphs, capturing dynamic shifts in user preferences across different temporal contexts, a critical aspect often overlooked by static fusion strategies. The introduction of a cross-channel fusion convolution module (CCFC) allows for the dynamic learning of metadata weights from different modalities, enabling more effective and personalized multimodal information fusion compared to simpler methods like concatenation or static weighted averaging.

Extensive experiments on three public datasets demonstrate that GC4MRec outperforms state-of-the-art models. These improvements are not just marginal but reflect GC4MRec’s ability to effectively model complex multimodal interactions and dynamic user preferences. For instance, the significant improvement on the Sports dataset in Recall@10 illustrates GC4MRec’s effectiveness in capturing long-tail user preferences and recommending relevant niche items, a direct result of the metadata-aware fusion mechanism and graph refinement. Ablation studies further validate the contributions of each component (the VGAE-based augmentation, the bidirectional residual module, and the CCFC fusion strategy), showcasing their synergistic effect in achieving superior recommendation accuracy.

The implications of GC4MRec extend beyond the datasets explored here. Its potential is applicable in various recommendation scenarios, such as social media feeds, news aggregation platforms, and e-commerce product suggestions, where diverse modalities and dynamic user behaviors are prevalent. Future work will focus on enriching data diversity by incorporating additional modalities, such as user reviews or product descriptions. This will provide a more holistic view of user preferences and item characteristics. Moreover, we will investigate the integration of large language models (LLMs). Specifically, we envision employing LLMs for generating synthetic user profiles or augmenting item descriptions, addressing the cold-start problem and enhancing the semantic representation of items. We also plan to explore personalized prompt engineering techniques for LLMs tailored to individual user preferences and item characteristics. This will enable more fine-grained control over the generation process and lead to even more accurate and personalized recommendations. By addressing these challenges and exploring these promising directions, GC4MRec provides a robust framework and paves the way for the next generation of multimodal recommendation systems.