Graph-Based Feature Crossing to Enhance Recommender Systems

Abstract

In recommendation tasks, most existing models that learn users’ preferences from user–item interactions ignore the relationships between items. Additionally, ensuring that the crossed features capture both global graph structures and local context is non-trivial, requiring innovative techniques for multi-scale representation learning. To overcome these difficulties, we develop a novel neural network, CoGraph, which uses a graph to build the relations between items. The item co-occurrence pattern assumes that certain items consistently appear in pairs in users’ viewing or consumption logs. First, to learn relationships between items, a graph whose distance is measured by Normalised Point-Wise Mutual Information (NPMI) is applied to link items for the co-occurrence pattern. Then, to learn as many useful features as possible for higher recommendation quality, a Convolutional Neural Network (CNN) and the Transformer model are used to parallelly learn local and global feature interactions. Finally, a series of comprehensive experiments were conducted on several public data sets to show the performance of our model. It provides valuable insights into the capability of our model in recommendation tasks and offers a viable pathway for the public data operation.

1. Introduction

In our daily activities, such as online shopping [1,2], entertainment consumption [3,4], and social networks [5,6], the constant introduction of new items poses a challenge for users to make informed decisions [7,8]. The abundance of choices can lead to fatigue and confusion in decisions, making it challenging for users to choose the most suitable options according to their needs, prompting users to seek assistance from recommender systems to streamline their selection process and alleviate decision-making burdens.

Recent progress in deep learning has shown promising results for recommendation systems, with advances in this field playing an essential role in enhancing the capabilities of recommendation algorithms [9,10]. Graph Neural Networks (GNNs) are designed to operate directly on graph structures. The core features of GNNs include realizing feature learning by aggregating node neighbor information, capturing complex non-Euclidean relationships in graphs, and flexibly adapting heterogeneous graphs and dynamic graphs [11]. Prominent GNN types include Graph Convolutional Networks (GCNs) [12], Graph Attention Networks (GATs) [13], Graph Autoencoders [14], and Adversarially Regularized Graph Autoencoders [15], each tailored for tasks such as representation learning and unsupervised feature extraction [16]. In addition to recommendation tasks, GNNs have been successfully employed to improve the reliability and efficiency of power systems, especially under high uncertainty from renewable energy sources [16]. In addition, a multi-fidelity GNN framework has demonstrated the potential to combine high-precision and low-precision data, thereby reducing computational demands and enhancing model robustness across diverse scenarios in power systems [17]. Despite these advancements, many existing recommendation systems focus primarily on user–item interactions, overlooking the intricate relationships that exist between different items within the system [4,18,19]. An item co-occurrence pattern is proposed to discover such relationships that assumes that some items always occur in pairs in users’ favorite lists. Figure 1 demonstrates a toy example of a movie co-occurrence pattern. In users’ viewing logs, it is always easy to view Harry Potter movies sequentially.

Recently, co-occurrence patterns have been explored to improve recommendation quality. For example, to improve Matrix Factorization (MF) performance, Liang et al. [18] first proposed an item co-occurrence pattern and combined it with MF, providing a new perspective to exhibit a connection between two items. Wu et al. [20] added item co-occurrence information to metric learning to provide a more detailed representation of the relationships between items. To achieve a higher recommendation quality, Chen et al. [21] proposed CoCNN to learn accurate features by directly applying a Convolutional Neural Network (CNN) to an item co-occurrence pattern. These models fully use the potential information of the user–item interactions. However, they are still confronted with key challenges; e.g., it is extremely stiff when it comes to capturing both global graph structures and local context from the user–item interactions.

To tackle the above difficulties, we designed a co-occurrence graph neural network (CoGraph), which utilizes a graph to link all items. Through powerful representation learning capabilities, graph methods have a strong ability to learn accurate features by message functions and a neighborhood aggregation strategy [22,23,24]. Next, to achieve better performance, Normalised Point-Wise Mutual Information (NPMI) [25] is applied to evaluate the strength of the item connection. According to NPMI, we built an undirected graph based on historical data and applied a Graph Neural Network (GNN) to model item relationships and generate precise embeddings.

Recent studies [26,27,28,29] show that feature crossing improves the performance of deep models. CNNs are powerful in acquiring local features from inputs [30,31], and Transformer networks demonstrate a strong capacity to learn the dependencies between elements of input features [32,33,34]. Thus, in this paper, to fully and deeply explore the relationships between features, CNNs and Transformer networks, which complement each other, are applied to parallelly capture local and global feature information, respectively.

Our main contributions are as follows:

$\left(1\right)$

To enhance recommendation systems, we developed a novel framework to consider both user–item and item–item correlations simultaneously.

$\left(2\right)$

To learn accurate features of items, co-occurrence relationships are built by a graph, in which NPMI is used to measure the distance.

$\left(3\right)$

To capture the interactions across features, CNNs and Transformer networks are used to parallelly learn both local and global levels of user intentions.

$\left(4\right)$

Detailed experiments on four public data sets demonstrate that our CoGraph model outperforms some competitive baseline methods.

2. Related Work

2.1. Feature Crossing-Based Recommendation

To learn information from user–item interactions, the following two methods are commonly used: feature crossing in neural networks and attention mechanisms.

Feature crossing in neural networks. Wang et al. [35] designed a cross-network to apply explicit feature crossings efficiently. To model feature interactions between items, Wang et al. [36] conserved collaborative signals with messages passing by element-wise products between two connected nodes. To capture fine-grained interactions between individual behaviors, Xie et al. [37] learned user unbiased preferences from different types of positive/negative and explicit/implicit feedback.

Attention mechanism. An attention mechanism is effective in learning a deeper representation of each item by analyzing its relationships with other items [38]. For example, the self-attention of Transformer models the relationships between items and, by memorizing dependencies, concentrates on important parts of inputs [2,39]. To capture high-order dynamic information in a sequence, self-attention learns the representation by modeling the long-range dependencies between collaborative items [40,41]. Rather than treating preferences uniformly, as most existing models do, attention-based models distinguish users’ preferences over items by automatically assigning weights to different items [4,42].

Both solutions completely use implicit knowledge of user–item interactions and show that the item relationships effectively improve recommendation quality. In this paper, we use CNNs and Transformer to parallelly learn both local and global levels of feature interactions.

2.2. Graph-Based Recommendation

In graph-based recommendation [43], the user and item are represented as nodes, and the users’ behaviors are considered as edges. Hence, the potential relations between the user and the item can be obtained by executing path exploration, and further realizing the recommendation. Compared with traditional recommendation approaches, graph-based recommendation facilitates the representation of interactions among users and items.

To handle the problem of high-order connectivity modeling, Wang et al. [44] proposed a Neural Graph Collaborative Filtering (NGCF) method based on a Graph Convolutional Network (GCN) to explicitly encode the high-order interconnections in the interaction graph to capture richer collaborative filtering signals. Based on this, Ye et al. [45] presented an improved NGCF by denoising the structure of the user–item graph and adding random noise into embeddings to enhance the robustness of NGCF. Because of the different weights and semantics of multiple types of user behavior, Jin et al. [46] introduced an approach to construct a heterogeneous graph network based on a GCN to improve performance. Meanwhile, He et al. [47] proposed a minimized GCN that only retains the neighborhood aggregation to lighten the model structure. Zhang et al. [19] designed a graph-based regularization method to enhance the quality of representations in the embedding layers, enabling a more effective capture of relationships between users and items.

With the exception of only considering the user–item graph as a reference, Fan et al. [5] designed a GraphRec model based on a GNN that aggregates user–item graphs and user–user social graphs to enhance the modeling performance of users. To address the issue of cold start, Cai et al. [48] introduced an approach based on a heterogeneous GNN, which applies a hierarchical attention aggregation to utilize the sparse attribute of new users and the heterogeneous relationship between existing users and items, thereby generating high-quality user representation. In terms of capturing high-order proximity, [49] proposed a method to boost the accuracy of implicit recommendation, through random-walk on user–item graphs to capture high-order proximity information. In addressing the problem of processing large-scale edges and nodes, Ying et al. [50] presented a GCN-based approach combined with random-walk to obtain essential proximity information from large-scale graphs.

Recent studies further showcase the evolution of graph-based recommendation methods. For instance, Bansal et al. [51] proposed a multilingual personalized hashtag recommendation system utilizing GNNs to improve hashtag relevance in low-resource Indic languages, addressing linguistic diversity challenges. Further, Chen et al. [52] introduced the CT-GNN model, integrating temporal information into GNNs for recommendations, enhancing accuracy and diversity by capturing temporal associations. The authors of Sedhain et al. [53] developed a knowledge graph-based recommendation system enriched by neural collaborative filtering, which leverages multi-hop neighborhood information to resolve cold-start and redundancy issues. Finally, Wu [54] presented KGAT-AX, which integrates auxiliary information into knowledge graph embeddings via an attention mechanism, achieving superior performance in personalized recommendations.

3. Proposed Model

3.1. Preliminary

Consider two sets: U, consisting of m users, and I, containing n items. Their interaction forms a matrix, $R\in {\mathbb{R}}^{m\times n}$, whose element $y_{ui}$ is defined as follows:

$$y_{ui}=\left\{\begin{array}{cc}1,& \mathrm{if}\mathrm{user}u\mathrm{has}\mathrm{interacted}\mathrm{with}\mathrm{item}i\\ 0,& \mathrm{otherwise}\end{array}\right..$$

(1)

Next, we establish a co-occurrence matrix, $S\in {\mathbb{R}}^{n\times n}$, whose element $s_{ij}$ is the distance between i and j. To retrieve valuable insights from a co-occurrence pattern, the relationship between items is modeled as a weighted bipartite graph, $G=\left\{(i,s_{ij},j)\right|i,j\in I\}$. In graph G, S also means the adjacent matrix. A graph embedding function maps the nodes of a graph into a lower-dimensional space [55].

3.2. Architecture

To learn accurate embeddings from user–item interactions, a graph-based approach is used to capture co-occurrence relationships effectively. Moreover, we enrich the model’s representation capabilities by incorporating feature cross-interactions at both local and global levels. Thus, CoGraph, with a co-occurrence pattern, has three key designs: (1) a weighted bipartite graph describing the relationships between items; (2) a CNN learning local feature crossings; and (3) Transformer learning global feature crossings. Figure 2 shows the framework of CoGraph.

3.2.1. Co-Occurrence Graph

We use a graph to learn embeddings from a co-occurrence pattern. A pattern is represented as a graph, where each edge is labeled with a weight to indicate the distance between items. Given two items, i and j, NPMI [25] measures the distance, defined as follows:

$$\begin{array}{c}PMI\left(i,j\right)=log{\displaystyle \frac{P\left(i,j\right)}{P\left(i\right)P\left(j\right)}}\approx log{\displaystyle \frac{\#\left(i,j\right)·N}{\#\left(i\right)·\#\left(j\right)}},\end{array}$$

(2)

$$\begin{array}{c}NPMI(i,j)={\displaystyle \frac{PMI(i,j)}{-log\left(P\left(i,j\right)\right)}},\end{array}$$

(3)

where $N={\sum }_i{\sum }_j\#\left(i,j\right)$, and $\#\left(·\right)$ is the number of items. $\#(i,j)$ denotes the number of items i and j co-occurring in the data; $\#\left(i\right)$ and $\#\left(j\right)$ denote the numbers of the items i and j, respectively.

Compared to other metrics like PMI, which can produce unbounded values and may be difficult to interpret, NPMI provides a standardized measure. Additionally, while correlation coefficients can indicate linear relationships, they may not adequately capture the nuances of probabilistic relationships, especially in the case of categorical data. NPMI’s probabilistic foundation allows it to reveal associations that may not be linear, making it particularly suited for our analysis. Recent studies [56,57] have successfully employed NPMI to analyze relationships in similar contexts, illustrating its effectiveness as a metric for understanding associations in complex datasets.

Some co-occurrence links may be redundant or irrelevant. Thus, considering all co-occurrence links may degrade learning performance. Only important and useful links should be generated. To obtain accurate embeddings, it is necessary to prune the graph. Thus, we use the threshold, k, to filter noisy edges. Finally, the distance of node, $s_{ij}$, is used to measure graph connectivity and create an edge list. The operation is defined as follows:

$$s_{ij}=max\left({\displaystyle \frac{PMI(i,j)-log\left(k\right)}{-log\left(P\left(i,j\right)\right)}},0\right).$$

(4)

In the construction of the graph, (1) a user–item interaction matrix is used to calculate co-occurrence matrices for both users and items, which quantify how often users interact with the same items and how often items are rated by the same users; (2) after calculating PMI scores, which reflect the strength of association between users and items based on their co-interaction patterns, we obtain NPMI scores to facilitate comparisons across different user–item pairs; and (3) we create an edge list that captures the connections between nodes ($s_{ij}$) based on their interactions, resulting in a graph representation.

GCNs, one of the most popular and promising methods for graphs, are used to generate embeddings of items. In the co-occurrence graph, G, to obtain the embedding of an item i, $h_i$, we first randomly sample neighbors from its neighbor set, $N\left(i\right)$, and then perform the graph convolution as follows:

$$\begin{array}{c}{h}_{N\left(i\right)}^l=AGGREGATE\left(\left\{s_{ij}·h_j^{l-1},\forall j\in N\left(i\right)\right\},W_g\right),\end{array}$$

(5)

$$\begin{array}{c}{h}_i^l=\sigma \left(W_c·CONCAT\left(h_i^{l-1},h_{N\left(i\right)}^l\right)\right),\end{array}$$

(6)

where $\sigma$ denotes the sigmoid function.

3.2.2. CNN Learning

By using the power of the receptive field, Convolutional Neural Networks (CNNs) effectively retrieve meaningful and abstract features from input data [21,58,59]. Our model introduces two innovative approaches to thoroughly investigate and understand the relationships between user–item interactions and item–item connections: (1) learning local feature interaction by a CNN and (2) global feature interaction by a Transformer network.

To construct a connection matrix that effectively links user and item representations, we define their input, $x_{ui}$, as follows:

$$x_{ui}=\left[e_u;e_i;h_i\right]\in {\mathbb{R}}^{3\times d},$$

(7)

where $e_u$ and $e_i$ denote the embedding of the user and item, respectively; $h_i$ is a contextual feature from an item graph; d denotes the dimensionality of each embedding.

Then, we apply a series of convolutional filters to the connection matrix $x_{ui}$ in order to extract meaningful features. The filtering operation is defined as follows:

$$c=f\left(W\otimes x_{ui}+b\right),$$

(8)

where W represents a filter with a size of $2\times 2$; ⊗ is a convolution operation; b denotes the bias; and $f(·)$ denotes an activation function.

We design CNN learning for processing input data through a series of convolutional layers, incorporating both 2D and 1D convolutions. The network takes input data, reshapes them to fit the 2D convolutional layer, applies the convolution and activation, then reshapes the output for a subsequent 1D convolutional layer, which further processes the data before producing a final output through another ReLU activation. The output size is determined based on the transformations applied during the convolutions, ultimately yielding a flattened output vector suitable for the further task.

3.2.3. Transformer Learning

In CNNs, the receptive field, with a limited ability to obtain the interaction of global features, relies on the size of the convolutional kernel and the depth of the layers. To further investigate the intricate connections among elements within the features, a Transformer encoder network has been carefully constructed to capture a wide global receptive field. Transformer aims to capture the global interactions among features by effectively constructing comprehensive relationships between users and their associated items. The architecture of the Transformer has N identical sublayers, with each sublayer consisting of two critical modules: multi-head self-attention and feed-forward mechanisms [33,60,61].

Multi-head self-attention is a crucial element of the Transformer architecture; it enables the model to simultaneously attend to various parts of the input sequence. This module determines the significance of each element within the sequence, facilitating the capture of long-range dependencies and relationships. A multi-head self-attention mechanism, characterized by h heads, initially learns representations simultaneously across multiple heads before concatenating all the acquired representations into a single output. This process can be mathematically expressed as follows:

$$\mathit{MultiHead}\left(x\right)=\left[hea{d}_1;hea{d}_2;\cdots ;hea{d}_h\right]W^O,$$

(9)

where $W^O$ represents a set of trainable weights that facilitate the final transformation of the concatenated representations.

Given an input $x_{ui}$, the attention scores for each pair of features are computed using the self-attention mechanism. Each individual attention head, indexed by $hea{d}_i$ (where $1\le i\le h$), is defined as follows:

$$hea{d}_i=\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\left({\displaystyle \frac{Q{K}^T}{\sqrt{d_k}}}\right)V,$$

(10)

where Q, K, and V represent the query, key, and value matrices, respectively; $d_k$ denotes the dimension of the key vectors. The softmax function is used to normalize the attention scores, assigning suitable weights to various features in the input.

The feed-forward mechanism, on the other hand, includes two linear transformations separated by a ReLU activation function. This module helps the model learn complex non-linear relationships within the data. The feed-forward mechanism in each sublayer is defined as follows.

$$\mathrm{FFN}\left(x\right)=\mathrm{ReLU}(x{W}_1+b_1)W_2+b_2,$$

(11)

where $W_1$, $b_1$, $W_2$, and $b_2$ are learnable parameters and ReLU denotes the rectified linear unit activation function. This module applies two linear transformations with a non-linear activation function in between, enabling the model to learn complex patterns in related data.

Together, these two modules work in harmony within each sublayer of the Transformer architecture to process and transform the input data. By stacking sublayers sequentially, the Transformer learns complex patterns and relationships within the data, making it a potent tool for a variety of tasks.

3.2.4. Prediction

Once CNN co-occurrence feature $c_{ui}$ and Transformer feature $t_{ui}$ have been extracted, they are merged to create a unified representation that captures feature interactions at both local and global levels within the data. To construct the preference representation $v_{ui}$, we concatenate the CNN and Transformer features as follows:

$$v_{ui}=\left[c_{ui};t_{ui}\right].$$

(12)

Subsequently, the merged representation $v_{ui}$ is fed into a Multi-Layer Perceptron (MLP) to learn complex non-linear mappings from input data to output predictions. The final prediction, denoted as ${\widehat{y}}_{ui}$, is generated by applying the MLP to the concatenated representation:

$${\widehat{y}}_{ui}=f_{MLP}\left(v_{ui}\right).$$

(13)

3.3. Optimization

In recommendation scenarios, the binary cross-entropy optimization criterion is widely utilized to assess loss during training. Let O and $O^{-}$ represent a set of observed and unobserved interactions, respectively, the loss function can be defined by

$$\mathcal{L}=-\sum _{\left(u,i\right)\in O\cup O^{-}}{y}_{ui}log{\widehat{y}}_{ui}+\left(1-y_{ui}\right)log\left(1-{\widehat{y}}_{ui}\right),$$

(14)

where $y_{ui}$ is the ground truth label for the user–item pair $(u,i)$, which takes a value of 1 if the interaction is observed (i.e., $(u,i)\in O$) and 0 if it is unobserved (i.e., $(u,i)\in O^{-}$); ${\widehat{y}}_{ui}$ represents the predicted probability that the user u will interact with item i, as output by the model.

By minimizing this loss function, the model is optimized to produce high probabilities for observed interactions and low probabilities for unobserved ones, thus improving its predictive accuracy. The binary cross-entropy loss is particularly effective because it provides a smooth gradient that facilitates the training of neural networks, allowing for efficient convergence towards optimal parameters.

4. Experiments

4.1. Experimental Setup

4.1.1. Datasets

We evaluated our model’s performance using four datasets: Lastfm, MovieLens100K, MovieLens1M, and BookCross, all of which are accessible online through the GroupLens platform (https://grouplens.org/datasets (accessed on 1 May 2024)). Lastfm is a music recommendation dataset suitable for evaluating the model’s ability to handle sparse data with its sparsity. MovieLens100K and MovieLens1M are benchmark datasets in movie recommendation that are suitable for testing the performance of models at different data sizes. BookCross provides a book recommendation scenario which tests the model’s generalizability across domains. Important statistics such as the number of users, items, and interactions for each dataset are summarized in

Table 1. Statistics for all data sets.

	Lastfm	MovieLens100K	MovieLens1M	BookCross
User	518	943	6040	19,571
Item	3488	1682	3706	39,702
Interaction	46,172	100,000	1,000,209	605,178
Density (%)	2.556	6.305	4.468	0.078

. Consistent with established evaluation procedures found in prior studies like [62,63,64], we applied a leave-one-out evaluation technique to evaluate the performance of the proposed model.

4.1.2. Metrics

The performance of our model is assessed using two primary evaluation metrics: Normalized Discounted Cumulative Gain (NDCG) and Hit Ratio (HR). NDCG, a ranking metric, assigns greater significance to correct recommendations positioned at the higher ranks of the recommendation list [65]. Specifically, NDCG@K is computed as follows:

$$\mathrm{NDCG}@\mathrm{K}={\displaystyle \frac{1}{\mathrm{IDCG}@\mathrm{K}}}\sum _{i=1}^K{\displaystyle \frac{2^{{\mathrm{rel}}_i}-1}{{log}_2(i+1)}},$$

(15)

where ${\mathrm{rel}}_i$ denotes the relevance score of the item at rank i, and $IDCG@K={\sum }_{i=1}^K{\displaystyle \frac{2^{{\mathrm{rel}}_i^{*}}-1}{{log}_2(i+1)}}$ is the ideal DCG@K, computed based on the relevance scores ${\mathrm{rel}}_i^{*}$ of items in the optimal ranking order.

On the other hand, HR@K, a common metric for evaluating recommendation quality, emphasizes the accuracy of the recommendations delivered:

$$\mathrm{HR}@\mathrm{K}={\displaystyle \frac{\mathrm{Number}\mathrm{of}\mathrm{hits}@K}{\mathrm{Number}\mathrm{of}\mathrm{users}}},$$

(16)

where $\mathrm{hits}@K$ refers to the number of users for whom at least one relevant item appears within the top K recommendations.

4.1.3. Baseline Methods

To verify the effectiveness of CoGraph, we compared our model against the following methods:

BPR [63], which assigns a higher score to an observed interaction than its unobserved counterparts.

NeuMF [64], which combines MLP with MF to capture high-rank and low-rank features, respectively.

CoFactor [18], which uses the shifted positive PMI to measure the distance between items. Then, co-occurrence item embedding regularizes the objective function of MF for better recommendations.

GCN [44], which first builds a graph by user–item interactions and then uses a GCN to learn the representations.

DeepLight [29], which models feature interactions in the shallow component.

CoCNN [21], which designs a framework to build the link between item–item and user–item and directly applies a CNN to an item co-occurrence pattern.

4.1.4. Parameter Settings

In our experiment, Python is used as the primary programming language for model development and data processing. Specifically, we employed PyTorch, a widely-used deep learning framework, for implementing our model architecture and training processes. PyTorch was chosen for its flexibility and ease of use, particularly for building complex neural networks and conducting experiments efficiently.

All methods have three key hyper-parameters: regularization coefficient ($\lambda$), learning rate ($\tau$), and feature size (d). We chose $\lambda$ from $\{{10}^{-4},{10}^{-3},{10}^{-2},{10}^{-1}\}$, $\tau$ from $\{{10}^{-4}$, ${10}^{-3},{10}^{-2}\}$, and d from $\{8,16,32,64,128,256\}$. Other hyper-parameters are determined by the default settings. To effectively tune hyperparameters, grid search was used to help us find the optimal set of hyperparameters. Finally, the average results of five random iterations were reported.

Our model is configured with multiple components: embeddings for users and items; a graph convolution layer for item relationships; CNNs for learning local features; a Transformer encoder for learning global features; and MLP for nonlinear mapping. We initialize the weights of the user and item embeddings using a normal distribution with a mean of 0 and a standard deviation of 0.1.

4.2. Results and Analysis

4.2.1. Overall Performance Comparison

Initially, a set of experiments was conducted to evaluate the performance of all methods on different datasets. The NDCG@10 and HR@10 scores on all datasets are summarized in

Table 2. HR@10 and NDCG@10 scores of all methods.

Datasets	Metrics	BPR	CoFactor	CoCNN	GCN	DeepLight	CoGraph
Lastfm	HR	0.7044	0.7134	0.7517	0.7610	0.7582	0.7831
	NDCG	0.4923	0.5155	0.5305	0.5382	0.5283	0.5508
MovieLens100K	HR	0.6801	0.6878	0.7083	0.7028	0.7006	0.7045
	NDCG	0.3949	0.4013	0.4099	0.4134	0.4055	0.4228
MovieLens1M	HR	0.6907	0.6981	0.7041	0.7086	0.7062	0.7154
	NDCG	0.4145	0.4176	0.4282	0.4253	0.4263	0.4321
BookCross	HR	0.2330	0.2768	0.3216	0.3145	0.3273	0.3336
	NDCG	0.1256	0.1503	0.1788	0.1705	0.1804	0.1913

. From the results in Table 2, the following conclusions are drawn:

First, it is evident that CoGraph consistently achieves the best results for both metrics in most cases. The consistent outperformance of GCN by CoGraph across all datasets highlights the fact that GCN is primarily designed to model user–item relationships and may lack the capability to capture feature interactions effectively. The possible reason might be that, in GCN, the propagation of item embeddings through neighboring users instead of items directly during graph embedding updates may limit the exploration of item relationships.

Second, in most cases, CoGraph demonstrates superior performance over CoCNN, suggesting that a basic feature concatenation strategy may not be sufficient in capturing the complex relationships between items. Using a co-occurrence graph in recommendation tasks is crucial for acquiring precise embeddings. Further analysis reveals that the limitations of CoCNN mainly lie in the following aspects: (1) CoCNN performs feature learning from the co-occurrence matrix primarily relying on CNNs, which makes it difficult to capture complex relationships between items. (2) CoCNN focuses more on local feature learning while failing to exploit the global graph structure. In contrast, CoGraph captures complex relationships between items through GNNs and employs parallel CNN and Transformer structures to explore feature interactions at local and global levels. However, in terms of HR, CoGraph exhibits inferior performance on the dense MovieLens100K dataset, with three potential explanations: (1) noisy item co-occurrence contexts in the dataset, leading to imprecise representations learned by CoGraph; (2) the uniform treatment of item relationships neglecting varying distances; and (3) the failure to model feature interactions, resulting in limited performance.

Finally, CoGraph outperforms the MF-based model, CoFactor, due to two critical advantages: the neural network foundation of CoGraph enables it to identify deep and non-linear features through its hidden layers, and its strategic use of an item graph ensures a more precise understanding of item relationships than can be achieved with co-occurrence regularization. High-quality recommendation systems necessitate a broader perspective that encompasses not only user–item but also item–item relationships, offering a richer and more accurate compilation of features.

4.2.2. Performance of Co-Occurrence Graph

Next, to investigate the effect of the co-occurrence graph within CoGraph, we introduce a simplified model of CoGraph, CoGraph-x, which lacks the co-occurrence graph. We then conduct a comparative analysis of the performances of CoGraph-x and CoGraph across several datasets, with the results shown in Figure 3.

From Figure 3, we can see that CoGraph achieves significantly better performance with larger HR and NDCG values than CoGraph-x on all datasets. The possible reason is that the features learned from the graph contain historical information about items; thus, the features represent items directly, whereas the features (embeddings) of CoGraph-x start with random initialization, leading to features without any useful information. Therefore, the co-occurrence pattern is effective in enriching user–item interactions, and the representations learned by the graph improve feature accuracy.

Moreover, we investigated the performance of graph depth in recommendation systems by adjusting the number of layers to one, two, three, and four. Our experiments were conducted on two datasets: the largest one, Movielens1M, and the smallest one, Lastfm, with the results illustrated in Figure 4. Our findings reveal that the recommendation performance reaches its peak when the graph consists of two layers. Conversely, increasing the number of layers beyond this optimal point diminishes the effectiveness of the graph, suggesting that a more complex structure may not always yield better results.

4.2.3. Parameter Sensitivity Study

There are two important hyper-parameters in the CoGraph model, i.e., the feature size d and the noisy edge k. In this subsection, we study the impacts of these parameters on CoGraph.

We empirically set the feature size d to 16, 32, 64, 128, and 256, respectively. As shown in Figure 5a, we can see that the performance of CoGraph is progressively higher in terms of HR on both Lastfm and Movielens1M, when the feature size increases. This indicates that CoGraph learns more useful information when the feature size is larger. Figure 5b illustrates that CoGraph performs relatively better with increasing feature sizes, in terms of NDCG on both Lastfm and Movielens1M. This result again verifies that CoGraph shows stronger representation ability when it has a larger feature size.

Subsequently, we explored the impact of the parameter k on the effectiveness of filtering out noisy edges in the graph. We selected values from the set $\{1,2,4,8,16,32,64,128\}$ for our analysis. Our experiments were carried out on two datasets: Lastfm and Movielens1M. The results are presented in Figure 6.

From Figure 6, these observations revealed that when k is greater than 0, the performance shows a slight improvement compared to when k is set to 0. This enhancement may be attributed to the effective filtering of certain noisy connections, allowing the graph neural network to learn more accurate features. However, as k increases significantly, we notice a decline in model performance. This suggests that overly aggressive filtering can lead to the loss of valuable information, ultimately resulting in a less effective model. Thus, while some degree of noise reduction is beneficial, excessive filtering may hinder the model’s ability to generalize and perform well.

4.2.4. Ablation Study

Finally, to evaluate the specific contributions of the CNN and Transformer components to the overall performance of the recommendation, we introduce two specific variants of CoGraph: CoGraph-CNN and CoGraph-Transformer. These variants were designed by omitting the CNN and Transformer segments from the original CoGraph model, respectively. All other conditions, including hyper-parameters like the regularization coefficient and learning rate, remained constant across these variants, matching those of the optimal configuration of CoGraph. To thoroughly examine their impact, we systematically tested feature sizes (d) from the set $\{32,64,128,256\}$ for CoGraph and its variants. The comparative results are shown in Figure 7.

Observed from the results in Figure 7, we conclude the following: CoGraph outperforms its simplified models across all datasets in evaluation metrics, highlighting the effectiveness of its feature crossing methods in autonomously learning feature interactions and thereby improving recommendation accuracy. In most scenarios, CoGraph-CNN presents the weakest performance, likely because it primarily learns localized feature interactions through the kernel and struggles with deep feature learning. Additionally, CoGraph demonstrates greater stability as the feature size (d) increases; stability is also enhanced by the additional fitting parameters of the Transformer component, which aids in achieving more accurate vectors and steadying the performance. Therefore, the CNN and Transformer components serve an indispensable role in the feature crossing process of CoGraph.

5. Conclusions

The potential of deep learning in recommendations is often hampered by an inadequate representation of item relationships. Thus, we propose a novel graph-based framework, namely, CoGraph, which fuses the capabilities of CNNs with Transformer to optimize recommendation performance. Our approach initiates with the creation of a graph that links items based on co-occurrence, enhancing representation depth. In addition, we adopt NPMI to assess item correlations and deploy a GNN to extract useful features from the graph. Finally, we integrate a CNN with a Transformer model to capture both local and global feature interactions, thereby enriching the feature discovery process. Experimental results show the success of CoGraph in extracting valuable features from co-occurrence data, underscoring its effectiveness for a range of recommendation tasks.

Nonetheless, the proposed method comes with some limitations and challenges when applied in practical settings. Firstly, it requires comprehensive data that capture the complex relationships between users, items, and context. Small or sparse datasets may not be sufficient for the model to learn meaningful patterns. In addition, CNNs and Transformers are often regarded as "black-box" models, which means it can be difficult to interpret or explain why a particular recommendation was made. In future work, we aim to explore several promising directions to extend the impact of our model. An area is the application of CoGraph in public large data operations, where the integration of publicly available data enables more robust cross-domain management and system improvement. Further work will also explore improving the interpretability of the crossed features generated by CoGraph, helping users better understand and trust the recommendation process.