De-Selection Bias Recommendation Algorithm Based on Propensity Score Estimation

Abstract

There are various biases present in recommendation systems, and recommendation results that do not consider these biases are unfair to users, items, and platforms. To address the problem of selection bias in recommendation systems, in this study, the propensity score was utilized to mitigate this bias. A selection bias propensity score estimation method (SPE) was developed, which takes into account both user and item information. This method accurately estimates the user’s choice tendency by calculating the degree of difference between the user’s selection rate and the selected preference of the item. Subsequently, the SPE method was combined with the traditional matrix decomposition-based recommendation algorithms, such as the latent semantic model (LFM) and the bias singular value model (BiasSVD). The propensity score was then inversely weighted into the loss function, creating a recommendation model that effectively eliminated selection bias. The experiments were carried out on the public dataset MovieLens, and root mean square error (RMSE) and mean absolute error (MAE) were selected as evaluation indicators and compared with two baseline models and three models with other propensity score estimation methods. Overall, the experimental results demonstrate that the model combined with SPE achieves a minimum increase of 2.00% in RMSE and 2.97% in MAE compared to its baseline model. Moreover, in comparison to other propensity score estimation methods, the SPE method effectively eliminates selection bias in the scoring data, thereby enhancing the performance of the recommendation model.

1. Introduction

The advent of the Internet and the subsequent era of big data have led to an overwhelming influx of information. Coping with the massive expansion of data has become increasingly challenging, making it arduous for users to extract useful information from this abundance [1]. To address the issue of information overload, the recommendation system has emerged as a solution. The primary objective of the recommendation system is to provide improved predictive outcomes by leveraging a user’s historical behavior data, analyzing user preferences, and generating product recommendations based on similarity.

Collaborative filtering technology serves as the foundation for most contemporary recommendation systems [2]. The fundamental concept underlying this approach is that users with similar preferences tend to make analogous choices in the future. The implementation of collaborative filtering algorithms commonly revolves around matrix decomposition. This technique involves decomposing the user-item rating matrix into smaller-dimensional matrices: a user preference matrix and an item attribute matrix. The user’s predicted rating for an item is then derived by computing the dot product of these two matrices.

Within the recommendation system, numerous biases emerge, including but not limited to selection bias, position bias, and popularity bias [3]. These biases undermine the system’s ability to accurately discern user preferences, consequently compromising the efficacy of the recommendation process [4,5,6]. The accuracy of recommendations serves as a crucial metric for evaluating the performance of recommendation systems. A higher level of accuracy directly correlates with improved system efficiency. In the task of predicting recommendation scores, the primary performance indicator is the recommendation accuracy, which measures the extent to which the predicted scores align with the actual scores, specifically by minimizing the discrepancy between them.

This paper aims to address the issue of selection bias in recommendation systems. Selection bias refers to the phenomenon where users have the freedom to select which products they rate, resulting in a propensity to rate their favored products more frequently. Consequently, the collected ratings fail to represent the true ratings of the products, leading to the missing not at random (MNAR) problem and resulting in missing rating data [7,8,9]. However, existing studies often assume that the rating data is missing at random (MAR), which is not realistic in practice. Since user-item interaction data in recommender systems is observational rather than experimental, selection bias is present in nearly all data. For instance, in a movie recommendation system, users tend to watch and rate movies they like while seldom providing ratings for movies they are uninterested in [10].

The inverse propensity score (IPS) is a method that offers an effective means to mitigate bias in recommender systems [11]. However, the algorithm’s ability to eliminate bias primarily relies on the accuracy of propensity score estimation. In the context of selection bias, past approaches primarily estimate propensity scores based on the marginal probability of users rating items, which inherently possess certain limitations.

The primary contributions of this paper are as follows:

(1)

We address the issue of selection bias in recommendation systems by developing a propensity score estimation method called the Selection bias Propensity score Estimator. This method takes into account both user and item information, providing a comprehensive approach to eliminating selection bias.

(2)

To construct a recommendation model that effectively removes selection bias, we integrate the SPE method with two traditional matrix factorization-based recommendation algorithms. We additionally introduce inverse weighting of the propensity score into the loss function, enabling the model to mitigate the impact of selection bias during the recommendation process.

(3)

Experimental evaluations are conducted on the widely used MovieLens dataset. The models we choose to compare include two traditional models and three innovative models which incorporate novel propensity score estimation methods proposed in recent years. The experimental results demonstrate that the SPE method significantly mitigates selection bias in the scoring data, leading to improved performance across all recommendation models.

The rest of this paper is organized as follows: Section 2 provides an overview of the current research landscape regarding selection bias in recommendation systems. Section 3 presents the novel propensity score estimation method proposed in this paper. It also details how this method is integrated with traditional recommendation algorithms to construct a recommendation model capable of effectively mitigating selection bias. Moreover, Section 4 provides the experimental design and analysis of results. Finally, Section 5 summarizes the specific contributions and implications of this paper and discusses possible future research directions.

2. Related Work

The issue of fairness in recommender systems has garnered increasing attention, leading researchers to focus on addressing selection bias. Various methods have been proposed to eliminate selection bias, such as inverse propensity scoring, meta-learning, homogeneous data, and causal diagrams. These approaches aim to mitigate the impact of selection bias and promote fair recommendations.

Wang et al. [12] introduced a novel approach that utilized two causal graphs to depict the generation process of biased and unbiased feedback within recommender systems. Their model incorporated the learning of biased embedding vectors, which consisted of independent biased and unbiased components, thus imposing constraints on the learning process. The presence of selection bias gives rise to issues of non-random missing data. Zhang et al. [13] introduced the concepts of item observability and user selection to elucidate the generation of non-random missing scores. They proposed the tripartite collaborative filtering (TCF) framework to jointly model the three key aspects of score generation: item observability, user selection, and scoring. This framework aims to estimate non-random missing scores by employing the matrix decomposition method. Subsequently, the TCF framework is transformed into a tripartite probabilistic matrix factorization (TPMF) model, which effectively mitigates the limitations associated with estimating non-random missing scores. In addition, Chen et al. [14] incorporated non-random missing assumptions into the domain of social recommendation and presented the SPMF-MNAR algorithm. This algorithm captures the scoring data acquisition process by considering user preferences and social influence, thereby enhancing the performance of social recommendation algorithms. Saito et al. [15] also introduced a meta-learning approach to address the challenges associated with propensity estimation model selection and high variance. Their method is based on an asymmetric three-training model framework, wherein two trainers generate data with reliable scores and a separate predictor performs the final prediction. This approach significantly mitigates the limitations of the propensity score model.

The propensity score method is a reliable approach to mitigate selection bias, allowing for unbiased recommendations by assigning weights to each data point based on the reciprocal of the propensity score. Schnabel et al. [11] introduced an inverse propensity score estimation method for learning recommender systems in the presence of selection bias. They derived a matrix factorization approach that effectively eliminates selection bias, offering a conceptually simple and highly scalable solution. However, the performance of propensity score-based recommendation systems is often influenced by the choice of the propensity estimation model and the issue of high variance. To address the high variance problem associated with propensity estimation models, Saito et al. [16] proposed a dual-robust estimator for ground truth ranking. This estimator enhances the variance and error bounds of existing unbiased estimators, ensuring unbiased ranking measurements for ground truth. Similarly, Wang et al. [17] introduced a doubly robust estimator (DR) that combines estimation error and propensity in a doubly robust manner. This approach yields unbiased performance estimates and mitigates the impact of propensity variance. Furthermore, Guo et al. [18] conducted a study on the bias and variance of the double robust estimator. Building upon existing estimators, they proposed a more robust estimator called more robust doubly robust (MRDR). This enhanced estimator further reduces variance while maintaining dual robustness.

The aforementioned related studies primarily rely on estimating the propensity score using the marginal probability of a user’s rating for an item, which possesses certain limitations. To overcome these shortcomings and enhance the accuracy of propensity score estimation, this paper proposes a novel method that incorporates both user and item information in the estimation process. By considering these additional factors, the proposed method aims to provide a more comprehensive and robust estimation of the propensity score.

3. Methodology

The propensity score method is widely used to mitigate selection bias and eliminate its influence on recommender systems. Constructing an unbiased estimator is crucial for improving recommendation performance by effectively addressing selection bias. However, the effectiveness of the unbiased estimator relies on the accurate and efficient computation of the bias. The inverse propensity score estimator can still exhibit bias if the bias is incorrectly specified. Given the intricate nature of selection bias mechanisms in recommender systems, it is of paramount importance to develop an unbiased estimator with excellent performance.

The propensity score, as defined in the study [11], corresponds to the marginal probability of a user providing a rating for an item. The proposed estimator can be expressed as follows:

$$\begin{array}{c}{\widehat{R}}_{IPS}\left(\widehat{\mathit{R}}|P\right)=\frac{1}{\mathit{m}\mathit{n}}{\displaystyle {\displaystyle \sum }_{\left(\mathit{u},\mathit{i}\right):{\mathit{O}}_{\left(\mathit{u},\mathit{i}\right)}=\mathbf{1}}}\frac{{\delta }_{\left(u,i\right)}\left(\mathit{R},\widehat{\mathit{R}}\right)}{{\mathit{\theta }}_{\left(\mathit{u},\mathit{i}\right)}}\end{array}$$

(1)

where R represents the actual user-item rating matrix, $\widehat{\mathit{R}}$ represents the predicted user-item rating matrix, and ${\mathit{O}}_{\left(\mathit{u},\mathit{i}\right)}$ is a binary variable. Specifically, ${\mathit{O}}_{\left(\mathit{u},\mathit{i}\right)}=1$ indicates that the user u has provided a rating for the item i, while ${\mathit{O}}_{\left(\mathit{u},\mathit{i}\right)}=0$ indicates no interaction between user u and item i. The variable ${\mathit{\theta }}_{\left(\mathit{u},\mathit{i}\right)}=P\left({\mathit{O}}_{\left(\mathit{u},\mathit{i}\right)}=1\right)$ represents the marginal probability of ${\mathit{O}}_{\left(\mathit{u},\mathit{i}\right)}=1$, indicating the likelihood that item i is rated by user u. Additionally, ${\delta }_{\left(\mathit{u},\mathit{i}\right)}\left(\mathit{R},\widehat{\mathit{R}}\right)$ denotes the error between the predicted rating matrix and the actual rating matrix, which can be measured using either the square absolute error or the mean square error.

3.1. Propensity Score Estimation

This paper employs the inverse propensity score method to address selection bias. Building upon previous research [19], which presents a novel propensity score estimation technique for exposure bias in recommender systems by considering item popularity and user popularity preference, this study introduces the SPE propensity score estimation method specifically tailored for selection bias. Additionally, this paper explores user and item information to introduce two key concepts: user selection rate and item selection preference. By combining these concepts, the propensity score is estimated. The calculation process of the propensity score is outlined as follows:

$${\mathit{s}}_{\mathit{u}}=\frac{{{\displaystyle \sum }}_{i\in I}{\mathit{O}}_{\left(\mathit{u},\mathit{i}\right)}}{\underset{u\in U}{\max }{{\displaystyle \sum }}_{i\in I}{\mathit{O}}_{\left(\mathit{u},\mathit{i}\right)}}$$

(2)

$${\mathit{s}}_{\mathit{i}}=\frac{1}{\left|U_i\right|}{\displaystyle \sum }_{u\in U_i}{\mathit{s}}_{\mathit{u}}$$

(3)

$${\mathit{\theta }}_{\left(\mathit{u},\mathit{i}\right)}=\alpha -\left|{\mathit{s}}_{\mathit{u}}-{\mathit{s}}_{\mathit{i}}\right|$$

(4)

where ${\mathit{s}}_{\mathit{u}}$ represents the selection rate of the user u, indicating the user’s willingness to rate the item. A higher value of ${\mathit{s}}_{\mathit{u}}$ suggests a greater inclination of the user to provide a rating. Similarly, ${\mathit{s}}_{\mathit{i}}$ denotes the selected preference of the item i, reflecting the degree of preference associated with its selection. A higher value of ${\mathit{s}}_{\mathit{i}}$ indicates a stronger preference for the item’s selection. $U_i$ is the set of all users who rated the item i, and $\left|{\mathit{s}}_{\mathit{u}}-{\mathit{s}}_{\mathit{i}}\right|$ represents the degree of matching between the user’s selection rate and the item’s selected preference. A smaller value of $\left|{\mathit{s}}_{\mathit{u}}-{\mathit{s}}_{\mathit{i}}\right|$ signifies a higher degree of concordance, resulting in a larger propensity score ${\mathit{\theta }}_{\left(\mathit{u},\mathit{i}\right)}$. Consequently, during the recommendation process, the sample (u,i) with a higher propensity score is more likely to receive recommendations.

Notably, when $\left|{\mathit{s}}_{\mathit{u}}-{\mathit{s}}_{\mathit{i}}\right|=0$, indicating a perfect match, the propensity score ${\mathit{\theta }}_{\left(\mathit{u},\mathit{i}\right)}$ attains the maximum value $\alpha$. Here, $\alpha$ corresponds to a threshold parameter signifying the maximum degree of recommendation that the sample (u,i) can obtain in the recommendation system.

The calculation of the propensity score using Formula (4) may result in negative or zero values, which is logically infeasible. To address this issue, a propensity score threshold, denoted as D, is introduced during propensity score estimation: ${\mathit{\theta }}_{\left(\mathit{u},\mathit{i}\right)}=\max \left\{{\mathit{\theta }}_{\left(\mathit{u},\mathit{i}\right)},D\right\}$. This threshold restricts the minimum value of the propensity score to ensure its validity. Overall, this method can effectively reduce the variance of the model [20].

3.2. Traditional Recommendation Model

This section introduces the two traditional matrix factorization-based recommendation models used in this paper.

3.2.1. Latent Factor Model

The latent factor model is a collaborative filtering recommendation algorithm renowned for its high recommendation accuracy [21]. The fundamental concept behind the LFM model involves decomposing the user-item rating matrix into two low-dimensional matrices, which serve to capture the latent features of both users and items. By utilizing the product of these two low-dimensional matrices, the LFM model generates predictions for ratings.

Consider a scenario where there are m users, denoted as $\mathit{U}=\left\{{\mathit{u}}_{\mathbf{1}},{\mathit{u}}_{\mathbf{2}},{\mathit{u}}_{\mathbf{3}},\dots {\mathit{u}}_{\mathit{m}}\right\}$, and there are n items, denoted as $\mathit{I}=\left\{{\mathit{i}}_{\mathbf{1}},{\mathit{i}}_{\mathbf{2}},{\mathit{i}}_{\mathbf{3}},\dots {\mathit{i}}_{\mathit{n}}\right\}$. Here, the rating matrix, denoted as $\mathit{R}\in {\mathit{R}}^{\mathit{m}\times \mathit{n}}$, exhibits a high degree of sparsity with limited available data. The primary objective of the latent factor model is to predict ratings for users and items that have not yet interacted with each other based on the existing rating data.

The LFM is comprised of K latent factors, as illustrated in Figure 1. Consequently, for each user u, the vector ${\mathit{p}}_{\mathit{u}}=\left\{{\mathit{p}}_{\mathit{u}\mathbf{1}},{\mathit{p}}_{\mathit{u}\mathbf{2}},{\mathit{p}}_{\mathit{u}\mathbf{3}},\dots ,{\mathit{p}}_{\mathit{u}\mathit{K}}\right\}$ represents their preference degree for each latent factor, while vector ${\mathit{q}}_{\mathit{i}}=\left\{{\mathit{q}}_{\mathit{i}\mathbf{1}},{\mathit{q}}_{\mathit{i}\mathbf{2}},{\mathit{q}}_{\mathit{i}\mathbf{3}},\dots ,{\mathit{q}}_{\mathit{i}\mathit{K}}\right\}$ represents the preference degree of each item i for the latent factors. Thus, the user’s rating of the item can be predicted using the inner product of ${\mathit{p}}_{\mathit{u}}$ and ${\mathit{q}}_{\mathit{i}}$, as demonstrated in the following formula:

$${\widehat{\mathit{r}}}_{\mathit{u}\mathit{i}}={\mathit{p}}_{\mathit{u}}{\mathit{q}}_{\mathit{i}}$$

(5)

where ${\widehat{\mathit{r}}}_{\mathit{u}\mathit{i}}$ denotes the predicted rating of user u for item i. In the matrix form, the predictive score matrix $\widehat{\mathit{R}}$ can be decomposed into the product of two low-rank matrices, as shown in the following formula:

$$\widehat{\mathit{R}}=\mathit{P}\mathit{Q}$$

(6)

The solution method for LFM involves minimizing the mean square error. The calculation formula for the loss function is as follows:

$$L={\displaystyle \sum }_{\begin{array}{c}\mathit{i}\in \mathit{I}\\ \mathit{u}\in \mathit{U}\end{array}}{\left({\mathit{r}}_{\mathit{u}\mathit{i}}-{\widehat{\mathit{r}}}_{\mathit{u}\mathit{i}}\right)}^2+\lambda \left(\Vert {\mathit{p}}_{\mathit{u}}\Vert {}^2+\Vert {\mathit{q}}_{\mathit{i}}\Vert {}^2\right)$$

(7)

where $\lambda \left(\Vert {\mathit{p}}_{\mathit{u}}\Vert {}^2+\Vert {\mathit{q}}_{\mathit{i}}\Vert {}^2\right)$ represents a regularization term that is added to prevent overfitting in the model. $\lambda$ is an overfitting penalty factor, which is a hyperparameter used to balance the data loss term and the regularization term.

3.2.2. BiasSVD

BiasSVD is an enhanced matrix decomposition method that builds upon singular value decomposition (SVD) [22]. This approach addresses the issue of bias in the user’s ratings for items and establishes a prediction model, as illustrated by the following formula:

$${\widehat{\mathit{r}}}_{\mathit{u}\mathit{i}}={\mathit{p}}_{\mathit{u}}{\mathit{q}}_{\mathit{i}}+\mu +b_u+b_i$$

(8)

where $\mu$ represents the overall mean rating value across all users and items in the user-item rating matrix. $b_u$ denotes the bias term associated with the user u, indicating the rating preference of user u. $b_i$ represents the bias term associated with the item i, reflecting the degree to which item i is preferred by users based on the mean rating value received by the item.

The solution method of bias is also solved by minimizing the mean square error. The loss function calculation formula is as follows:

$$L={\displaystyle \sum }_{\begin{array}{c}\mathit{i}\in \mathit{I}\\ \mathit{u}\in \mathit{U}\end{array}}{\left({\mathit{r}}_{\mathit{u}\mathit{i}}-{\widehat{\mathit{r}}}_{\mathit{u}\mathit{i}}\right)}^2+\lambda \left({\mathit{b}}_{\mathit{u}}^{\mathbf{2}}+{\mathit{b}}_{\mathit{i}}^{\mathbf{2}}+\Vert {\mathit{p}}_{\mathit{u}}\Vert {}^2+\Vert {\mathit{q}}_{\mathit{i}}\Vert {}^2\right)$$

(9)

3.3. LFM-SPE

To assess the efficacy of the proposed SPE method, it is integrated with conventional LFM and BiasSVD models, resulting in the LFM-SPE and BiasSVD-SPE models, respectively. Subsequently, this section presents the procedure for combining the LFM-SPE model.

Utilizing the propensity score to mitigate bias in recommendation systems primarily involves employing inverse propensity weighting. This approach entails assigning weights to the rating data based on the reciprocal of the propensity score, thereby enabling unbiased estimation and mitigating the influence of selection bias within the dataset.

The propensity score is incorporated into the loss function of LFM using the inverse propensity weighting approach, as demonstrated by the following equation:

$$L_{SPE}={\displaystyle \sum }_{\begin{array}{c}\mathit{i}\in \mathit{I}\\ \mathit{u}\in \mathit{U}\end{array}}\frac{{\left({\mathit{r}}_{\mathit{u}\mathit{i}}-{\widehat{\mathit{r}}}_{\mathit{u}\mathit{i}}\right)}^2}{{\mathit{P}}_{\left(\mathit{u},\mathit{i}\right)}}+\lambda \left(\Vert {\mathit{p}}_{\mathit{u}}\Vert {}^2+\Vert {\mathit{q}}_{\mathit{i}}\Vert {}^2\right)$$

(10)

The loss function is solved using the stochastic gradient descent algorithm [23], which involves the following steps:

Step1. Deriving the parameters ${\mathit{p}}_{\mathit{u}}$ and ${\mathit{q}}_{\mathit{i}}$ to determine the direction with the steepest gradient descent.

Step2. Iteratively optimizing the parameters through calculations while continuously updating the direction of the steepest decline for each parameter until convergence.

Step3. Obtaining the user implicit semantic preference vector ${\mathit{p}}_{\mathit{u}}$ and the product implicit semantic attribute vector ${\mathit{q}}_{\mathit{i}}$, which effectively eliminates selection bias.

Step4. Combining the preference vectors of each user and the attribute vectors of each item to form the user latent semantic preference matrix $\mathit{P}$ and the item latent semantic attribute matrix $\mathit{Q}$.

By following this iterative process, the LFM-SPE model can effectively learn the latent factors that capture the underlying patterns and preferences in the data, resulting in accurate recommendations. The overall algorithm flow is shown in Algorithm 1.

Algorithm 1: LFM-SPE Algorithm

Input: user-item rating matrix $\mathit{R}$, number of latent factors K, recommended threshold $\mathit{\alpha }$,

propensity score threshold D, iterations T

1 Compute ${\mathit{O}}_{\left(\mathit{u},\mathit{i}\right)}$ based on whether the user has rated the item

2 Compute the propensity score ${\mathit{\theta }}_{\left(\mathit{u},\mathit{i}\right)}$

3 ${\mathit{\theta }}_{\left(\mathit{u},\mathit{i}\right)}=\max \left\{{\mathit{\theta }}_{\left(\mathit{u},\mathit{i}\right)},D\right\}$

4 Initialize $\mathit{P}$ and $\mathit{Q}$ randomly

5 for t = 1, …, T do

6 Compute the loss function

7 Compute the derivative of the parameter

8 Update $\mathit{P}$ and $\mathit{Q}$

9 end

10 Compute the predicted user-item rating matrix $\widehat{\mathit{R}}$

Output: the predicted user-item rating matrix $\widehat{\mathit{R}}$

The approach of incorporating the SPE method into the BiasSVD matrix factorization model is similar to the methodology employed in the LFM-SPE model, where the propensity score is weighted reciprocally in the model’s loss function. Hence, the training process of the BiasSVD-SPE model follows a similar framework and is not explicitly discussed in this section.

4. Experiment

This chapter presents the experiments conducted to evaluate the proposed methodology. Section 4.1 provides an overview of the datasets used in the experiments. Subsequently, Section 4.2 introduces the evaluation indicators employed for assessing the experiment outcomes. Section 4.3 presents the comparative model used in the experiments, and parameter analysis is performed in Section 4.4. Finally, Section 4.5 presents, analyzes, and discusses the experimental results.

4.1. Data Set

To assess the accuracy and effectiveness of the proposed SPE propensity score estimation method in enhancing recommendation performance, a series of experiments were conducted. These experiments utilized three publicly available datasets from MovieLens [24], namely Movielens 100K, Movielens latest, and Movielens 1M, which will be referred to as ML 100K, ML latest, and ML 1M, respectively, for brevity. For each record in the dataset, three features are utilized: user ID, movie ID, and the user’s rating of the movie. An overview of the key statistics for the three datasets is presented in

Table 1. Data set information. The table mainly shows the number of dataset users, items, ratings, and the dataset sparsity.

Data Set	User	Item	Rating	Sparsity
ML 100K	948	1682	100,000	93.7%
ML 1M	6040	3706	1,000,209	95.5%
ML latest	610	9742	100,836	98.3%

.

For each dataset, we randomly selected 80% as training data, 10% as validation data, and the remaining 10% as test data.

4.2. Evaluation Indicators

The prediction accuracy of the model was evaluated using two regression model evaluation metrics: root mean square error and mean absolute error. These metrics provide insights into the performance and accuracy of the predictions.

$$RMSE=\sqrt{\frac{1}{\left|T\right|}{\displaystyle \sum }_{\left(u,i\right)\in T}{\left({\widehat{\mathit{r}}}_{\mathit{u}\mathit{i}}-{\mathit{r}}_{\mathit{u}\mathit{i}}\right)}^2}$$

(11)

$$MAE=\frac{1}{\left|T\right|}{\displaystyle \sum }_{\left(u,i\right)\in T}\left|{\widehat{\mathit{r}}}_{\mathit{u}\mathit{i}}-{\mathit{r}}_{\mathit{u}\mathit{i}}\right|$$

(12)

where $T$ represents the test set, ${\widehat{\mathit{r}}}_{\mathit{u}\mathit{i}}$ denotes the score predicted by the model, and ${\mathit{r}}_{\mathit{u}\mathit{i}}$ represents the actual score from the test set. The evaluation indicators RMSE and MAE are employed to assess the prediction performance, where smaller values indicate better prediction results.

4.3. Comparison Model

To assess the effectiveness of the SPE propensity score estimation method in mitigating selection bias, we compared it with two baseline models, namely LFM and BiasSVD. Additionally, we included three existing propensity score estimation methods, namely IPS [11], DR [17], and MRDR [18], for comparison. By combining these methods with the LFM model, we constructed three additional comparison models.

LFM-IPS: IPS is a framework for eliminating selection bias, which uses the marginal probability of the user’s item selection as the estimation method of the propensity score and combines the framework and LFM model.

LFM-DR: DR is a dual-robust estimator that integrates the estimation error and the propensity in a doubly robust manner to obtain unbiased performance estimates and mitigates the effect of propensity variance, combining this estimator with an LFM model.

LFM-MRDR: MRDR is a more robust estimator, which further reduces the variance based on the dual robust estimator, combining this estimator with the LFM model.

Considering that the models utilized in this experiment are all founded on matrix decomposition, their time complexity is denoted as O(tmnK). These algorithms exhibit no substantial disparity in terms of time complexity; the variance lies primarily in the number of iterations, while their time consumption remains within the same order of magnitude.

4.4. Parameter Analysis

In the proposed LFM-SPE model presented in this paper, two key parameters significantly impact the recommendation effectiveness of the model: the number of latent factors, $K$, and the recommendation threshold parameter, $\mathit{\alpha }$. In this section, we analyze the impact and variation trends of these two parameters on the recommendation performance individually. Throughout the analysis, while one parameter is examined, the remaining parameters are maintained at their optimal settings.

We set the number of latent factors $K$ to $K=\left\{10,20,30,\dots ,70\right\}$, and observed the impact of its changes on the model RMSE and MAE, as shown in Figure 2.

The number of latent factors $K$ reflects the number of potential association features between users and items. The optimal result obtained by the model depends on whether the appropriate number of hidden factors is selected. As shown in Figure 2, as the value of $K$ increases from 10 to 70, the RMSE and MAE indicators show an overall trend of decreasing first and then increasing. This observation demonstrates that as the value of K increases up to a certain threshold, the algorithm’s performance reaches its optimum level. Beyond this threshold, a further increase in the number of hidden factors does not enhance the recommendation effectiveness; instead, it may result in a decline in model performance. This phenomenon can be attributed to overfitting, wherein an excessive number of parameters contribute to suboptimal generalization.

Moreover, we set the recommended threshold parameter $\mathit{\alpha }$ to $\mathit{\alpha }=\left\{0.5, 0.6,0.7,\dots ,1.0\right\}$, and observed the impact of its changes on the model RMSE and MAE, as shown in Figure 3.

The recommendation threshold parameter $\mathit{\alpha }$ represents the maximum value that the propensity score can reach, and $\mathit{\alpha }=1$ means that 100% of the items appear in the user’s recommendation, which is impossible to observe in the real world. As depicted in Figure 3, when $\mathit{\alpha }$ is set to one, both the RMSE and MAE values are significantly higher, indicating a substantial decline in recommendation performance. It is evident that setting the recommendation threshold parameter $\mathit{\alpha }$ too high has a detrimental impact on the model’s effectiveness. When $\mathit{\alpha }$ is varied from 0.5 to 1, the RMSE and MAE indicators exhibit a decreasing trend initially, followed by an increasing trend. This implies that there exists an optimal value for α that maximizes the recommendation effect. Specifically, when $\mathit{\alpha }$ is set to 0.8, the model achieves the best recommendation performance.

Based on the experimental results presented above, it can be concluded that the optimal parameter setting for the recommendation system is a recommended threshold parameter $\mathit{\alpha }=0.8$ and a number of hidden factors $K=20$.

4.5. Experimental Results and Discussion

The score prediction results of the proposed LFM-SPE model and BiasSVD-SPE model, along with other comparative models, are presented in

Table 2. Comparison of RMSE experimental results of each model.

Data Set	BiasSVD	LFM	LFM-IPS	LFM-DR	LFM-MRDR	BiasSVD-SPE	LFM-SPE
ML 100K	0.826	0.810	0.805	0.801	0.797	0.803	0.795
ML 1M	0.771	0.749	0.745	0.738	0.733	0.746	0.736
ML latest	0.991	0.988	0.980	0.974	0.971	0.973	0.965
Average	0.862	0.849	0.843	0.838	0.834	0.841	0.832

and

Table 3. Comparison of MAE experimental results of each model.

Data Set	BiasSVD	LFM	LFM-IPS	LFM-DR	LFM-MRDR	BiasSVD-SPE	LFM-SPE
ML 100K	0.702	0.683	0.675	0.671	0.665	0.679	0.661
ML 1M	0.678	0.659	0.652	0.641	0.635	0.654	0.638
ML latest	0.901	0.881	0.873	0.865	0.860	0.877	0.857
Average	0.760	0.741	0.733	0.726	0.720	0.737	0.719

. By analyzing the experimental results, the following conclusions can be drawn.

The BiasSVD-SPE model outperforms the BiasSVD matrix factorization algorithm, with an improvement of 2.44% and 3.03% in RMSE and MAE indicators, respectively. Unlike the traditional BiasSVD algorithm, which simply incorporates bias items to account for user and item deviations in score prediction, the BiasSVD-SPE model addresses selection bias by calculating propensity scores for each sample group. It effectively handles selection bias through inverse propensity score weighting, considering variations across different data groups. Consequently, the BiasSVD-SPE model exhibits a more substantial performance enhancement compared to the traditional BiasSVD algorithm as it comprehensively addresses data deviations and offers increased interpretability.

The LFM-SPE model proposed in this paper demonstrates superior recommendation performance compared to other models. When compared to the traditional LFM model, the LFM-SPE model exhibits improvements of 2.00% and 2.97% in the RMSE and MAE evaluation indicators, respectively. Additionally, the recommendation performance of the LFM-IPS, LFM-DR, and LFM-MRDR models, which incorporate propensity scores, also shows enhancement. These findings highlight the significant impact of incorporating propensity scores to mitigate selection bias, leading to substantial improvements in the model’s recommendation performance.

Among the four LFM models with added propensity scores, the LFM-SPE model exhibited the highest recommendation performance. Specifically, the LFM-SPE model outperformed the LFM-IPS model, resulting in improvements of 1.30% and 1.91% in the RMSE and MAE evaluation indicators, respectively. This suggests that the SPE method is superior to IPS in eliminating selection bias. While the IPS algorithm estimates the propensity score based on the marginal probability of user-item selection, the proposed method in this paper simultaneously analyzes and leverages the user and item information, incorporating both the user’s selection rate and the item’s selection preference to estimate the propensity score. As a result, the propensity score estimate is more accurate, leading to improved recommendation performance.

Furthermore, the LFM-SPE model demonstrated slightly better average recommendation performance across the three datasets compared to LFM-MRDR. This indicates that the LFM-MRDR model benefits from the effective reduction of variance through the MRDR double robust estimation method, which in turn mitigates the influence of bias on the recommendation algorithm. It is worth noting that the recommendation effect of LFM-MRDR surpassed that of LFM-DR, further highlighting the efficacy of the MRDR approach.

Overall, the experimental results demonstrate that the LFM-SPE model, with its combined analysis of user and item information and accurate propensity score estimation, outperforms other LFM models with propensity scores in terms of recommendation performance.

5. Conclusions

This paper addresses the issue of selection bias in recommendation systems by introducing the concept of inverse propensity score weighting and proposing a propensity score estimation method called SPE. The SPE method takes into account both user and item information and combines it with traditional matrix factorization-based algorithms, such as LFM and BiasSVD. Experimental evaluations were conducted on the publicly available MovieLens dataset, demonstrating that the SPE method effectively eliminates selection bias and improves recommendation performance. Furthermore, the proposed SPE method outperformed existing propensity score estimation methods.

While the propensity score estimation method presented in this paper has shown improvements in recommendation performance, there are still areas for further enhancement in future research. Firstly, the estimation of propensity scores may introduce high variance issues, and addressing this variance can lead to further performance improvements. Secondly, the methods employed in this paper primarily focus on explicit feedback data, but incorporating implicit feedback data generated by users and items could be explored for bias elimination research.

Overall, the proposed SPE method offers a promising approach to mitigating selection bias in recommendation systems, and future investigations can explore strategies to enhance its performance.