Implicit Stochastic Gradient Descent Method for Cross-Domain Recommendation System

Abstract

The previous recommendation system applied the matrix factorization collaborative filtering (MFCF) technique to only single domains. Due to data sparsity, this approach has a limitation in overcoming the cold-start problem. Thus, in this study, we focus on discovering latent features from domains to understand the relationships between domains (called domain coherence). This approach uses potential knowledge of the source domain to improve the quality of the target domain recommendation. In this paper, we consider applying MFCF to multiple domains. Mainly, by adopting the implicit stochastic gradient descent algorithm to optimize the objective function for prediction, multiple matrices from different domains are consolidated inside the cross-domain recommendation system (CDRS). Additionally, we design a conceptual framework for CDRS, which applies to different industrial scenarios for recommenders across domains. Moreover, an experiment is devised to validate the proposed method. By using a real-world dataset gathered from Amazon Food and MovieLens, experimental results show that the proposed method improves 15.2% and 19.7% in terms of computation time and MSE over other methods on a utility matrix. Notably, a much lower convergence value of the loss function has been obtained from the experiment. Furthermore, a critical analysis of the obtained results shows that there is a dynamic balance between prediction accuracy and computational complexity.

1. Introduction

Recent achievements in the Internet and computing technologies have made it possible for organizations to collect, store, and process large amounts of data. These data contain detailed information related to the behaviors of users. Accurately, they represent the set of user evaluations for specific items. For example, Amazon (https://www.amazon.com/) collects information about the user’s habits shopping-wise, or even regarding surfing on their website. Netflix (https://www.netflix.com/) also has substantial data related to the subject of movies. These data are beneficial for recommending useful decisions when supporting their clients. In this scenario, each firm designs a unique and maximally efficient system that can recommend as pleasant as possible items to its customers [1]. Nevertheless, not all users give ratings for items that they like or dislike. This limitation causes the fragmentation of the dataset obtained from the user, which is called data sparsity. In the real-life, a dataset is sparse at around 0.05% [2]. Therefore, the system can not produce useful recommendations when a new user or item has entered the system due to the insufficient previous ratings. This problem is named the cold-start [3], which is the most challenging issue for researchers to overcome. The cold-start problem is that problem wherein a system is not able to recommend items to users. Every recommender system is required to build a user’s profile by considering his/her preferences and likes. The user’s profile is developed by considering his/her activities and behaviors being perform with the system. Based on user’s previous history and activities, the system makes decisions and recommends items consequently. Many investigations have been proposed to solve the cold-start problem by locating extra information among the intradomain objects to imply the association between a user and item [4,5,6,7]. Nevertheless, we cannot always obtain this kind of extra useful information.

On the other hand, the cold-start problem in insufficient data in one domain can be solved if another domain has relatively abundant data [8]. In other words, since there exists either implicitly or explicitly correlated between domains, we could overcome the cold-start problem by grouping multiple domains. In particular, the latent features existing among domains may improve recommendation accuracy. By this approach, a recommendation system can be built to exploit valuable information from one domain to contribute to another domain. These systems are known as cross-domain recommendation systems (CDRSs) [9]. In CDRSs, one of the most popular and efficient methods that has been used is matrix factorization collaborative filtering (MFCF) [10]. This method could handle both two major problems regarding two directions of CDRS development. The first direction focuses on collecting preference data from users and items from all domains. Oppositely, the CDRS in the second direction aims to connect domains based on other information, such as the properties of items or the social relations of users [11]. The preference data are exclusively focused on our research, since they is not affected by other information yet can be applied widely. Zhang et al. (2018) classified the preference-based CDRS into two groups: the first one is the situation in which there are no common areas between domains, while the other group has at least a partial overlap between domains [12]. In this paper, we concentrate on the first class of CDRS, where there is not any overlap among domains, since this situation is prevalent in real life. Regarding this type of CDRS, the existing method has used shared information from the items and users in domains [13]. Notably, similar information related to the item’s contents and user’s preferences was extracted from all domains to build the group-level knowledge, which is used for the utility matrices afterward [14]. Nevertheless, there are some limitations to this approach, since it is unsteady to transfer knowledge from one domain to the other. This unstable state will adversely affect the performance of the recommendation system [15].

Differently from previous works, to overcome the limitations mentioned above, we propose an efficient framework for a cross-domain recommendation system. In this framework, multiple domains that are presented by matrices are consolidated into one. Then we apply the MFCF to predict the unknown ratings from user to item. In this way, it is possible to extract the latent features from the user-group and item-group. An implicit update technique is adopted while optimizing the objective function to increase prediction accuracy. Additionally, the optimization convergence is significantly improved.

The main contributions of this study are as follows:

• We propose an efficient framework for a cross-domain recommendation system based on a constrained optimization model. In our model, the optimal solution and computation time are simultaneously taken into consideration.
• We devise an approximation algorithm that is suitable for objective function optimization in a cross-domain related problem. In particular, an implicit updating technique is applied to improve convergence time.
• We conduct extensive experiments on two real-world datasets to validate the effectiveness and efficiency of our method. The results demonstrate that the proposed framework can achieve better performance in comparison with the previous approach.

The remainder of the paper is organized as follows. Section 2 explains the background knowledge of MFCF in a single domain and reviews literature related to CRDS. Section 3 formally defines the problem formulation. In Section 4, we present our conceptual framework for CDRS. Section 5 presents an experiment. Finally, we draw conclusions and suggest directions for future study in Section 6.

2. Related Work and Background

2.1. Related Work

Recent researchers have studied cross-domain related work, as mentioned in [11,16,17], wherein there are two types of cross-domain recommended tasks. The first task is to use the information of the source domain to enhance the quality of the target domain recommendation [18,19,20]. Karatzoglou et al. used a machine learning method to transfer dense knowledge from the source domain to the target areas, which is much more sparse [21]. Enrich et al. used the user tags as connections between multiple domains, from which they learn the users’ rating models to gain performance in the target domain [22]. The second task is recommending items in separate domains concurrently. They proposed a method for creating a rating matrix, which is the multidisciplinary shared latent factor [23,24]. Shi et al. [25] used the user-generated tags to calculate the similarity between cross-domain users and items, respectively, and then integrated these similarities into a matrix factorization model to improve the recommended accuracy. Gao et al. presented the clustering latent factor model based on a joint non-negative matrix framework [26].

For recommendation using matrix factorization, work was done by Gogna et al. [27]. They proposed a matrix completion framework that can be implemented in different domains. Zhenzhen et al. presented a cross-domain recommendation algorithm to overcome cold-start and sparsity problems and mentioned that this could be extended to consider temporal dynamics, as user preferences may change over time [28]. A cross-domain collaborative framework for recommending the venue proposed by Farseev et al. [29] is not able to address the cold-start problem. Loni et al. [30] presented a cross-domain factorization machine that can exploit additional knowledge from an auxiliary domain by encoding specific knowledge from a domain in terms of the real-valued feature vector.

In this study, we apply the MFCF for multiple domains using an updated technique to increase the convergence time of the objective function. Additionally, the implicit stochastic gradient descent-based algorithm is utilized to apply to the cross-domain recommendation system.

2.2. Background

In a single domain, let us suppose there are $\mathit{M}$ users and $\mathit{N}$ items. The relationship between the users and the items is presented by the user-item rating matrix $\mathbf{Y}\in {\mathbb{R}}^{M\times N}$, called $\mathit{utility}\mathit{matrix}$. Any rating $r_{ij}$ in $\mathbf{Y}$ is subject to $r_{ij}\in \{1,2,3,4,5,?\}$, where “?” represents missing value. To predict the missing values, users and items are clustered. The utility matrix $\mathbf{Y}$ can be factorized into two matrices $\mathbf{Y}\approx \widehat{\mathbf{Y}}=\mathbf{X}{\mathbf{W}}^T$, where $\mathbf{X}\in {\mathbb{R}}^{M\times K}$ is the user-group membership matrix, and $\mathbf{W}\in {\mathbb{R}}^{N\times K}$ is the item-group membership matrix.

Figure 1 represents the matrix factorization, in which the full utility matrix $\mathbf{Y}$ is decomposed into two matrices $\mathbf{X}$ and $\mathbf{W}$, where K is much smaller than M, N. Each row in $\mathbf{X}$ represents a user profile$\mathbf{x}$, and each column in $\mathbf{W}$ denotes an item profile$\mathbf{w}$. On the other hand, the i-th item and the j-th user are represented by the i-th and j-th rows of the two matrices as ${\mathbf{W}}_{i*}$ and ${\mathbf{X}}_{j*}$. After matrix factorization, the users and items are mapped to a latent factor feature of a lower dimensionality K.

To predict the missing values in the utility matrix, the low-rank matrix factorization is approximated as an optimization problem given by

$$\underset{\mathbf{X},\mathbf{W}}{\min }\mathcal{L}(f(\mathbf{X},\mathbf{W}),\mathbf{Y})+\lambda \mathcal{R}(\mathbf{X},\mathbf{W}),$$

(1)

where $\mathcal{L}$ is the loss function of the predicted ratings $f(\mathbf{X},\mathbf{W})$ and the original ratings $\mathbf{Y}$, $\mathcal{R}(\mathbf{X},\mathbf{W})$ is the regularization term, and $\lambda$ is the regularization tradeoff parameter. $\lambda \mathcal{R}(\mathbf{X},\mathbf{W})$ is regularization component to avoid overfitting. Regarding probabilistic matrix factorization (BMF) [31,32],  the objective function to measure the loss with regularization terms and a Frobenius norm is expressed as

$$J(\mathbf{X},\mathbf{W})=\frac{1}{2}\parallel \mathbf{I}\odot (\mathbf{Y}-\mathbf{X}{\mathbf{W}}^T){\parallel }_F+\frac{\lambda }{2}{\parallel \mathbf{X}\parallel }_F+\frac{\lambda }{2}{\parallel \mathbf{W}\parallel }_F,$$

(2)

where $\mathbf{I}$ is the rating indicator matrix, $I_{ij}\in \{0,1\}$. $I_{ij}=1$ indicates that the rating is observed, or $I_{ij}=0$ otherwise. ⊙ denotes the Hadamard product [33] of the matrices.

3. Problem Formulation

3.1. Definition of User-Preference Matrix

Let ${\mathbf{D}}_l$ with $l\in (1,L)$ be the user-preference matrix with response to l-th domain. Then, the entries of ${\mathbf{D}}_l$ which are denoted by ${\left(D_l\right)}_{ij}$ indicate the ratings of the i-th user for the j-th items of set ${\mathcal{D}}_l$.

By $\mathcal{U}$ we denote the set of all users that exist in multiple domains:

$$\mathcal{U}=\{{\mathcal{U}}^{D_1},{\mathcal{U}}^{D_2},\cdots ,{\mathcal{U}}^{D_L}\},$$

(3)

where ${\mathcal{U}}^{D_l}$ is the sets of users in l-th domain. Although these matrices are overlapping or nonoverlapping, the matrix $\mathbf{V}$, which is built from the consolidation of matrices ${\mathbf{D}}_1,$${\mathbf{D}}_2$, $...$${\mathbf{D}}_L$ has the number of rows given by $\left|\mathcal{U}\right|$. Given a user ${\mathrm{U}}_u,u\in \{1,2,\cdots ,|\mathcal{U}\left|\right\}$ in $\mathcal{U}$, the matrix ${\mathbf{D}}_l$ can be rewritten as follows:

$${\mathbf{D}}_l={[{\left({\mathbf{d}}_1^{D_l}\right)}^T,\cdots ,{\left({\mathbf{d}}_u^{D_l}\right)}^T,\cdots ,{\left({\mathbf{d}}_{\left|\mathbf{U}\right|}^{D_l}\right)}^T]}^T,$$

(4)

where the row vector ${\mathbf{d}}_u^X,X\in \{D_1,D_2,D_L\}$, contains the corresponding rating values of all items in $\mathcal{X}\in \{{\mathcal{D}}_1,{\mathcal{D}}_2,{\mathcal{D}}_L\}$, voted by user ${\mathrm{U}}_u$. Clearly, ${\mathbf{d}}_u^X=\mathbf{0}$, if ${\mathrm{U}}_u\in \mathcal{U}\backslash {\mathcal{U}}^X$.

For generality, all the user’s ratings can be described by the following matrix $\mathbf{V}$:

$$\mathbf{V}=[{\mathbf{D}}_1{\mathbf{D}}_2\cdots {\mathbf{D}}_L].$$

(5)

Matrix $\mathbf{V}$ is the expandable matrix since its dimensionality increases when adding new items and users to the data. We denote the transpose matrix of $\mathbf{V}$ by ${\mathbf{V}}^T$.

A column vector $\mathbf{b}$ is given as

$$\begin{array}{cc}\hfill \mathbf{b}& =\left[\begin{array}{c}{\mathrm{b}}_1\\ {\mathrm{b}}_2\\ ⋮\\ {\mathrm{b}}_n\end{array}\right],\hfill \end{array}$$

(6)

where n denotes the number of rows (the number of users) of $\mathbf{V}$. Each entry ${\mathrm{b}}_i$ is the inverse of a square root of the element ${\mathrm{a}}_{ii}$ in the $\mathbf{V}{\mathbf{V}}^T$ diagonal. Therefore, ${\mathrm{b}}_i$ is as follows:

$${\mathrm{b}}_i=\frac{1}{\sqrt{a_{ii}}}.$$

(7)

Then it is possible to write formula (6) in the following form

$$\begin{array}{cc}\hfill \mathbf{b}& =\left[\begin{array}{c}\frac{1}{\sqrt{a_{11}}}\\ \frac{1}{\sqrt{a_{22}}}\\ ⋮\\ \frac{1}{\sqrt{a_{nn}}}\end{array}\right].\hfill \end{array}$$

(8)

By ${\mathbf{b}}^T$ we denote the transform matrix of $\mathbf{b}$, and matrix $\mathbf{B}=\mathbf{b}{\mathbf{b}}^T$. In this regard, a similarity matrix $\mathbf{S}$ can be written as follows:

$$\mathbf{S}=\left(\mathbf{V}{\mathbf{V}}^T\right)\odot \mathbf{B}.$$

(9)

This operator is a Hadamard product, in which each element p, q in $\mathbf{S}$ is the product of elements p and q of the original two matrices $\left(\mathbf{V}{\mathbf{V}}^T\right)$ and $\mathbf{B}$. After this operator, $\mathbf{S}$ will be the symmetric matrix with rows and columns being users, in which each element $S_{ij}$ is the cosine of the angle between two vectors ${\mathbf{u}}_i$ and ${\mathbf{u}}_j$, where ${\mathbf{u}}_i$ and ${\mathbf{u}}_j$ are the i-th and j-th rows of $\mathbf{V}$, respectively.

Remark 1.

By considering the preference vectors ${\mathbf{u}}_i$ and ${\mathbf{u}}_j$, $i\ne j$, the similarity between i-th and j-th users is properly given by $S_{ij}$; i.e.,

$$S_{ij}=cos({\mathbf{u}}_i,{\mathbf{u}}_j)=\frac{{\mathbf{u}}_i{\mathbf{u}}_j^T}{\left|\right|{\mathbf{u}}_i{\left|\right|}_2·\left|\right|{\mathbf{u}}_j{\left|\right|}_2}.$$

(10)

Given by the i-th row and j-th column entry of matrix, $\mathbf{V}{\mathbf{V}}^T$ is equal to ${\mathbf{u}}_i{\mathbf{u}}_j^T$, with the corresponding entry $B_{ij}$ of matrix $\mathbf{B}$ equivalent to

$$B_{ij}=\frac{1}{\left|\right|{\mathbf{u}}_i{\left|\right|}_2·\left|\right|{\mathbf{u}}_j{\left|\right|}_2}.$$

(11)

Therefore, (9) is considered as a generalized formulation to derive the similarity matrix among users with respect to all items.

Similarly, we can find the item-similarity matrix as follows.

$$\mathbf{K}=\left({\mathbf{V}}^T\mathbf{V}\right)\odot \mathbf{C},$$

(12)

where $\mathbf{C}=\mathbf{c}{\mathbf{c}}^T$, $\mathbf{c}$ is a row vector $\mathbf{c}$ = $[c_1,c_2\cdots ,c_m]$ with ${\mathrm{c}}_m$ denoting the inverse of a square root of the element ${\mathrm{d}}_{mm}$ in the ${\mathbf{V}}^T\mathbf{V}$ diagonal of $\mathbf{V}$. Therefore, ${\mathrm{c}}_m$ is as follows:

$${\mathrm{c}}_m=\frac{1}{\sqrt{d_{mm}}}.$$

(13)

Now we will factorize matrix $\mathbf{V}$. As mentioned in the previous Section, the user n gives a rating to the item m that can be approximated as $y_{mn}={\mathbf{x}}_m^T{\mathbf{w}}_n$. However, the actual ratings have biases for users or/and items, since users tend to rate the items according to their rating behaviors, resulting in ratings that may be larger or smaller than the actual values the items receive. We use $bias$ to overcome this problem. By ${\mu }_m$ and ${\mu }_n$, we denote $biases$ for $item$ $\mathit{m}$ and $user$ $\mathit{n}$, respectively. Then the rating is approximated by

$$y_{mn}\approx {\mathbf{x}}_m{\mathbf{w}}_n+{\mu }_m+{\mu }_n+\mu ,$$

(14)

where $\mu$ is median value of all ratings.

Therefore, the loss function (2) can be written as

$$\begin{array}{c}\hfill \mathcal{L}(\mathbf{X},\mathbf{W},{\mathit{\mu }}_m,{\mathit{\mu }}_n)=\frac{1}{2s}\sum _{n=1}^N\sum _{m=1}^M{({\mathbf{x}}_m{\mathbf{w}}_n+{\mu }_m+{\mu }_n+\mu -y_{mn})}^2+\\ \hfill +\frac{\lambda }{2}{\left(\right|\left|\mathbf{X}\right||}_F^2+{\left|\right|\mathbf{W}\left|\right|}_F^2+\left|\right|{\mu }_{\mathbf{m}}{\left|\right|}_F^2+\left|\right|{\mu }_{\mathbf{n}}{\left|\right|}_F^2).\end{array}$$

(15)

In the previous works, this loss function is solved by optimizing one of the pairs ($\mathbf{X},{\mu }_m$) and ($\mathbf{W},{\mu }_n$) respectively, while fixing the other pair. This process is repeated until the loss function converges. This push–pull [34] gradient method will get a sub-optimal solution [35]. In contrast with the earlier investigations, we will solve this loss function by optimizing ($\mathbf{X},\mathbf{W},{\mu }_m,{\mu }_n$) simultaneously.

3.2. Algorithm for Prediction Error Minimization

In this section, we investigate the following joint design problem for prediction error model minimization:

$$\underset{\mathbf{X},\mathbf{W},{\mathit{\mu }}_m,{\mathit{\mu }}_n,\mathbf{t}}{\mathrm{minimize}}\mathcal{L}=\frac{1}{2s}\sum _{n=1}^N\sum _{m=1}^M{t}_{mn}^2+\frac{\lambda }{2}{\left(\right|\left|\mathbf{X}\right||}_F^2+{\left|\right|\mathbf{W}\left|\right|}_F^2+\left|\right|{\mathit{\mu }}_{\mathbf{m}}{\left|\right|}_F^2+\left|\right|{\mathit{\mu }}_{\mathbf{n}}{\left|\right|}_F^2),$$

(16)

where $\mathbf{t}\triangleq \left[t_{mn}\right],\forall m\in \{1,\cdots ,M\},\forall n\in \{1,\cdots ,N\}$, with $t_{mn}$ satisfying the following constraint:

$${\mathbf{x}}_m{\mathbf{w}}_n+{\mu }_m+{\mu }_n+\mu -y_{mn}\le t_{mn}.$$

(17)

Although the objective function in (16) is a quadratic representative, which is convex, constraint (17) is still non-convex. To efficiently solve this problem, we derive a successive convex program based on an inner approximation method [36] as follows:

It is observed that (17) is equivalent to the convex constraint:

$$\sum _{k=1}^K{u}_{mnk}^2\le t_{mn}-{\mu }_m-{\mu }_n-\mu +y_{mn}$$

(18)

with the following constraint imposed

$$x_{mk}{w}_{nk}\le u_{mnk}^2.$$

(19)

However, constraint (19) is still non-convex. Inspired from ([37], Lemma 1), (19) can be approximated as

$$\frac{{\overline{w}}_{nk}}{2{\overline{x}}_{mk}}{x}_{mk}^2+\frac{{\overline{x}}_{mk}}{2{\overline{w}}_{nk}}{w}_{nk}^2\le u_{mnk}^2,$$

(20)

which is convex as a second order cone constraint. Here, ${\overline{x}}_{mk}$ and ${\overline{w}}_{nk}$ are respectively the values of $x_{mk}$ and $w_{nk}$ at the previous iteration. Therefore, the successive convex program is formulated as

$$\begin{array}{cccc}\hfill & \underset{\mathbf{X},\mathbf{W},{\mathit{\mu }}_m,{\mathit{\mu }}_n,\mathbf{t},\mathbf{u}}{\mathrm{minimize}}\hfill & \hfill & \mathcal{L}=\frac{1}{2s}\sum _{n=1}^N\sum _{m=1}^M{t}_{mn}^2+\frac{\lambda }{2}{\left(\right|\left|\mathbf{X}\right||}_F^2+{\left|\right|\mathbf{W}\left|\right|}_F^2+\left|\right|{\mathit{\mu }}_{\mathit{m}}{\left|\right|}_F^2+\left|\right|{\mathit{\mu }}_{\mathit{n}}{\left|\right|}_F^2)\hfill \end{array}$$

(21a)

$$\begin{array}{cccc}\hfill & \mathrm{subject}\mathrm{to}\hfill & \hfill & \left(18\right),\left(20\right).\hfill \end{array}$$

(21b)

It is realized that the problems in (21) can be efficiently solved per iteration by the existing solver (e.g., SPDT3 [38], MOSEK [39], or SeDuMi [40]), so that we obtain at least a locally optimal solution at the convergence. The algorithm for solving problem in (21) is briefly described in Algorithm 1.

Algorithm 1: Iterative algorithm for the prediction error optimization.

1 Initialization: Set ${\mathcal{L}}^{min}$ := +∞, $({\mathbf{x}}^{*},{\mathbf{t}}^{*},{\mathit{w}}^{*},{\mathit{u}}^{*})$ := $\mathbf{0}$

2 for each k $\in \mathcal{K}$ $\mathbf{do}$ {solving subproblem (19)}

3  Generating an initial points: Set k:= 0 and solve (20) to generate (${\mathbf{x}}^{\left(0\right)},{\mathit{\mu }}_m^{\left(0\right)},{\mathbf{w}}^{\left(0\right)},{\mathit{\mu }}_n^{\left(0\right)}$).

4  repeat

5   Solve (21) to obtain (${\mathbf{x}}^{*},{\mathit{\mu }}_m^{*},{\mathbf{w}}^{*},{\mathit{\mu }}_n^{*}$) and ${\mathcal{L}}^{(k+1)}$.

6   Update (${\mathbf{x}}^{(k+1)},{\mathit{\mu }}_m^{(k+1)},{\mathbf{w}}^{(k+1)},{\mathit{\mu }}_n^{(k+1)}$):=(${\mathbf{x}}^{*},{\mathit{\mu }}_m^{*},{\mathbf{w}}^{*},{\mathit{\mu }}_n^{*}$).

7   Set $k=k+1$.

8  until Convergence

9  if ${\mathcal{L}}^{\left(k\right)}<{\mathcal{L}}^{min}$ $\mathbf{then}$

10   Update ${\mathcal{L}}^{min}$ := ${\mathcal{L}}^{\left(k\right)}$ and (${\mathbf{x}}^{*},{\mathit{\mu }}_m^{*},{\mathbf{w}}^{*},{\mathit{\mu }}_n^{*}$):=(${\mathbf{x}}^{\left(k\right)},{\mathit{\mu }}_m^{\left(k\right)},{\mathbf{w}}^{\left(k\right)},{\mathit{\mu }}_n^{\left(k\right)}$).

11  end if

12 end for

In Algorithm 1, we use the implicit update technique to increase the convergence speed. The initial values of $\mathbf{x}$ and $\mathbf{w}$ are random. For the practical implementation, Algorithm 1 terminates upon reaching ${\mathcal{L}}^{(k+1)}-{\mathcal{L}}^{\left(k\right)}<\epsilon$ after a finite number of iterations [41].

4. CDRS Framework

In this section, we propose a conceptual framework for a cross-domain recommender system that applies the proposed method [42]. When businesses launch multiple products or services, a mass number of data are processed to make the recommendations to clients. These data are heterogeneous and imbalanced, since their sources are from different domains. Data from users, such as ratings, number of likes, and website surfing history, are collected, clustered, and stored into the database. The cross-domain recommendation system engine will process these data to build the model. A set of parameters could be adjusted at this stage to obtain the best accuracy. The system output is the user-preferences prediction that is used to recommend items to the customers.

Particularly, according to the Figure 2, multiple datasets from various domains are preprocessed in a knowledge transfer module. Here, these data will have similarities identified and latent features extracted, and we will perform knowledge transformation. Then in the next phase, the prediction model will analyze all the information exported from the preprocessing phase in order to apply the appropriate algorithms for training and prediction generation. In this phase, most parameters are turned repeatedly to choose the best set for maximizing the whole system’s accuracy. By this workflow, a CDRS can deal with heterogeneous input data and produce recommendation items in various scenarios.

5. Experiments

In this section, we report experiments done to evaluate the recommendation quality of the proposed recommendation model against some baseline state-of-the-art recommendation techniques.

5.1. Dataset

To better illustrate our method, this section outlines a small-scale example. There were two datasets used: Movielens (https://movielens.org/) and Amazon Food (https://www.kaggle.com/snap/amazon-fine-food-reviews). The statistical information for these datasets is presented in

Table 1. Statistics of datasets.

	Movielens100k	Amazon Food
#user	943	1072
#item	1675	1819
#rating	90,570	113,895
rating range	1–5	1–5

.

As shown in Table 1, the movielens100k dataset includes 943 users with 90,570 ratings for 1675 items. Therefore, its sparsity is extremely high (0.057%). Similarly, the Amazon food dataset is sparse, at around 0.058%. This sparsity is natural with respect to real-world situations in recommendation services [43]. The remaining unknown ratings are a big challenge for the recommender system to predict.

We chose three other related algorithms to compare with the proposed algorithm:

• The rating matrix generative model (RMGT): [23] one of the most popular algorithms for testing cross-domain recommended performance.
• The singular value decomposition-based MF (SVD) [44].
• The SVD++-based MF (SVD++) [45] is an extension of the SVD considering implicit ratings.

For each algorithm, we used gradient descent and implicit stochastic gradient descent, respectively, for optimization.

5.2. Evaluation Metric

We adopt the mean square error (MSE) to measure the accuracy of predicted ratings, which measures the sum of squared distances between our target ratings and predicted values. MSE is defined as follows:

$$\mathbf{MSE}=\frac{{\displaystyle \sum _{i=1}^n{(y_i-y_i^p)}^2}}{n}.$$

(22)

Additionally, we use mean absolute error (MAE) which has frequently been used to compare prediction errors of recommendation methods. This measurement is defined as follows:

$$\mathbf{MAE}=\frac{{\displaystyle \sum _{i=1}^n|y_i-y_i^p|}}{n},$$

(23)

where n denotes the number of tested ratings, $y_i$ is real ratings, and $y_i^p$ is predicted ratings. This approach is used because the predicted rating values create an ordering across the items in which the predictive accuracy can also be used to measure the ability of a recommendation system to rank items with respect to user preference [46].

We use k-fold cross-validation to split the dataset. A k-fold cross-validation is where a given dataset is split into a k number of sections/folds where each fold is used as a testing set at some point. To select a proper k is important since a poorly chosen value may cause a misrepresentation of the methods. In this experiment, k is set as 10, because 5 and 10 have empirically shown to yield test error rate estimates that suffer neither from excessively high bias nor very high variance, according to [47]. Here, the dataset is split into ten folds. In the first iteration, the first fold is used to test the model, and the rest is used to train the model. In the second iteration, the second fold is used as the testing set, while the rest serves as the training set. This process is repeated until each fold of the ten folds has been used as the testing set. As we repeat the process k times, we get k times mean square error (MSE). $MS{E}_1$, $MS{E}_2$, …$MS{E}_k$, so k-fold cross-validation error is computed by taking average of the MSE over k folds.

5.3. Baseline

A matrix factorization method is applied to solve the problem in (15). Eventually, we have to optimize the loss function $\mathcal{L}$. An optimized method based on the gradient descent algorithm is used to solve this problem. Notably, four variables will be separated into two pairs. For each iteration, one of the pairs is kept constant, while the other is optimized [48]. This process repeats sequentially until convergence is achieved based on the push–pull gradient. After convergence, the sub-optimal solution can be obtained. This solution is used as a baseline.

5.4. Experiment Parameters

Two optimization methods are used for comparison: gradient descent (GD) and implicit stochastic gradient descent (ISGD) [49,50]. The set of parameters is presented in

Table 2. Experiment parameters.

Parameters	Values
Regularization parameter $\lambda$	0.01–0.1
K	10–50
Learning rate	50
Initial value of $\mathbf{w},\mathbf{x}$	random
Number of iterations	10

. We have chosen these parameters based on a series of empirical tests.

5.5. Evaluation and Discussions

Now we solve the problem in this paper by optimizing all the variables simultaneously. The implicit stochastic gradient descent (ISGD) method is applied. Firstly, it is necessary to transform the original problem in (15) into the convex problem [51] formulated in (21). The parameters listed in Table 2 are the same as the baseline case. To deal with a vast quantity of variables, it is required to apply some techniques for accelerating convergence rate. Algorithm 1 shows the updating step in each iteration.

Figure 3 shows the typical convergence behavior of the algorithms for the loss function minimization problem. As a result, ISGD needs only a few iterations to reach the convergence value. Moreover, its convergence value is much lower in comparison with the baseline.

When K varies from 10 to 40, as shown in Figure 3. The slope of the ISGD convergence line also changes accordingly. When K is more extensive, this slope also increases. This leads to the initial value of the objective function also increasing significantly. The results showed the larger the K selected, the higher the objective value obtained at the first iteration. When K is larger, the dimensions of $\mathbf{x},\mathbf{w}$ increase accordingly. This leads to an increase in the number of elements in $\mathbf{x},\mathbf{w}$ that makes their values larger. Finally, the value of the objective function will be larger. However, the convergence value is approximately the same.

Let K be 10; the results according to changing the value of $\lambda$ from 0.01 to 0.1 are shown in Figure 4. It shows the difference in the convergence rates when we change the regularization parameter $\lambda$. When $\lambda$ is small, the objective value is obtained as a small value at the first iteration, and the convergence rate is slow. Nevertheless, with the higher $\lambda$ is selected, a higher objective value is obtained at the first iteration accordingly, and the convergence value is reached faster. When K is increased (e.g., 20, 30, 40) and $\lambda$ value is set as the highest value (0.1), the initial objective value is much larger since it is affected by two factors, and the convergence value is reached faster.

We recognize that the parameter K is used to adjust the approximation process. It acts the role of the dimension for approximation. The bigger K is the more accurate approximation. Nevertheless, when K increases, the value of the objective function will increase accordingly. It will be a penalty since it has a norm of $\mathbf{x}$ and $\mathbf{w}$. This leads to a trade-off problem between the MSE and the computation complexity. K can not be so large, and the MSE has to be as small as possible.

Regarding convergence time, we have measured the time until convergence between GD and ISDG. The results are shown in

Table 3. Computation time comparison (seconds).

K Value	GD Method	ISGD Method
K = 10	452	389
K = 20	583	476
K = 30	697	595
K = 40	812	699

. In Figure 5, we can notice that the proposed method shows efficiency in terms of reducing computation time. On average, computation time has been reduced 15.2%.

Furthermore, the proposed method shows a significant result regarding prediction accuracy.

Table 4. Comparison of MAE with other techniques.

	SVD_GD	SVD_ISGD	SVD++_GD	SVD++_ISGD	RMGM_GD	RMGM_ISGD
MAE	0.7812	0.6019	0.7964	0.5938	0.8211	0.612

and Figure 6 show an MAE comparison between our method and other techniques. It shows that the effect of the method in this paper is better than that of other comparison methods on all tests. That is, the experimental result shows that using the ISGD technique to optimize the objective function in MFCF improves the performance of the cross-domain recommendation system.

When K varies from 10 to 40, the implicit update techniques show its efficiency to increase the convergence time. Unfortunately, the objective function has a norm of $\mathbf{x}$ and $\mathbf{w}$, which can lead to a trade-off problem between the MSE and the computation time. Additionally, our goal is to make the MSE to be as small as possible, so K can not be so large. This issue will be the limitation of our paper. We have to make a balance between the accuracy of the recommender system and the computation time.

6. Conclusions and Future Works

In this paper, we proposed a new method to consolidate multiple matrices from multiple domains for building a cross-domain recommendation system. After the consolidation, the matrix was factorized by using MFCF. The problem was to maximize the accuracy of the prediction of unknown ratings of users. To address the design problem, we transformed the original problem into sub-problems of lower dimensions. Then the iterative algorithm was proposed based on the inner approximation method to solve the sequence of convex programs. We applied the implicit stochastic gradient descent method for implicit updating each iteration. Our method with realistic parameters monotonically improved the objective function, and the convergence to a stationery point is guaranteed. Through the experiment, we demonstrated the usefulness of our approach in improving the accuracy of the CDRS.

As future work, we plan to consider using multiple data that have different distributions and attributes to test the performance of a cross-domain recommendation system. Based on this way, we can investigate the appropriate set of parameters for each specific type of data or type of domain in general.