Learning to Co-Embed Queries and Documents

Abstract

Learning to Rank (L2R) methods that utilize machine learning techniques to solve the ranking problems have been widely studied in the field of information retrieval. Existing methods usually concatenate query and document features as training input, without explicit understanding of relevance between queries and documents, especially in pairwise based ranking approach. Thus, it is an interesting question whether we can devise an algorithm that effectively describes the relation between queries and documents to learn a better ranking model without incurring huge parameter costs. In this paper, we present a Gaussian Embedding model for Ranking (GERank), an architecture for co-embedding queries and documents, such that each query or document is represented by a Gaussian distribution with mean and variance. Our GERank optimizes an energy-based loss based on the pairwise ranking framework. Additionally, the KL-divergence is utilized to measure the relevance between queries and documents. Experimental results on two LETOR datasets and one TREC dataset demonstrate that our model obtains a remarkable improvement in the ranking performance compared with the state-of-the-art retrieval models.

1. Introduction

Ranking is one of the most essential techniques in many real-world applications, such as collaborative filtering [1,2,3,4,5], document retrieval [6,7,8], online advertising [9,10,11], and sentiment analysis [12,13]. Good ranking results would positively contribute to the success of all these ranking-based applications. The main objective of ranking is to rank the candidate entities, such as documents in ad hoc retrieval and items in recommendation systems, by their relevance scores to a given query, which can be a set of keywords in ad hoc retrieval and users in recommendation systems. To deal with the ranking problems, most of the traditional approaches [14,15] are built based on some score functions that combine a series of rules according to the historical data and the characteristics of the entities to be ranked. However, such score functions rely strongly on manual design, which is not desirable in many retrieval applications with large scale and various types of data. Compared to score functions, machine learning-based ranking models [16,17,18,19,20,21] have a higher computational efficiency and perform better ranking results than those produced by the traditional score function-based approaches in many applications. Learning to rank (L2R) as one of the most important machine learning techniques to solve the ranking problems have been wildly applied in information retrieval as its ability to improve performance, e.g., accuracy, of the ranking results. Therefore, how to design L2R models for specific applications and how to optimize the models have achieved significant attention in information retrieval these years. Specifically, in this paper, we study the core problem in information retrieval, i.e., designing a L2R model for document retrieval and optimizing the model to produce a final rank list of documents in response to a given query.

Most of the L2R models for document retrieval can be divided into three categories of approaches according to the way they train the models [22]: pointwise, pairwise, and listwise L2R models. The basic idea of pointwise L2R models is to take each query–document pair in the training set as a training data and adopt classification or regression approaches to train the model to obtain a ranking, where each document is treated as a separate training data point. The objective of these ranking methods is to solve the ordinary regression or classification problem. They do not explicitly consider the relationship among relevant documents. Pairwise L2R approaches are to consider the order of two documents in each document pair in terms of their relevances to the given query during training. The learning goal is to make the number of partial errors in the result list as few as possible. Listwise approaches use the entire document lists as training instances. In these approaches, each result list of documents in response to a given query is regarded as training data. The key to the design of these algorithms is to define a loss function based on listwise and select appropriate tools for learning. Listwise L2R models can be broadly divided into two categories: (a) List-level sorting algorithms based on the probability model, such as the ListNet [19] algorithm, one of the well-known representatives that falls into this category. (b) Ranking algorithms based on direct optimization evaluation metrics, e.g., NDCG [23] and MAP [24]. However, it is difficult to select an appropriate optimization algorithm to solve it. Generally, the pairwise and listwise approaches work better than the pointwise approaches [25] because the key issue of ranking in search is to determine the orders of documents but not to judge the relevance of documents, which is exactly the goal of the pairwise and listwise approaches. Compared to listwise approaches, pairwise approaches are more widely used due to the consideration of the two documents’ order in each document pair and the low complexity.

All the aforementioned L2R models for document retrieval suffer from the following defects: (a) Most of the existing L2R models need to extract features of queries and documents. These features are usually extracted from different semantic space, which would result in the mismatch [26,27] between the input query and the documents. (b) Instead of applying language models for document retrieval, some of these L2R models apply word/document embedding techniques [28]. These embedding based L2R approaches represent each word in queries and documents by a single point in a low-dimensional continuous vector semantic space. However, simply representing words as points in the semantic space has a critical limitation: uncertainty of the representations is missing. However, uncertainty is inherent in embeddings and critical for measuring the similarity between queries and documents for document retrieval. Consider the case that we have two documents and a query: the similarity between the first document and the query (the similarity can be measured via, e.g., the cosine similarity between the average of all words’ embeddings in the documents and the average of all words’ embeddings in the input query, with higher cosine score indicating more similar.) is the same as that between the second document and the query, but the certainty (the certainty can be measured via, e.g., taking the covariances of all words’ embeddings in the documents and the input query into account.) between the first document, and the query is higher than that between the second document and the query. Then, the first document should be ranked higher than the second document.

Accordingly, to tackle the drawbacks of the existing L2R approaches, we propose a novel embedding based pairwise L2R model, Gaussian Embedding model for Ranking, abbreviated as GERank, to learn to infer the embeddings and their covariances of queries and documents such that the similarities between any given queries and documents can be effectively measured. In our GERank, queries and documents are co-embedded into the same semantic space, such that the semantic similarities between them can be effectively measured. To further enhance the retrieval performance, GERank represents each query and document as a Gaussian distribution with its mean and covariance. Specifically, given a query and a document pair, GERank enforces embedding of more relevant document in the document pair to be closer to the embedding of the input query compared to the embedding of less relevant document in the document pair. Each document pair in response to a given query in the training dataset naturally leads to our embedding learning constraints in our GERank. Taking into account the constraints of the embeddings and their covariances of queries and documents, we learn more powerful embeddings as we incorporate information about the relevance orders of the documents rather than just the relevance scores of the documents. Since GERank infers not only the embeddings but also the covariances of queries and documents, uncertainty of the embeddings can be applied to measure the similarity between queries and documents and thus contributing to better document retrieval performance. To evaluate the performance of our GERank algorithm, we conduct experiments on two LETOR datasets and one TREC dataset. In our experiments, we aim at answering these research questions: (a) Compared with traditional pairwise approaches, how does our model perform in the document ranking task? (b) Can we make the advantages of our GERank L2R model more explainable? (c) How do the embedding size and the size of hidden layer affect the retrieval performance of our model?

The main contributions of our paper can be summarized as follows:

• We propose Gaussian Embedding model for Ranking, GERank, to co-embed queries and documents into the same semantic space so as to alleviate the mismatch problem between queries and documents.
• To the best of our knowledge, our GERank algorithm is the first attempt to incorporate embedding techniques into L2R algorithms with constraints.
• To further enhance the performance of document retrieval, GERank learns to infer both the embeddings and their covariances of queries and documents such that their similarities can be effectively measured.
• We provide the experimental evidence of the effectiveness of our model. Our model can outperform the state-of-the-art L2R models on two LETOR datasets and one TREC dataset.

The remainder of the paper is organized as follows: Section 2 discusses related work. Section 3 formulates the ranking problem. Section 4 defines our embedding task and GERank algorithm. Section 5 describes experimental setup. Section 6 analyzes experimental results. Finally, Section 7 concludes the paper.

2. Related Work

There are two lines of previous studies related to our work: L2R algorithms and representation learning algorithms.

2.1. Learning to Rank

Learning to rank (L2R) [29,30] applies machine learning techniques to solve ranking problems. There are many classic and efficient L2R algorithms used in information retrieval. These algorithms can be categorised into pointwise, pairwise, and listwise L2R algorithms. Given an input query, pointwise L2R algorithms take each labeled document as a training instance, where the relevance scores of the documents are provided during the training, pairwise L2R algorithms take each pair of document as a training instance, where the ranking order of the two documents in each pair document are provided, while listwise L2R algorithms take different ranked lists of documents as training instances, where the information of which ranked list of documents performs better than another is provided during the training. Well-known pointwise L2R algorithms include subset regression [31] and Mcrank [32], and well-known listwise L2R algorithms include ListNet [19], AdaRank [33] and SVM${}^{map}$ [34]. Ranking SVM [16,17], RankBoost [18], and Frank [35] are the well-known algorithms based on the pairwise approach. Point-wise L2R algorithms ignore the order information of the documents, while generating a number of ranked lists of documents in response to a given query via different methods is challenging in listwise L2R algorithms. Of special interest to us are the pairwise L2R algorithms.

In recent years, deep learning is a hot research topic because of its powerful representation abilities. Combining deep learning to solve problems in computer vision, natural language processing, and other fields has achieved great success. In information retrieval, some researchers try to solve ranking problems with deep learning. Severyn et al.  [36] applied convolutional deep neural networks to rank short text pairs. Wang et al.  [37] proposed an attention-based deep net for listwise. ConvRankNet [38] combines a Siamese Convolutional Neural Network encoder and the RankNet ranking model which could be trained in an end-to-end fashion. Ai et al.  [39] employed a RNN to encode the top results using their feature vectors, learn a context model and use it to re-rank the top results. Zhao et al.  [40] proposed a novel joint learning-to-rank approach called Deep Latent Structural SVM (DL-SSVM). DL-SSVM can effectively model the intrinsic interaction relationships between the feature-level and ranking-level components of a ranking model. They presented an effective auxiliary variable-based alternating optimization approach with respect to deep neural network learning and structural latent SVM learning. These deep learning-based models can outperform the state-of-the-art. However, they are not parameterless and scalable. In addition, all the previous L2R algorithms do not take uncertainty of the relevances of the documents to the query into account, resulting in the fact that there is still some room to boost the performance of document retrieval via capturing and utilizing the uncertainty information of the queries and documents.

2.2. Representation Learning

Learning representations for data allows many data mining and information retrieval tasks to be solved more effectively and efficiently. For instance, DeepWalk [41] employs Skip-gram [42] to learn the representations of nodes. The model learns low-dimensional vectors for the nodes to capture the potential relationships among them. It outperforms challenging baselines which are allowed a global view of the network, especially in the presence of missing information. Shen et al.  [43] use convolutional neural networks for Web Search by learning semantic representations. Their model significantly outperforms other semantic models on a large-scale, real-world dataset.

In natural language processing, word embedding is widely used in a set of language modeling and feature learning techniques. In embedding techniques [44,45,46], entities, e.g., words, phrases, or documents, are mapped to low-dimensional continuous vectors, which are also called representations or embeddings. Generally, representation learning techniques involve a mathematical embedding from a high dimensional vector space to a continuous much lower dimensional vector space. Lai et al.  [47] provide several simple guidelines for training good word embeddings. They systematize existing neural-network-based word embedding methods and experimentally compare them using the same corpus. Zamani et al.  [48] propose two learning models with different objective functions: one learns a relevance distribution over the vocabulary set for each query, and another classifies each term as belonging to the relevant or non-relevant class for each query. Shen et al.  [49] conduct a point-by-point comparative study between Simple Word-Embedding-based Models, consisting of parameter-free pooling operations, relative to word-embedding-based RNN/CNN models. Their model exhibits comparable or even superior performance in the majority of cases considered.

Graph embedding [50,51,52,53] uses low-dimensional dense vectors to represent the points in the graph. In essence, the more adjacent the points are shared between two points, the more similar the context of two points is. The distance between two corresponding vectors is closer. The greatest advantage of graph embedding is that the vector representations can be taken as an input of any machine learning models to solve specific application problems. At the same time, it outperforms some traditional methods in a number of ways. For instance, the method based on matrix factorization (MF) requires too much computation. Constructing artificial features requires domain knowledge and a large amount of work. Graph embedding can be used in recommendation, node classification, link prediction, visualization, and other scenario. Vilnis et al.  [54] use Gaussian embeddings to represent words, which have great performance. He et al.  [55] represent knowledge graphs by Gaussian embedding. Zhou et al.  [56] propose a novel Gaussian Visual-Semantic Embedding (GVSE) model to jointly represent images and texts, which leverages the visual information to model text concepts as Gaussian distributions in semantic space. Bojchevski et al.  [57] learn versatile node embeddings on large scale graphs that show strong performance on tasks such as link prediction and node classification. However, there are few studies on L2R by Gaussian embedding.

Recently, few works begin to apply representation learning to the field of L2R. Yi et al.  [28] proposed a long short-term memory (LSTM) network with holographic composition (HD-LSTM) to model the relationship between question and answer representations. Benefitting from a rich representational learning approach without incurring huge parameter costs of holographic composition, HD-LSTM outperforms many other neural architectures on the benchmark dataset [58]. Inspired by their work and the idea of embedding nodes as Gaussian distribution, we adopt co-embedding queries and documents methods to capture the relationship between them.

3. Notations and Problem Formulation

Let $\mathcal{C}$, $\mathcal{Q}$ and ${\mathcal{D}}_{\mathcal{Q}}$ be a document corpus, a set of queries, and the relevant documents to a set of queries $\mathcal{Q}$, respectively. Let q, d, and ${\mathcal{D}}_q$ denote a query, a document, and a set of relevant documents to the query q, respectively.

Table 1. Main notations used across the whole paper.

Symbol	Gloss
$\mathcal{C}$	Document corpus
$\mathcal{Q}$	Set of queries
${\mathcal{D}}_{\mathcal{Q}}$	Relevant documents to the queries
${\mathcal{D}}_q$	Set of relevant documents to the query q
M	Number of queries
L	Dimension of embedding space
$K^i$	Number of relevant document for a given ith query
$r_j^i$	Relevance degree of $d_j^i$ for a given query $q^i$
$\mathcal{T}$	Triplet set of query and two relevant documents
$q^i$	ith query in $\mathcal{Q}$, $i\in M$
$d_j^i$	jth document for a given ith query, $j\in K^i$
${\mathcal{N}}_{q^i},{\mathbf{q}}^i$	Gaussian distribution of ith query
${\mathcal{N}}_{d_j^i},{\mathbf{d}}_j^i$	Gaussian distribution of document $d_j^i$, $j\in K^i$
${\mathbf{\mu }}_{q^i}$	Mean of query $q^i$ in Gaussian distribution
${\mathbf{\mu }}_{d_j^i}$	Mean of document $d_j^i$ in Gaussian distribution
${\Sigma }_{q^i}$	Covariance of ith query in Gaussian distribution
${\Sigma }_{d_j^i}$	Covariance of document $d_j^i$ in Gaussian distribution
$\Delta ({\mathbf{q}}^i,{\mathbf{d}}_j^i)$	Dissimilarity measure between query ${\mathbf{q}}^i$ and document ${\mathbf{d}}_j^i$
$D_{KL}({\mathcal{N}}_{d_j^i}\Vert {\mathcal{N}}_{q^i})$	KL divergence between ${\mathcal{N}}_{d_j^i}$ and ${\mathcal{N}}_{q^i}$
$d_j^i\prec d_{j^{\prime }}^i$	Document pair in which $d_j^i$ is ranked lower than $d_{j^{\prime }}^i$ to query $q^i$

summarizes the main notations used across the whole paper.

Given a document corpus $\mathcal{C}$, a set of queries $\mathcal{Q}$, and their relevant documents ${\mathcal{D}}_{\mathcal{Q}}\in \mathcal{C}$ in the corpus, we aim at training a Gaussian embedding model, GERank, such that it is able to infer a Gaussian distribution $\mathbf{q}=\mathcal{N}({\mathbf{\mu }}_q,{\Sigma }_q)$ for any given query q and a Gaussian distribution $\mathbf{d}=\mathcal{N}({\mathbf{\mu }}_d,{\Sigma }_d)$ for any given document d via a supervised way, where $\mathbf{\mu }\in {\mathbb{R}}^L$ is the embedding (mean of the Gaussian distribution) of the query/document, and $\Sigma \in {\mathbb{R}}^{L\times L}$ is the corresponding uncertainty of the embedding (covariance of the Gaussian distribution), such that documents semantically similar to the input queries are also close to each other in the same embedding space measuring by a dissimilarity metric $\Delta (\mathbf{q},\mathbf{d})$. Here, $\mathbf{q}=\mathcal{N}({\mathbf{\mu }}_q,{\Sigma }_q)$ and $\mathbf{d}=\mathcal{N}({\mathbf{\mu }}_d,{\Sigma }_d)$ are the inferred Gaussian distributions ${\mathbf{\mu }}_q$ and ${\mathbf{\mu }}_d$ for the query q and the document d and their covariances ${\Sigma }_q$ and ${\Sigma }_d$, respectively; and L is the size of the embedding dimension. Specifically, we aim at seeking a low dimensional Gaussian distribution co-embedding query and document function h, i.e., the Gaussian embedding model, GERank that satisfies the following via a supervised way:

$$\begin{array}{cc}\hfill & \mathcal{C},\mathcal{Q},{\mathcal{D}}_{\mathcal{Q}}\stackrel{h}{⟶}\mathbf{q}=\mathcal{N}({\mathbf{u}}_q,{\Sigma }_q),\mathbf{d}=\mathcal{N}({\mathbf{u}}_d,{\Sigma }_d),\mathrm{for}\mathrm{any}\mathrm{given}q\mathrm{and}d,\hfill \\ \hfill & \mathrm{subject}\mathrm{to}:\forall q\in \mathcal{Q},r_{d_i}^q>r_{d_j}^q,\hfill \\ \hfill & \Delta (\mathbf{q},{\mathbf{d}}_i)<\Delta (\mathbf{q},{\mathbf{d}}_j),\hfill \end{array}$$

(1)

where $r_{d_i}^q$ and $r_{d_j}^q$ are the relevance scores of documents $d_i$ and $d_j$ in response to the input training query q, respectively. Once the model is optimized, given an input query q and a document d, GERank is able to infer their corresponding Gaussian embeddings, and then rank the documents according the dissimilarity metric $\Delta (\mathbf{q},\mathbf{d})$ where uncertainty is taken into account as $\mathbf{q}$ and $\mathbf{d}$ not only containing their means but also their corresponding covariances.

4. Learning to Co-Embed Queries and Documents

In this section, we detail our proposed model, GERank, that aims at inferring the embeddings of the queries and the documents, ${\mathbf{\mu }}_q$ and ${\mathbf{\mu }}_d$, and their corresponding covariances, ${\Sigma }_q$ and ${\Sigma }_d$. Specifically, in Section 4.1, we detail the way to compute the dissimilarity for each query document pair, i.e., the way to compute $\Delta (\mathbf{q},\mathbf{d})$; Section 4.2 details the constraints related to the dissimilarity between the inferred low dimensional Gaussian embeddings and their covariances of the queries $\mathbf{q}=\mathcal{N}({\mathbf{\mu }}_q,{\Sigma }_q)$ and those of the documents $\mathbf{d}=\mathcal{N}({\mathbf{\mu }}_d,{\Sigma }_d)$; Section 4.3 details our L2R model that aims at inferring $\mathbf{q}$ and $\mathbf{d}$.

4.1. Dissimilarity Metric

In order to effectively infer the embeddings and their covariances of the queries and documents for document retrieval, we employ the Kullback–Leibler (KL) divergence as a metric to measure the dissimilarities between queries and documents in our model GERank, which has been widely applied in many L2R models [55,57,59,60,61]. Specifically, given the embeddings and their covariances of a query–document pair, $(\mathbf{q},\mathbf{d})$, i.e., $\mathbf{q}=\mathcal{N}({\mathbf{\mu }}_q,{\Sigma }_q)$ and $\mathbf{d}=\mathcal{N}({\mathbf{\mu }}_d,{\Sigma }_d)$, we define the dissimilarity between $\mathbf{q}$ and $\mathbf{d}$ as follows:

$$\begin{array}{cc}\Delta (\mathbf{q},\mathbf{d})\hfill & \stackrel{def}{=}{\mathrm{D}}_{\mathrm{KL}}\left(\mathcal{N}({\mathbf{\mu }}_d,{\Sigma }_d)\parallel (\mathcal{N}({\mathbf{\mu }}_q,{\Sigma }_q)\right)\hfill \\ \hfill & \stackrel{def}{=}{\mathrm{D}}_{\mathrm{KL}}\left(M_{\mathbf{\theta }}\left(d\right)\parallel M_{\mathbf{\theta }}\left(q\right)\right)\hfill \\ \hfill & =\frac{1}{2}\left[\mathrm{tr}\left({\Sigma }_q^{-1}{\Sigma }_d\right)+{({\mathbf{\mu }}_q-{\mathbf{\mu }}_d)}^{\top }{\Sigma }_q^{-1}({\mathbf{\mu }}_q-{\mathbf{\mu }}_d)-L-\log \frac{\det \left({\Sigma }_d\right)}{\det \left({\Sigma }_q\right)}\right]\hfill \end{array}$$

(2)

where $\mathrm{tr}(·)$ denotes the trace of a matrix, ${\mathrm{D}}_{\mathrm{KL}}(·\parallel ·)$ is the KL divergence between two distributions, $M_{\mathbf{\theta }}\left(q\right)$ and $M_{\mathbf{\theta }}\left(d\right)$ are our learning to co-embedding model, GERank, with parameters $\mathbf{\theta }$ that need to be tuned during training, L is the size of the embedding dimension, and $\det (·)$ is the determinant of a matrix. Once the parameters $\mathbf{\theta }$ of our model $M_{\mathbf{\theta }}$ are tuned, i.e., the optimal parameters ${\mathbf{\theta }}^{*}$ is obtained, given an input query q (either in the training set of queries or not) and an input document d (either in the training corpus or not), we can infer their optimal embeddings and the corresponding covariances $({\mathbf{\mu }}_q^{*},{\Sigma }_q^{*})$ and $({\mathbf{\mu }}_d^{*},{\Sigma }_d^{*})$, such that we have $({\mathbf{\mu }}_q^{*},{\Sigma }_q^{*})=M_{{\mathbf{\theta }}^{*}}\left(q\right)$ and $({\mathbf{\mu }}_d^{*},{\Sigma }_d^{*})=M_{{\mathbf{\theta }}^{*}}\left(d\right)$. Detailed derivation of Equation (2) is shown in Appendix A.

4.2. Constraints

Given an input training query, our GERank model aims at imposing a ranking of all training documents with regard to their relevance scores to the query in the same semantic space. Specifically, GERank exploits the relevance scores of each training document to an input training query. Let $q^i$ be the i-th query in the training set of queries, $d_{j_1}^i,d_{j_2}^i,\dots ,d_{j_k}^i$ be a set of labeled documents with their relevance scores $r_{d_{j_1}}^i,r_{d_{j_2}}^i,\dots ,r_{d_{j_k}}^i$ in response to the query $q^i$ having this order $r_{d_{j_1}}^i<r_{d_{j_2}}^i<,\dots ,<r_{d_{j_k}}^i$. GERank aims at obtaining the model’s optimal parameters ${\mathbf{\theta }}^{*}$ such that the following constraints would be satisfied as many as possible:

$$\Delta ({\mathbf{q}}^i,{\mathbf{d}}_{j_1}^i)>\Delta ({\mathbf{q}}^i,{\mathbf{d}}_{j_2}^i)\cdots ,>\Delta ({\mathbf{q}}^i,{\mathbf{d}}_{j_k}^i),\forall q^i\in \mathcal{Q},\mathrm{with}{r}_{d_{j_1}}^i<r_{d_{j_2}}^i<,\dots ,<r_{d_{j_k}}^i,$$

(3)

where ${\mathbf{q}}^i=\mathcal{N}({\mathbf{\mu }}_{q^i},{\Sigma }_{q^i})=M_{\mathbf{\theta }}\left(q^i\right)$ and ${\mathbf{d}}_{j_k}^i=\mathcal{N}({\mathbf{\mu }}_{d_{j_k}^i},{\Sigma }_{d_{j_k}^i})=M_{\mathbf{\theta }}\left(d_{j_k}^i\right)$ are the embeddings and their corresponding covariances of the query $q^i$ and the document $d_{j_k}^i$ that need to be inferred by GERank, respectively. The motivation of defining such constraints in Equation (3) is straightforward: more relevant documents should be closer to the input query in the same semantic space. As we infer both the embeddings and covariances (uncertainty) of queries and documents by the same model $M_{\mathbf{\theta }}$, they are co-embedded into the same semantic space. Of special interest to us is the setting of pairwise L2R, where a set of training triples, $(q^i,d_j^i\prec d_{j^{\prime }}^i)\in \mathcal{T}$, are provided with the information of $d_j^i$ should be ranked lower (less relevant) than $d_{j^{\prime }}^i$ in response to query $q^i$ according to the ground truth, denoted as $d_j^i\prec d_{j^{\prime }}^i$ (in some cases, the relevance scores of $d_{j^i}$ and $d_{j^{\prime }}^i$ may not be provided but only their ranking order is provided; documents that are likely to be relevant to the query should be ranked higher in the final ranking). Accordingly, the constraints in Equation (3) can be transformed as the following:

$$\Delta ({\mathbf{q}}^i,{\mathbf{d}}_j^i)>\Delta ({\mathbf{q}}^i,{\mathbf{d}}_{j^{\prime }}^i),\forall (q^i,d_j^i\prec d_{j^{\prime }}^i)\in \mathcal{T}.$$

(4)

4.3. Co-Embedding Queries and Documents

As it is intractable to obtain the optimal parameters $\mathbf{\theta }$ in our GERank model, $M_{\mathbf{\theta }}$, which is able to satisfy all the pairwise constraints defined in Equation (4), we instead turn to minimizing an energy based learning objective. We follow the previous energy-based objective function [62] to define our own energy-based objective function that needs to be minimized during training. The main idea is to define an objective function that satisfies this: for each document pair in response to an input training query, document in the pair ranked higher (more relevant documents) should obtain lower energy compared to another document in the pair ranked lower. Thus, the energy between an input query q and a document d can be defined as: $\mathrm{E}(q,d)={\mathrm{D}}_{\mathrm{KL}}(M_{\mathbf{\theta }}\left(d\right)\parallel M_{\mathbf{\theta }}\left(q\right))={\mathrm{D}}_{\mathrm{KL}}\left(\mathcal{N}({\mathbf{\mu }}_d,{\Sigma }_d)\parallel (\mathcal{N}({\mathbf{\mu }}_q,{\Sigma }_q)\right)$. Accordingly, given a set of training triples, $(q^i,d_j^i\prec d_{j^{\prime }}^i)\in \mathcal{T}$, we define the following energy-based objective function that aims at satisfying as many constraints as possible so as to co-embed queries and documents in the same semantic space:

$$\mathcal{L}\left(\mathcal{T}\right)=\sum _{(q^i,d_j^i\prec d_{j^{\prime }}^i)\in \mathcal{T}}{\mathrm{E}}^2(q^i,d_{j^{\prime }}^i)+exp\left(-\mathrm{E}(q^i,d_j^i)\right).$$

(5)

Once the energy-based objective function in Equation (5) is minimized, given a new query q and a new document d, we can infer their embeddings and their covariances according to $M_{{\mathbf{\theta }}^{*}}\left(q\right)$ and $M_{{\mathbf{\theta }}^{*}}\left(d\right)$, and then the documents are ranked based on the similarities between $M_{{\mathbf{\theta }}^{*}}\left(q\right)$ and $M_{{\mathbf{\theta }}^{*}}\left(d\right)$. In practice, $M_{\mathbf{\theta }}$ can be defined as a neural network with the parameters being $\mathbf{\theta }$. The parameters $\mathbf{\theta }$ can be optimized using the Adam algorithm [63]. The proof that the loss score of the objective function in Equation (5) can be converged after enough number of iterations can be converged is provided in Appendix B. The training process of our GERank is shown in Algorithm 1. Framework of our GERank is provided in Figure 1. Details of how we build our neural network $M_{\mathbf{\theta }}$ for a query and a document can be found later in Section 5.5.

Algorithm 1: Training in GERank.

5. Experimental Setup

In what follows, we first list our research questions in Section 5.1. Next, we describe our datasets as well as the data preprocessing procedure in Section 5.2. Section 5.3 and Section 5.4 detail the baselines and metrics for evaluation, respectively. Finally, we describe our experimental settings in Section 5.5.

5.1. Research Questions

The research questions guiding the remainder of the paper are as follows:

• In terms of the performance of GERank:

RQ1 

How does our GERank perform in the task of document retrieval?

RQ2 

Can our GERank outperform traditional pairwise-based L2R methods?

RQ3 

Can we make the advantages of our GERank L2R model explainable?

In terms of the parameters, $\mathbf{\theta }$, of our GERank model:

RQ4 

How does the embedding size affect the retrieval performance of our GERank model?

RQ5 

How does the size of the hidden layer affect the retrieval performance of our GERank model?

To answer the research questions RQ1, RQ2 and RQ3, we conduct a series of experiments on the document retrieval task compared with the state-of-the-art baseline methods (Section 6.1). To answer RQ4, we fix other variables and tune the embedding dimension to analyze the impact on the document retrieval performance of our model (Section 6.2). Similarly, to understand the impact of hidden layer size in our model (RQ5), we conduct experiments based on different sizes of hidden layers of our model and analyze the result (Section 6.3).

5.2. Datasets

For our experiments, we use the LETOR package (version 3.0) (Available from https://www.microsoft.com/en-us/research/project/letor-learning-rank-information-retrieval, accessed on 10 August 2022) provided by Microsoft Asia [22]. The LETOR package consists of several benchmark datasets and baseline results including those generated by some L2R algorithms. In these datasets, information is represented as query–document pairs by means of “meta-level” features that try to capture the relevant relationships between queries and documents in an information retrieval system. We conduct experiments on all the two datasets in the LETOR 3.0 package, i.e., NP2004 and OHSUMED datasets, to examine the effectiveness of our GERank model. These two datasets have different properties which help to demonstrate the properties of our GERank model in different scenarios. We also use a widely used TREC (https://trec.nist.gov/, accessed on 10 August 2022) dataset, Robust04, which is a TREC collection with documents from the news domain that many previous and most recent works used for retrieval evaluation purposes [64,65,66,67]. In the following, we describe each of these datasets in more details:

• NP2004: The NP2004 dataset is originally collected for the task of named-page finding in TREC2004 [68]. It contains a query set with 75 queries and 73,834 query–document pairs with ratings of 0 and 1 corresponding to “non-relevant” and “relevant” relationships between the queries and the documents, respectively. In the NP2004 dataset, each query–document pair is represented by 64 features, including the BM25 scores, Inverse Document Frequency (IDF) scores, PageRank scores, etc. These features are categorized into three classes, i.e., those depending only on the query, only on the document and on both of the query and the document, respectively. Specifically, five features in each query–document pair are only depended on query, which are constructed by the IDF of query term in body, anchor, title, URL and whole document.
• OHSUMED: The OHSUMED dataset is obtained by Qin et al. [22] from the OHSUMED corpus [69]. It contains a query set with 106 queries and 16,140 query–document pairs with three types of ratings, i.e., 0, 1, and 2, corresponding to “non-relevant”, “partially relevant” and “definitely relevant” relationships between the queries and the documents, respectively. As for OHSUMED, each query–document pair can be represented by 45 features, including the language model features, BM25 scores, and the others. These features are categorized into two classes, i.e., those depending only on the query, and on both the query and the document. There are 9 query-depend features in each query–document pair, and the details of these features can be found in [22].
• Robust04: This TREC dataset consists of documents from the Financial Times Limited, the Congressional Record of the 103rd Congress, the Federal Register, the Foreign Broadcast Information Service, and the Los Angeles Times, and has been widely used in a variety of TREC tasks, including TREC ad hoc collections 6–8, TRECs 8–9 question answering track, and the TREC Robust track. We apply the BERT model to extract 100-dimensional features for each document and also 100-dimensional query-depend features.

The statistics information of the two datasets is summarized in

Table 2. Statistics information of the NP2004 and OHSUMED datasets.

Datasets	#Queries	#Documents	#Features	#Query-Depend Features	#Relevances
NP2004	75	73,834	64	5	2
OHSUMED	106	16,140	45	9	3
Robust04	249	556,077	100	100	2

.

In order to transfer the original dataset into the desired format for our model, we apply two procedures to process the data. Figure 2 provides an example of transferring the data in NP2004 dataset into the input format of our model. The data in the OHSUMED and Robust04 datasets are processed in the same way. On top of Figure 2, each row represents a document. The first column denotes the relevant score of the document being relevant to the query. A larger score indicates a higher relevance level of the document to the input query. For instance, the relevant scores “1” and “0” in the first row denote the document is “non-relevant” and “relevant” to the query, respectively, and the second column denotes the query ID. In the figure, “qid:1” indicates that this document is retrieved for the query with its query ID being 1. The last column of the first row, i.e., “#docid=G29-66-2836593”, gives the document ID. The columns in the middle are features extracted from this query–document pair.

For data preprocessing, we adopt the Minmax normalization to normalize the scores of all the features. We apply a uniform sampling strategy to obtain the triple set $\mathcal{T}$ from the training set. Specifically, we randomly choose a query $q^i$ and a relevant document $d_j^i$ with rating score being 0 from the dataset, then sample another document $d_{j^{{}^{\prime }}}^i$ whose score is larger than 0. We then obtain each triple sample $(q^i,d_j^i\prec d_{j^{{}^{\prime }}}^i)$ for training the model. As for the construction of the query $q^i$, we use the same features of the query–document pair to represent it. The query-depend features of the query $q^i$ are extracted from one of the related query–document pairs and the scores of other features are 0.

5.3. Baselines

We compare our model with three types of L2R baselines, i.e., the pointwise, pairwise, and listwise L2R approaches. Specifically, one pointwise approach, i.e., the Linear regression based algorithm (LR), three pairwise approaches, i.e., Ranking SVM [16,17], RankBoost [18] and FRank [35], and four listwise approaches, i.e., ListNet [19], AdaRank-MAP [33], AdaRank-NDCG [33], and SVM${}^{map}$ [34] are taken as our L2R baselines. Details of the baselines are provided as follows:

• LR maps a feature vector to a retrieval score by a linear function. Given a training query, relevance score of each document is used to train the model. The model aims at enforcing the relevance score of more relevant documents to be greater than that of a less relevant document.
• Ranking SVM [16,17] transforms L2R to a binary classification problem. It considers the partial ordering of the documents and solves the binary classification problem using a Support Vector Machine. SVM-light, a public tool (The SVM-light program is available from the web site http://www.cs.cornell.edu/people/tj/svm_light/svm_rank.html, accessed on 10 August 2022), is used in the experiment. The ranking function is linear.
• RankBoost [18] adopts the Adaboost algorithm for the classification over document pairs given the queries. The distribution in RankBoost is defined over document pairs. For each iteration, RankBoost trains a weak document ranking model. These weak models are combined to obtain the final retrieval model. The weights of document pairs are changed by decreasing the weights of correctly ranked pairs. We follow Qin et al. [22] to define each weak ranker on the basis of a single feature.
• FRank [35] is a pairwise ranking algorithm with a fidelity loss function. The fidelity was originally used in quantum physics to measure the difference between two probabilistic states of a quantum. In our experiments, as a baseline, it is used to measure the difference between the target probability and the modeled probability.
• SciBERT [70] is a representation learning mechanism that integrates structural relations with semantic information to enrich the document embeddings for document retrieval. It involves training a document representation model that encodes the corpus structure along with the content semantics into the learned document embeddings in a metric learning setting. It makes similar documents close in the representation space while dissimilar documents separated. It accepts similar and dissimilar document pairs as input, one each for content semantics and corpus structure.
• ListNet [19] is based on probability distribution on permutations. The main work of ListNet is to map the relevance of query–document pairs to a real-value score. For a document list, the Luce model used by definition a permutation probability distribution based on score and another based on the ground truth labels. The loss function of the model is the cross enthalpy between two distributions.
• AdaRank-MAP and AdaRank-NDCG [33] are built based on the Adaboost algorithm. They aim to repeatedly construct weak rankers on the basis of re-weighted training queries and finally linearly combine the weak rankers to make ranking predictions. The difference is that AdaRank-MAP uses MAP to measure the effectiveness of a weak ranker, while AdaRank-NDCG utilizes NDCG to optimize the model.
• SVM${}^{map}$ [34] utilizes the framework of structured SVM to optimize the evaluation measure, i.e., MAP. The main idea of SVM${}^{map}$ is to use Support Vector Machines to solve the ranking problem.

5.4. Evaluation Metrics

For model evaluation, we adopt three categories of evaluation metrics that are commonly used in existing document retrieval algorithms [24]: mean average precision (MAP), precision at position k (P@k), and normalized discounted cumulative gain at position k (NDCG@k). Note that all evaluation scores in our experiments were computed under the trec_eval public code (The trec_eval program is available from the TREC web site http://trec.nist.gov, accessed on 10 August 2022).

• NDCG: NDCG is a measure of ranking quality that is computed based on the discounted cumulative gain. The discounted cumulative gain (DCG) [23] at a particular rank threshold k is defined as

  $$\mathrm{DCG}(\mathcal{S},k)=\sum _{j=1}^k\frac{2^{r\left(j\right)}-1}{log(1+j)},$$

  where $r\left(j\right)$ is the judgment ($0=\mathsf{Bad}$, $1=\mathsf{Fair}$, $2=\mathsf{Good}$, $3=\mathsf{Excellent}$, etc.) at rank j in set $\mathcal{S}$. The ideally ordered set $\mathcal{R}$ contains all documents rated for the given query sorted descending by the judgment value. Then, the normalized discounted cumulative gain (NDCG) [23] at a particular rank threshold k is defined as

  $$\mathrm{NDCG}(\mathcal{S},k)=\frac{\mathrm{DCG}(\mathcal{S},k)}{\mathrm{DCG}(\mathcal{R},k)}.$$
  NDCG discounts the contribution of a document to the overall score as its rank increases. NDCG value at rank threshold k when the set $\mathcal{S}$ is clear from the context is often written as $\mathrm{NDCG}@k$.
• MAP: The mean average precision (MAP) [24] of a test query set is the mean of the average precision (AP) values of all queries in the query set. The average precision of a ranked result set in response to a given query is defined as:

  $$\mathrm{AP}=\frac{{\sum }_{j=1}^{k}P\left(j\right)\ast \mathrm{Relevance}\left(j\right)}{{\sum }_{j=1}^k\mathrm{Relevance}\left(j\right)},$$

  where j is the position of the document (in our case, group), $\mathrm{Relevance}\left(j\right)$ denotes the relevance of the document (in our case, group) in position j, and $P\left(j\right)={\sum }_{i=1}^j\mathrm{Relevance}\left(i\right)/j$. Typically, a binary value for $\mathrm{Relevance}\left(j\right)$ is used by setting it to 1 if the document (group) in position j has a human judgment of $\mathsf{Fair}$ or better and 0 otherwise.
• Precision:Precision at k [24] only considers the total number of relevant documents ranked within the top k positions and can be simply computed as

  $$\mathrm{P}@k=\frac{\#\mathrm{relevant}\mathrm{documents}\mathrm{among}\mathrm{the}\mathrm{top}k}{k},$$

  where “relevant documents” are those that have human judgments of Fair or better.

5.5. Experimental Settings

We use 5-fold cross validation for reporting all the experimental results in terms of the evaluation metrics, NDCG, MAP, and Precision. We use the evaluation script provided by the LETOR package to generate the evaluation scores. In our experiments, we use a 3/1/1 split for our training, validation, and test sets, respectively. We train our L2E model and the baseline models using different values of the parameters in the models. The best values of the parameters in each model are then chosen on the validation set, and evaluated on the test queries. The training/validation/testing splits are permuted until all the queries in each of the datasets were chosen once for the test set. We repeat the experiments 10 times and report the average evaluation results.

We conduct experiments for all the baselines using the codes released by the authors, and the parameters of them are tuned to be optimal. Specifically for RankBoost, the best weak ranker was selected from (255 thresholds) × (number of features) candidates in each iteration. The number of iterations is determined by the evaluation metric MAP on the validation set. To the baseline FRank, it combines several weak learners to minimize the fidelity loss function. The number of weak learners for it was determined by verifying on the validation set. As for ListNet, it is trained by minimizing the cross entropy between output score and ground truth label. ListNet maps the relevance label of query–document pair to a real-value score. The mapping is determined by utilizing the validation set. AdaRank-MAP and AdaRank-NDCG manage to optimize MAP and NDCG to measure the effectiveness of weak rankers. These two algorithms work based on AdaBoost. The number of weak rankers which are combined to the final model is determined by the validation set. SVM${}^{map}$ is a structured SVM approach by optimizing a loss function concerned about MAP. The value of hyper parameter in SVM${}^{map}$ is determined by using the validation set.

We define our GERank, $M_{\mathbf{\theta }}$, as a two-layer feed-forward network to embed queries and documents. The first layer uses Relu as the activation function. Then, we treat queries and documents in the same way. It means that they share the same neural network. Then, queries and documents are passed through deep feed-forward nonlinear neural networks. We obtain their means $\mathbf{\mu }$ and diagonal covariance matrices $\Sigma$ (To guarantee their positive definite, we design $\tilde{\sigma }=elu\left({\sigma }^{\prime }\right)+1$, where ${\sigma }^{\prime }$ is the output in the final layer) from the output in the final layer. We learn the neural network parameters to satisfy the pairwise constraints by minimizing the loss function $\mathcal{L}$ and using the Adam Optimizer to optimize the loss function for our GERank model. In order to shorten the training time, avoid the risk of over-fitting, and guarantee good performance of the model, we control the training steps based on the performance of GERank on the validation set. It should be emphasized that $\mathbf{\mu }$ and $\Sigma$ share the parameters of the neural network.

The statistical significance on differences of the performance between two models is tested using a two-tailed paired t-test. Here, we use ${}^{▴}$ to denote a significance difference for $p<0.01$, and ${}^{\Delta }$ for $p<0.05$.

6. Results

In this section, we report and analyze our experimental results. To answer RQ1, we start by reporting the overall document retrieval performance of our model and the baseline models in Section 6.1. Then, to answer RQ2, we compare the performance of GERank with several pairwise baselines over nine evaluation metrics in Section 6.1. After that, to answer RQ3, we analyze the experimental results and try to give a reasonable explanation in Section 6.2. To further investigate the properties of the model, we also vary the embedding dimension (to answer RQ4) and the size of the hidden layer (to answer RQ5) and report the results in Section 6.3.

6.1. Overall Retrieval Performance

In order to answer research questions RQ1 and RQ2, we first compare our GERank with the baseline methods over NDCG, MAP, and Precision metrics on the NP2004 dataset.

Table 3. Performance of our GERank and the baseline models in terms of Precision and MAP on the NP2004 dataset. The best performance per metric among the pairwise L2R algorithms is in boldface. Statistically significant improvement of the performance of our GERank model per metric compared to the best pairwise L2R baseline algorithm is marked in the upper right-hand corner of GERank performance score.

Approach	Model	P@1	P@3	P@5	P@10	MAP
Pointwise	LR	0.3733	0.2000	0.1440	0.0820	0.5142
Listwise	ListNet	0.5333	0.2667	0.1787	0.0940	0.6720
	AdaRank-MAP	0.4800	0.2444	0.1627	0.0880	0.6220
	AdaRank-NDCG	0.5067	0.2489	0.1653	0.0900	0.6269
	SVM${}^{map}$	0.5200	0.2667	0.1787	0.0960	0.6620
Pairwise	Ranking SVM	0.5067	0.2622	0.1787	0.0930	0.6588
	RankBoost	0.4267	0.2311	0.1520	0.0880	0.5640
	SciBERT	0.4326	0.2324	0.1554	0.0892	0.5721
	FRank	0.4800	0.2356	0.1600	0.0930	0.6008
	GERank	${\mathbf{0}.\mathbf{5734}}^{▴}$	0.2489	0.1653	0.0893	${\mathbf{0}.\mathbf{6657}}^{▴}$

shows the performance of all the methods on precision and MAP, while

Table 4. NDCG on NP2004 dataset. The best performance per metric among the pairwise L2R algorithms is in boldface. Statistically significant improvement of the performance of our GERank model per metric compared to the best pairwise L2R baseline algorithm is marked in the upper right hand corner of GERank performance score.

Approach	Model	NDCG@1	NDCG@3	NDCG@5	NDCG@10
Pointwise	LR	0.3733	0.5554	0.6135	0.6530
Listwise	ListNet	0.5333	0.7587	0.7965	0.8120
	AdaRank-MAP	0.4800	0.6979	0.7310	0.7490
	AdaRank-NDCG	0.5067	0.6722	0.7122	0.7380
	SVM${}^{map}$	0.5200	0.7489	0.7869	0.8079
Pairwise	Ranking SVM	0.5067	0.7503	0.7957	0.8060
	RankBoost	0.4267	0.6274	0.6512	0.6914
	SciBERT	0.4374	0.6352	0.6678	0.7021
	FRank	0.4800	0.6431	0.6870	0.7290
	GERank	${\mathbf{0}.\mathbf{5734}}^{▴}$	0.6620	0.6914	0.7095

shows the performance on NDCG@k with k being set to be 1, 3, 5, and 10. In general, we can observe that GERank always yields very high competitive performances over all the evaluation metrics. From Table 3, we can observe that LR is the worst baseline among all the other baseline models, while our GERank achieves the best performance on P@1, MAP, and NDCG@1 metrics, and the improvement is significant comparing with other baselines. As for listwise L2R approaches, ListNet performs the best on this dataset. In terms of the MAP metric, GERank still works very well, and it is the second best model among the nine baseline algorithms. In terms of the P@1 metric, GERank has a significant improvement compared to all the baseline algorithms. In terms of P@3, P@5, and P@10, it does not perform as well as the listwise approach, but it is still comparable to pairwise L2R approaches. Table 4 shows the performance in terms of NDCG@k evaluation metrics with k being set to be 1, 3, 5, and 10. It can be seen that ListNet and Ranking SVM yield the best performance on this dataset. Although ListNet and Ranking SVM outperform our GERank, they will suffer from a mismatch problem when the feature dimension used to represent the document is low. In addition, GERank still yields the best performance in terms of the NDCG@1 evaluation metric. Overall, the performance of our model, GERank, is comparable with or even better than the state-of-the-art pairwise L2R models especially in terms of NDCG@1 and P@1 that evaluate documents ranked higher in the final ranked list of documents in response to a query.

We now examine our GERank model and the baseline models on the OHSUMED dataset.

Table 5. Precision and MAP on the OHSUMED dataset. The best performance per metric among the pairwise L2R algorithms is in boldface. Statistically significant improvement of the performance of our GERank model per metric compared to the best pairwise L2R baseline algorithm is marked in the upper right-hand corner of GERank performance score.

Approach	Model	P@1	P@3	P@5	P@10	MAP
Pointwise	LR	0.5965	0.5768	0.5337	0.4660	0.4220
Listwise	ListNet	0.6524	0.6016	0.5502	0.4970	0.4457
	AdaRank-MAP	0.6338	0.5895	0.5674	0.4970	0.4487
	AdaRank-NDCG	0.6719	0.5984	0.5767	0.5080	0.4498
	SVM${}^{map}$	0.6433	0.5802	0.5523	0.4910	0.4453
Pairwise	Ranking SVM	0.5974	0.5427	0.5319	0.4860	0.4334
	RankBoost	0.5576	0.5609	0.5447	0.4966	0.4411
	SciBERT	0.5682	0.5478	0.5305	0.4872	0.4331
	FRank	0.6429	0.5925	0.5638	0.5010	0.4439
	GERank	${\mathbf{0}.\mathbf{6909}}^{▴}$	${\mathbf{0}.\mathbf{6085}}^{▴}$	0.5412	0.4929	0.4373

shows the performance of the algorithms on precision and MAP metrics. LR, again, has the worse performance on the OHSUMED dataset. Among the pairwise approaches, FRank has a significant improvement in terms of these two metrics. Similar to those reported on the NP2004 dataset, the listwise methods perform very well. In particular, AdaRank-NDCG performs the best. As for our GERank, it behaves as well as the other algorithms in terms of the MAP metric. In particular, GERank achieves the best performance in terms of the metrics P@1 and P@3, which indicates that GERank has an obvious advantage compared to other traditional L2R algorithms when predicting the top documents, especially those ranked within the top three positions. As for the other metrics, the performance of GERank is still comparable with that of other algorithms. Table 5 shows the performance of our GERank and the baseline algorithms on NDCG@k metrics. According to the table, AdaRank-NDCG and FRank yield the best performance among the baselines. In total, listwise approaches perform better than pairwise approaches. For our model, GERank, there is a significant improvement over the best baseline algorithm in terms of all the NDCG metrics on the OHSUMED dataset. Compared with the algorithms based on pairwise approach, the performance of GERank in terms of NDCG metrics has a significant improvement. Similar findings can be found in

Table 6. NDCG on the OHSUMED dataset. The best performance per metric among the pairwise L2R algorithms is in boldface. Statistically significant improvement of the performance of our model per metric compared to the best pairwise L2R baseline algorithm is marked in the upper right-hand corner of performance score.

Approach	Model	NDCG@1	NDCG@3	NDCG@5	NDCG@10
Pointwise	LR	0.4456	0.4426	0.4278	0.4110
Listwise	ListNet	0.5326	0.4732	0.4432	0.4410
	AdaRank-MAP	0.5388	0.4682	0.4613	0.4420
	AdaRank-NDCG	0.5330	0.4790	0.4673	0.4490
	SVM${}^{map}$	0.5229	0.4663	0.4516	0.4319
Pairwise	Ranking SVM	0.4958	0.4207	0.4164	0.4140
	RankBoost	0.4632	0.4555	0.4494	0.4302
	SciBERT	0.4589	0.4472	0.4210	0.4243
	FRank	0.5300	0.4812	0.4588	0.4430
	GERank	${\mathbf{0}.\mathbf{6008}}^{▴}$	${\mathbf{0}.\mathbf{5326}}^{▴}$	${\mathbf{0}.\mathbf{4920}}^{▴}$	${\mathbf{0}.\mathbf{4650}}^{▴}$

,

Table 7. Precision and MAP on the Robust04 dataset. The best performance per metric among the pairwise L2R algorithms is in boldface. Statistically significant improvement of the performance of our GERank model per metric compared to the best pairwise L2R baseline algorithm is marked in the upper right-hand corner of GERank performance score.

Approach	Model	P@1	P@3	P@5	P@10	MAP
Pointwise	LR	0.5212	0.4814	0.4242	0.4317	0.3415
Listwise	ListNet	0.5356	0.5238	0.4913	0.4320	0.3421
	AdaRank-MAP	0.5440	0.5215	0.4871	0.4313	0.3433
	AdaRank-NDCG	0.5455	0.5232	0.4751	0.4237	0.3413
	SVM${}^{map}$	0.5345	0.5173	0.4828	0.4332	0.3421
Pairwise	Ranking SVM	0.5870	0.5721	0.5235	0.4856	0.3761
	RankBoost	0.5742	0.5627	0.5043	0.4722	0.3755
	SciBERT	0.5548	0.5474	0.4925	0.4656	0.3722
	FRank	0.5932	0.5871	0.5525	0.4926	0.3814
	GERank	${\mathbf{0}.\mathbf{6123}}^{▴}$	${\mathbf{0}.\mathbf{6052}}^{▴}$	${\mathbf{0}.\mathbf{5725}}^{▴}$	0.4901	${\mathbf{0}.\mathbf{4018}}^{▴}$

and

Table 8. NDCG on the Robust04 dataset. The best performance per metric among the pairwise L2R algorithms is in boldface. Statistically significant improvement of the performance of our GERank model per metric compared to the best pairwise L2R baseline algorithm is marked in the upper right-hand corner of GERank performance score.

Approach	Model	NDCG@1	NDCG@3	NDCG@5	NDCG@10
Pointwise	LR	0.5212	0.5521	0.6220	0.7322
Listwise	ListNet	0.5356	0.5544	0.6213	0.7251
	AdaRank-MAP	0.5440	0.5647	0.6321	0.7421
	AdaRank-NDCG	0.5455	0.5678	0.6402	0.7452
	SVM${}^{map}$	0.5345	0.5427	0.6398	0.7423
Pairwise	Ranking SVM	0.5870	0.6327	0.6872	0.7824
	RankBoost	0.5742	0.6247	0.6745	0.7743
	SciBERT	0.5548	0.6122	0.6554	0.7532
	FRank	0.5932	0.6472	0.6923	0.7951
	GERank	${\mathbf{0}.\mathbf{6133}}^{▴}$	${\mathbf{0}.\mathbf{6645}}^{▴}$	${\mathbf{0}.\mathbf{7142}}^{▴}$	$0.7887\phantom{▴}$

according to the performance evaluation of both our proposed model GERank and the baseline models on the Robust04 dataset.

According to the analysis based on Table 3, Table 4, Table 5, Table 6, Table 7 and Table 8, it can be concluded that listwise approaches perform well on these datasets. LR is the worse baseline among them. GERank outperforms the baseline algorithms on OHSUMED and has a marked improvement. On the NP2004 dataset, the performance of our GERank model is still comparable with that of other pairwise approaches. Specifically, GERank has a clear advantage in predicting the top documents as indicated by the retrieval performance on the Precision and NDCG metrics. For instance, the best result of the others pairwise approaches on P@1 on NP2004 dataset is 0.5067 for Ranking SVM, whereas our model GERank achieves 0.5734 and obtains about 13.2% improvement over Ranking SVM. On NDCG@1, the best result of the other pairwise approaches on the OHSUMED dataset is 0.5300 for FRank, whereas our method achieves 0.6008 and obtains about 13.4% improvement over the FRank model. As a pairwise L2R algorithm, GERank is able to significantly improve over the traditional pairwise approaches on all the dataset, the NP2004, the OHSUMED, and the Robust04 datasets.

In conclusion, the answers to research questions RQ1 and RQ2 are now clear: GERank outperforms state-of-the-art of traditional pairwise L2R algorithms and also has an improvement on several metrics compared with listwise L2R approaches.

In order to research question RQ3, we analyze the performance and the structure of our model. As shown in Table 3, Table 4, Table 5 and Table 6, we find that GERank can achieve a considerable performance on the datasets. In terms of NDCG and precision metrics, our model has significantly improved the performance compared to the best baseline. Such improvement can be clearly explained as follows: Firstly, our Gaussian Embedding model for Ranking, GERank, co-embed queries and documents into the same semantic space so as to alleviate the mismatch problem between queries and documents. With the constraint in Equation (4), the semantic similarities between query and documents can be effectively measured. Secondly, we employ the square-exponential loss which has an infinite margin and pushes the energy of the negative terms to infinity with exponentially decreasing force [62]. Thus, it is able to retrieve the most relevant documents and rank them within the top positions. It can be explained that the performance on dataset OHSUMED is better than that on dataset NP2004. On the NP2004 dataset, 5 of the 64 features are only dependent on query. However, there are 9 of the 45 features which are only dependent on queries on the OHSUMED dataset. In other words, query has more features on the OHSUMED dataset. For the traditional linear method, they does not use these query information, so the accuracy of prediction is not high on the OHSUMED dataset.

Therefore, the answer to research question RQ3 is now clear: we have clearly explained the mechanism of GERank for document retrieval; see the above.

6.2. Impact of Embedding Dimension

We examine the impact of embedding dimension on the retrieval performance (RQ4). We fix the number of hidden layers I to be 512, and run our model with different embedding dimensions L. In Figure 3 and Figure 4, the horizontal axis is for evaluation metrics, and the vertical axis is for the value of evaluation metrics in detail. From these two figures, we can observe that, when L = 15, the performance of our model shows the worst performance in terms of all the evaluation metrics. It is due to the information loss of documents caused by low dimension embedding. In terms of low dimension embedding, the number of features of the document is too small to cause mismatching.

All metrics as a whole increase with the increase of embedding dimension, as it is shown in Figure 3 and Figure 4. However, the training time will be longer when the embedding dimension is higher. Therefore, increasing the embedding dimension appropriately is beneficial to improve the accuracy of our model GERank. In particular, we can achieve the best performance on OHSUMED dataset when the embedding dimension L is set to be 45. It means that equal dimensional embedding to our model is the most effective on the OHSUMED dataset.

6.3. Effect of Hidden Layer

Finally, we study the influence of hidden layer (RQ5). Specifically, we fix the embedding dimension L to be 45, and run our model with different numbers of hidden layers denoted as I. The retrieval performance with different numbers of hidden layers is shown in Figure 5 and Figure 6. From these two figures, we can observe that, when $I=32$, the performance of our model shows the worst performance on all the evaluation metrics. This is due to the fact that fewer numbers of neurons results in the poor representation ability of the overall model.

According to Figure 5 and Figure 6, we can also see that the performance of our model improves as the hidden layers increase. However, increasing the number of neurons may lead to the risk of over-fitting. For instance, the performance of our model with the number of neurons being 1024 is worse than 512. We show that 512 is the best choice of the number of neurons in our model on the OHSUMED dataset.

In conclusion, the dimension embedding and the number of hidden layer neurons do effect the performance of our model. In the case of embedding dimension being 45 and hidden layer being 512, our model achieves the best performance on all the evaluation metrics on the OHSUMED dataset.

7. Conclusions

In this paper, we have studied the core problem in information retrieval, i.e., designing a L2R model for document retrieval and optimizing the model to produce a final rank list of documents in response to a given query. To obtain better document retrieval performance, we have proposed a novel Gaussian embedding based pairwise L2R model, GERank, to learn to infer the embeddings and their covariances of queries and documents such that the similarities between a query and a document can be effectively measured for ranking documents in response to a given query. Specifically, our GERank model is able to co-embed queries and documents into the same semantic space such that the mismatch between queries and documents can be alleviated. GERank is the first model to incorporate embedding techniques into pairwise L2R algorithms with constraints. For example, in Gaussian embedding space, we transform the ranking problem into pairwise constraints. To yield better performance, GERank not only infers the embeddings of queries and documents, but also their covariances (uncertainty is captured by GERank) such that the similarities between queries and documents can be effectively measured, via an energy-based loss function being defined for learning the embeddings and covariances of queries and documents.

We have conducted experiments on two LETOR datasets. Experimental results show our model has a significant improvement over the datasets. Our evaluation results have also shown that GERank outperforms state-of-the-art traditional pairwise algorithms and even some state-of-the-art listwise algorithms on the OHSUMED dataset. Specifically, our model is able to yield better retrieval performance within the top-k document. We also systematically study the influence of embedding dimension and size of hidden layers in the model and provide a reasonable explanation.

As for future work, we aim to improve the proposed model, GERank, in the following ways:

(a)

We plan to infer embeddings and the covariances of queries and documents via other embedding techniques such as variational auto-encoders [71].

(b)

We plan to apply GERank to other information retrieval applications such as: given a question, rank answers that are relevant to the question in question-answering communities, where users’ interaction historical information in the communities can be utilized to boost the retrieval performance.

(c)

We intend to integrate our GERank into listwise L2R approaches, as listwise approaches have shown their better performance compared to that produced by the pairwise L2R approaches on the testing datasets.

(d)

We also intend to improve big data law for a number of information retrieval applications such as document retrieval.