Analysis of Thurstone-Motivated Models in the Case of Non-Evaluable Data: Methods for Extracting Information

Abstract

We study Thurstone-motivated paired comparison models from the perspective of data evaluability, focusing on cases where datasets cannot be directly evaluated. Despite this limitation, such datasets may still contain extractable information. Three main strategies are known in the literature: increasing the number of options, inserting artificial data through perturbation methods, and requesting new real comparisons. We propose a new approach closely related to the latter two. We analyze the structure of the data and introduce the concept of the optimal limit point, related to the supremum of the log-likelihood function. We prove a theorem for determining optimal limit points based on the data structure, which characterizes the information content of the available dataset. We also prove a theorem linking optimal limit points to the limiting behavior of evaluation results obtained via perturbations, thereby explaining why different perturbation methods may yield different outcomes. In addition, we propose a new perturbation method that adds the minimum possible amount of artificial data. Furthermore, the method identifies the most informative object pairs for new real comparisons, enabling a full evaluation of the dataset.

1. Introduction

Deep learning is among the most popular evaluation methods. Excellent surveys that detail both the methods and their applications include, for example, [1,2,3]. The range of applications is extremely broad, including image processing [4], speech recognition [5], traffic prediction [6], healthcare [7], recommender systems [8], and weather forecasting [9], among others. A common feature of these methods is the requirement for relatively large amounts of data to ensure reliable performance.

Despite the widespread use of neural networks, relatively few papers have been published on their application to ranking problems [10]. In [11], the authors compared the performance of paired (pairwise) comparison methods and a deep learning approach in the context of the top European football leagues. They concluded that, when ranking the leagues separately, all paired comparison methods and the neural network performed well. However, for the aggregated ranking, the deep learning method was limited due to the small amount of available data, whereas the paired comparison methods remained effective. Although deep learning methods hold fantastic potential and their development is explosively fast, at the current stage of research, they cannot replace specialized evaluation methods in cases of a small amount of data.

Pairwise comparisons are common in practical applications when the amount of data is limited. They are frequently applied in decision making [12,13], education [14,15], politics [16,17], healthcare [18,19], psychology [20,21], economy [22], social sciences [23,24], architecture [25], and even in sports performance evaluation [11,26]. One can find numerous further applications in the new book by Mazurek [27] as well.

The most frequently used paired comparison method is based on pairwise comparison matrices (PCMs). The most commonly used variant is the analytic hierarchy process [28,29]. In this method, the verbal decisions are converted into the numbers 1, 3, 5, 7, 9, and their reciprocals, and the resulting reciprocal symmetric PCM is evaluated by the eigenvector method [30] or by the logarithmic least squares methods [31]. In the methods based on the PCMs, one of the key issues is the consistency/inconsistency of the data. Various methods have been developed to measure inconsistency [32,33], and reducing inconsistency is also the subject of many publications [27,34].

Another branch of paired comparison models has a stochastic background. These models are inspired by Thurstone’s idea [35]; we call them Thurstone-motivated models. The original assumptions have been generalized in many ways: the distribution of the latent random variables [36,37,38]; the number of options that can be chosen in decisions [39,40,41,42]; incorporation of home-field advantage [43,44]; consideration of time effects [45]; and so on. The model parameters are estimated based on the comparison results. The most preferable parameter estimation method is the maximum likelihood estimation (MLE) method, which has favorable asymptotic properties and additional benefits, such as constructing confidence intervals and testing hypotheses [46]. However, when applying maximum likelihood estimation, it is necessary to determine whether the likelihood function attains its maximum value and whether the maximizing argument is unique. This requires analyzing the likelihood function—or its logarithm—from the perspective of mathematical analysis, as done in [47,48]. If the maximum value is attained and the argument is unique, we obtain a unique strength value for each object and a unique ranking based on the comparison data. We refer to this case as evaluable data.

Nevertheless, sometimes there are cases when the data derived from pairwise comparisons cannot be evaluated, meaning that the strengths of individual objects to be evaluated cannot be determined definitively based on the data. From a mathematical perspective, this occurs when the maximum value of the likelihood function is not attained, or when it is attained at multiple arguments. We refer to this scenario as having non-evaluable data. However, even in such cases, the data contain information that we would like to extract. For example, we might possess relevant information about the relative positions of certain objects within a ranking or rating. To ensure evaluability, additional information must be introduced into the system. There are three different approaches to achieve this: modifying the model for another option number, inserting artificial comparison results, or requesting additional real comparisons and incorporating their results into the data.

The first case—that is, changing the number of options—can sometimes be implemented, while in other cases it cannot. For example, if we are evaluating the results of a sports tournament, it may be feasible to increase or decrease the option number. If we distinguish between ‘much better’ and ‘slightly better’ options instead of simply ‘better’; we introduce additional information. When evaluating opinions gathered through questionnaires, the possible responses determine the maximum number of options. The number of options cannot be increased; although it can be decreased, this does not improve evaluability. The potential method of ensuring evaluability by changing the number of options is analyzed in [49].

The second case—i.e., making artificial data supplements—is called data perturbation. It was considered in the papers of Conner and Grant [50] and Yan [51] in the case of two- and three-option Bradley–Terry models. The original data do not contain all the information about the complete system, but by making the data evaluable, some of the information (e.g., partial information about ranking—about subsets) can be extracted. The insertion of new data is intended to address this task. Data modification can be performed in several ways. It is a natural requirement to make as few modifications as possible. However, this approach only allows for limited information retrieval. As it turns out, by analyzing the structure of the dataset, we can still extract information about some of the entities. However, for other entities, sometimes nothing definitive can be stated.

Although [50,51] proved the convergence of the evaluation results obtained by their perturbation methods, they did not provide the values of the limits. Moreover, different perturbation techniques may yield different results, even in their limits. This is surprising, as the log-likelihood function was modified only slightly in each case. It is also unsettling: What causes these different limits? The reason for this phenomenon has not yet been explored. Under what conditions do different perturbation methods produce the same or different evaluation results in their limits? What information can be extracted from the available dataset?

To address this, we investigate the supremum rather than the maximum of the log-likelihood function and introduce the concept of the optimal limit point. Based on a detailed analysis of the data structure, we determine the optimal limit point(s). These, together with the data structure, capture all information contained in the available dataset. We clarify the conditions under which the optimal limit point is unique and when it is not. Our results apply to a wide class of distributions for the differences of the latent random variables. Moreover, we prove a theorem that establishes a connection between optimal limit points and perturbation methods, thereby explaining why different perturbation methods may lead to different results in their limits.

The third case, to enlarge the data set, i.e., requesting further real comparison results, is the most natural approach, but it is sometimes impossible. In cases of completed, previously conducted studies, or even the evaluation of sports championships, obtaining new data is not always possible. If it is still possible to obtain additional data, it matters how many comparisons we request. Moreover, it is very useful to know which pairs are worth comparing. The structural analysis of the available data and the elaborated new perturbation algorithm also answer this problem.

In addition to generalizing the scope of applicability of previously published perturbation methods, this paper presents four main contributions in information extraction in case of non-evaluable data: (a) analyzing the data structure; (b) introducing the concept of the optimal limit point, which characterizes the information contained in the dataset; (c) establishing a connection between this new concept and perturbation methods; and (d) developing a new algorithm for determining the optimal supplements in terms of inserted edges or additional comparisons.

The structure of the paper is as follows: Section 2 summarizes the model under investigation and presents the necessary and sufficient condition for data evaluability. Section 3 contains the analysis of the data structure. On the one hand, it is the basis of the theoretical results in Section 6, containing the determination of the optimal limit points and theoretical results between the perturbation methods and the optimal limit points. On the other hand, data structure analysis the basis of a new special perturbation method, named S. Section 4 presents the perturbation methods known from the literature, along with a simple modification (M) and a new algorithm (S) based on the structural analysis of the data. Section 5 presents some interesting special examples of different data structures, along with perturbation algorithms. Section 7 contains numerical results related to different methods of data supplementation, highlighting the properties of the individual methods. In Section 8, the results of a tennis tournament are analyzed, serving as an interesting application of the findings presented in the paper. Finally, the paper concludes with a short summary.

2. The Investigated Model

Let us denote the objects to be evaluated by the numbers $1,2,\dots ,n$. The Thurstone-motivated models have a stochastic background; it is assumed that the current performances of the objects are random variables denoted by ${\xi }_i$, for $i=1,2,\dots ,n$, with the expectations $E\left({\xi }_i\right)=m_i$. These expectations are the strengths of the objects. Decisions comparing two objects i and j are related to the difference between these latent random variables, i.e., ${\xi }_i-{\xi }_j$. We can separate the expectations as follows:

$${\xi }_i-{\xi }_j=m_i-m_j+{\eta }_{i,j},$$

(1)

where ${\eta }_{i,j}$ are independent identically distributed random variables with the common cumulative distribution function (c.d.f.) F. F is assumed to be three times continuously differentiable, where $0<F<1$. The probability density function of F is $0<f$, which is symmetric to zero (i.e., $f(-x)=f\left(x\right)$, $x\in \mathbb{R}$). Moreover, the logarithm of f is strictly concave. The set of these c.d.f.s is denoted by $\mathbb{F}$. Logistic distribution and Gaussian distribution belong to this set, among many other distributions. If ${\eta }_{i,j}$ is Gaussian-distributed, then we call it the Thurstone model [35]. If the distribution F is logistic, i.e.,

$$F\left(x\right)={\displaystyle \frac{1}{1+exp(-x)}},x\in \mathbb{R},$$

(2)

we speak about the Bradley–Terry model [36].

We note that the assumption of a common cumulative distribution function F is usual in the literature. From a mathematical point of view, one could assume a specific c.d.f. $F_{i,j}$ for each pair $(i,j)$, with the only requirement being that $F_{i,j}\in \mathbb{F}$. Such a model is investigated in [37].

The simplest model in terms of structure is the two-option model. In the case of the two-option model, the differences are compared to zero: the decision ‘better’/‘worse’ (relating i to j) indicates whether the difference (1) is positive or negative, respectively. The set of real numbers $\mathbb{R}$ is divided into two disjoint parts by the point zero. If the difference of the latent random variables  ${\xi }_i-{\xi }_j$ is negative, then the object i is ‘worse’ than object j, while if  $0\le {\xi }_i-{\xi }_j$, then i is ‘better’ than j. Denote the possible options by $C_k$, $k=1,2$. Figure 1 represents the intervals of the set of real numbers belonging to the decisions.

If only two options are allowed, then the probabilities of ‘worse’ and ‘better’ can be expressed as

$$p_{i,j,1}=P({\xi }_i-{\xi }_j<0)=F(0-(m_i-m_j))=F(m_j-m_i)$$

(3)

and

$$p_{i,j,2}=P(0\le {\xi }_i-{\xi }_j)=1-F(0-(m_i-m_j))=1-F(m_j-m_i)=F(m_i-m_j),$$

(4)

respectively.

The data are included in a three-dimensional ($n\times n\times 2$) data matrix A. Its elements $A_{i,j,k}$ represent the number of comparisons in which decision $C_k$$(k=1,2)$ is the outcome when objects i and j are compared. In detail, $A_{i,j,1}$ is the number of comparisons when object i is ‘worse’ than object j, while $A_{i,j,2}$ is the number of comparisons when object i is ‘better’ than object j. Of course, $A_{i,j,1}=A_{j,i,2}$ for $i,j=1,\dots ,n,i\ne j$, and $A_{i,i}=0$. Assuming independent decisions, the likelihood function is

$$L\left(A\right|m_1,\dots ,m_n)=\prod _{k=1}^2\prod _{i=1}^{n-1}\prod _{j=i+1}^n{p}_{i,j,k}^{A_{i,j,k}}.$$

(5)

The log-likelihood function, denoted by $LogL$, is the logarithm of the above and is given by

$$LogL\left(A\right|m_1,m_2,\dots ,m_n)=\sum _{k=1}^2\sum _{i=1}^{n-1}\sum _{j=i+1}^n{A}_{i,j,k}·log\left(p_{i,j,k}\right)=\sum _{k=1}^2\sum _{i,j=1,i\ne j}^{n}0.5·A_{i,j,k}·log\left(p_{i,j,k}\right).$$

(6)

One can see that the likelihood Function (5), and also the log-likelihood Function (6), depend only on the differences of the parameters $m_i$; hence, one parameter (or the sum of all parameters) can be fixed. Let us fix $m_1=0$. The maximum likelihood estimation of the parameters $\underline{m}=(m_1,\dots ,m_n)$ is the argument at the maximum value of (5), or equivalently, of (6); that is,

$$\widehat{\underline{m}}\left(A\right)=\underset{\underline{m}\in {\mathbb{R}}^n,m_1=0}{argmax}LogL\left(A\right|\underline{m}).$$

(7)

Introduce notation

$${\pi }_i={\displaystyle \frac{exp\left(m_i\right)}{{\sum }_{l=1}^{n}exp\left(m_l\right)}}.$$

(8)

We will refer to them as weights to distinguish this exponential form from the original one. If the distributions of the differences are logistic, then Equations (3) and (4) can be written in the following equivalent form:

$$p_{i,j,1}={\displaystyle \frac{{\pi }_j}{{\pi }_i+{\pi }_j}},$$

(9)

$$p_{i,j,2}={\displaystyle \frac{{\pi }_i}{{\pi }_i+{\pi }_j}}.$$

(10)

The log-likelihood function is

$$\begin{array}{c}\hfill LogL\left(A\right|{\pi }_1,\dots ,{\pi }_n)=\sum _{i=1}^{n-1}\sum _{j=i+1}^n{A}_{i,j,1}·(log\left({\pi }_j\right)-log\left({\pi }_i+{\pi }_j\right))+\\ \hfill +\sum _{i=1}^{n-1}\sum _{j=i+1}^n{A}_{i,j,2}·(log\left({\pi }_i\right)-log({\pi }_i+{\pi }_j)),\end{array}$$

(11)

which has to be maximized under the conditions

$$0<{\pi }_i,i=1,2,\dots ,n\mathrm{and}\sum _{i=1}^n{\pi }_i=1.$$

(12)

Explicitly, the maximum likelihood estimate of the parameter vector $\underline{\pi }$ is given by

$$\widehat{\underline{\pi }}\left(A\right)=\underset{\underline{\pi }\in {\mathbb{R}}_{+}^n,{\sum }_{i=1}^n{\pi }_i=1}{argmax}LogL\left(A\right|\underline{\pi }).$$

(13)

This is the standard form of the two-option Bradley–Terry model [36]. We note that the maximum can also be found under the constraint ${\pi }_1=1$.

To find the maximum value of Function (6) or (11), it is not necessary for $A_{i,j,k}$ to be integer-valued. This fact will be used in the perturbation step when the original data matrices A are transformed into matrices with nonnegative elements.

We note that every additive term in (6) and (11) is negative, or, if $A_{i,j,k}=0$, is zero. Therefore, the log-likelihood function itself can only take negative values under the constraint $m_1=0$ or (12), respectively. The log-likelihood function has a finite supremum, which can be negative or zero. The limits of some terms in (6) and (11) can be zero if $m_i\to -\infty$ for some indices i, or, equivalently, if ${\pi }_i\to 0$ for some indices i, respectively.

In this parameter estimation method, the key issue is the existence of the maximum value and the uniqueness of its argument. If the maximum is not attained, we cannot provide the estimated values. If the argument is not unique, we cannot determine which of the values should be chosen as the estimate. The existence and the uniqueness are guaranteed by the Theorem 1, proved in [47], for the Bradley–Terry model and in [48] for general distributions. To present it, let us first introduce the following definitions.

Definition 1 

(Graph of comparisons $CG\left(A\right)$ defined by data set A).  Let the objects to be evaluated be the nodes of the graph $CG\left(A\right)$. There exists an undirected edge between nodes i and j ($i-j$) if $0<A_{i,j,1}+A_{i,j,2}$.

Definition 2 

(Directed graph $DG\left(A\right)$ defined by data set A).  Let the objects to be evaluated be the nodes of the graph $DG\left(A\right)$. There exists a directed edge between nodes i and j, directed from i to j ($i\to j$), if $0<A_{i,j,2}$.

It may happen that there is a directed edge from i to j and also a directed edge from j to i.

We note that if the graph $CG\left(A\right)$ is not connected, then the data set A is non-evaluable, since the maximum, if it is attained, can be reached at multiple arguments. Even if $CG\left(A\right)$ is connected, it is still possible that A is non-evaluable; see Examples 1–3. We now present the criterion for data evaluability.

Theorem 1 

([48]). Let $F\in \mathbb{F}$. The log-likelihood Functions (6) and (11) attain the maximum value and its argument is unique if and only if the directed graph $DG\left(A\right)$ is strongly connected.

If the graph $DG\left(A\right)$ is strongly connected, then the graph $CG\left(A\right)$ is also connected. In the case of a two-option model, there are two possible reasons why the data set may not be evaluable: (1) the graph $CG\left(A\right)$ is not connected, meaning that there are separate components that are not compared with other parts; or, (2) there exists an object, or a subset of objects, that has only one-way comparison results—either always ‘better’ or always ‘worse’—with respect to all other objects or subsets. These issues are examined and analyzed in Section 3.

3. Data Structure Analysis

First, consider the connected subgraphs of $CG\left(A\right)$ that cannot be enlarged. These subgraphs will be called maximal connected subgraphs. These will be denoted by $C{\left(CG\left(A\right)\right)}_1,\dots ,C{\left(CG\left(A\right)\right)}_{NC\left(A\right)}$, where $NC\left(A\right)$ is the number of the maximal connected subgraphs of $CG\left(A\right)$. It is clear that there is no edge between the nodes belonging to different maximal connected subgraphs, and $1\le NC\left(A\right)\le n$. If $NC\left(A\right)=1$, then the graph $CG\left(A\right)$ is connected. If $1<NC\left(A\right)$, there is no comparison between the elements of the different maximal connected subgraphs; therefore, there is no information about how the strengths of the objects in different subgraphs relate to each other. Yan did not address this case. We also mainly focus on the case where $NC\left(A\right)=1$.

Second, consider the strongly connected components of the maximal connected subgraphs—that is, those strongly connected subgraphs that cannot be enlarged. Let u denote the index of a maximal connected subgraph, where $1\le u\le NC\left(A\right)$. Furthermore, let the strongly connected components of $C{\left(CG\left(A\right)\right)}_u$ be denoted by $S{\left(C{\left(CG\left(A\right)\right)}_u\right)}_v$, where $1\le v\le N{S}_u$, and $N{S}_u$ denotes the number of strongly connected components of the maximal connected subgraph $C{\left(CG\left(A\right)\right)}_u$. We define a ’supergraph’ corresponding to $C{\left(CG\left(A\right)\right)}_u$ based on the edges between nodes in different strongly connected components. This supergraph characterizes the relationships among the strongly connected components of $C{\left(CG\left(A\right)\right)}_u$. Essentially, we merge the nodes within each component and remove the edges between them. The edges between the nodes of the supergraph are then defined corresponding to the edges between the nodes in the different strongly connected components, as if merging them too. The supergraph characterizes the relations among the strongly connected components of $C{\left(CG\left(A\right)\right)}_u$. For the sake of simplicity, we denote the maximal connected subgraphs $C{\left(CG\left(A\right)\right)}_u$ by X.

Definition 3 

(The supergraph characterizing the directed connections in a maximal connected subgraph). Let X be a maximal connected subgraph of $CG\left(A\right)$, and let $DG\left(X\right)$ be the directed graph consisting of the nodes in X and the original directed edges between them. Denote the strongly connected components of $DG\left(X\right)$ by $S{\left(X\right)}_1$, …, $S{\left(X\right)}_{NS}$, where $NS=NS\left(X\right)$ is the number of strongly connected components in $DG\left(X\right)$. The supergraph $SUP\left(X\right)$, corresponding to X, consists of $NS$ nodes, referred to as super-nodes. Two super-nodes a and b (with $a\ne b$) are connected by a directed super-edge from a to b if there exists an object $i\in S{\left(X\right)}_a$ and $j\in S{\left(X\right)}_b$ such that $0<A_{i,j,2}$. The edge from a to b is called an outgoing edge from a and an incoming edge to b.

We make the following observations:

A

The connection between two super-nodes can only be one-way.

If there are multiple edges between the elements of different strongly connected components, then they must all have the same direction. If directed edges exist in both directions between two super-nodes, then there would exist elements in the two strongly connected components such that $i_1,i_2\in S{\left(X\right)}_a$ $j_1,j_2\in S{\left(X\right)}_b$, with $0<A_{i_1,j_1,1}$ and $0<A_{i_2,j_2,2}$. This implies that $a=b$, which contradicts the assumption that $S{\left(X\right)}_a$ and $S{\left(X\right)}_b$ are distinct strongly connected components.

B

The supergraph does not contain a directed cycle; that is, it is acyclic.

If there was a directed cycle in the supergraph, let us denote it by $a_1,a_2,\dots ,a_h,a_1$, with $2\le h$. Find those object pairs of the form $(i_{a_j}^{\left(out\right)},i_{a_{j+1}}^{\left(in\right)})$ between which there exist directed edges in $DG\left(A\right)$ that generate the edges of the supergraph. This means that $i_{a_j}^{\left(in\right)}$ and $i_{a_j}^{\left(out\right)}$ are elements of the strongly connected component corresponding to the super-node $a_j$. The upper indices ‘(out)’ and ‘(in)’ indicate the direction of the edges; moreover, $0=A_{i_{a_j}^{\left(out\right)},i_{a_{j+1}}^{\left(in\right)},1}$, and $0<A_{i_{a_j}^{\left(out\right)},i_{a_{j+1}}^{\left(in\right)},2}$. In each strongly connected component, there is a directed path between any two elements; in particular, $i_{a_j}^{\left(in\right)}$ and $i_{a_j}^{\left(out\right)}$ for $j=1,\dots ,h$ are connected via a directed edge. Consequently, starting from $i_{a_1}^{\left(out\right)}$, we can find a directed cycle in $DG\left(A\right)$ that includes objects from different strongly connected components, which leads to a contradiction.

C

The supergraph is not strongly connected. If we omit the directions of the edges between the super-nodes of the supergraph and we replace the directed edges with undirected ones, the obtained undirected graph is connected through these undirected edges.

If the supergraph was strongly connected, it would contain a cycle. The connected property is the consequence of the connectedness of X.

D

There exists at least one super-node in the supergraph that has only an outgoing super-edge.

Assume the opposite of the statement. This means that every super-node has either only incoming super-edges or both incoming and outgoing super-edges. It is not possible that all super-nodes have only incoming super-edges, since each incoming super-edge must also be an outgoing super-edge for another super-node.

If every super-node has both incoming and outgoing super-edges, then start from such a super-node and find the super-node from which it has an incoming super-edge. This new super-node must also have an incoming super-edge. This is because, by assumption, the set of super-nodes with only outgoing super-edges is empty, and the current node cannot belong to the set of super-nodes with only incoming super-edges, since it has an outgoing super-edge.

Repeat this process until the set of selectable super-nodes is exhausted, i.e., you must choose a super-node that has already appeared, thereby forming a directed cycle—again contradicting condition C. Hence, the original statement D must be true.

Now, we can define the levels of the super-nodes of the supergraph by the following algorithm. Level 0, called top level, contains the super-nodes that have only outgoing edges. We note that there are no connections between these super-nodes because they all have only outgoing edges. Now, remove these super-nodes from the supergraph along with their outgoing edges. If there are super-nodes that have not yet been classified into a level, there must be super-nodes in the remaining part of the supergraph that have no edges or have only outgoing edges. If the $(i-1)$th level (for $i=1,\dots$) has been constructed, then these latter super-nodes will form the ith level. The process continues until all super-nodes are classified into a level.

If there is more than one maximal connected subgraph, then each of them gives rise to a separate supergraph. These supergraphs are disconnected from one another.

Examples illustrating the levels in the supergraph are contained in Section 5.

A very similar levelization process, applied in genetics, is presented in [52].

4. Perturbation Methods

4.1. Previously Defined Perturbation Methods and a Modification

First, Conner and Grant published a paper [50] dealing with the idea that every data set can be evaluable if every comparison data $A_{i,j,k}$ is increased by a $0<\epsilon$ value. In this case, the perturbed data matrix will be denoted by $CA\left(\epsilon \right)$, and its elements—the perturbed data—are as follows:

Definition 4 

(Perturbation by Conner and Grant denoted by C). 

$$C{A}_{i,j,k}\left(\epsilon \right)=A_{i,j,k}+\epsilon \mathit{for}\mathit{every}i\ne j\mathit{and}\mathrm{k}=1,2.$$

(14)

Regardless of the data matrix A, perturbation C inserts $n·(n-1)$ data. Now, we can formulate the following statement.

Theorem 2. 

Let $F\in \mathbb{F}$. Then, the perturbed data matrix $CA\left(\epsilon \right)$ is evaluable.

Proof. 

Regardless of the matrix A, the graph of connection $CG\left(CA\right(\epsilon \left)\right)$ is a complete comparison graph, and $DG\left(CA\right(\epsilon \left)\right)$ is a complete directed comparison graph. Recalling Theorem 1, the statement is obvious.    □

We note that Conner and Grant proved this statement only in the case of the Bradley–Terry model, but it is also true for general $F\in \mathbb{F}$. Yan modified the method in [51] so that additions were made only for pairs that were originally compared.

Definition 5 

(Perturbation given by Yan denoted by Y). 

$$Y{A}_{i,j,k}\left(\epsilon \right)=\left\{\begin{array}{c}{A}_{i,j,k}+\epsilon ,\mathit{if}0<A_{i,j,1}+A_{i,j,2},k=1,2\hfill \\ 0,\mathit{otherwise}\hfill \end{array}\right.$$

(15)

We can observe that this method does not make every data matrix A evaluable. Yan proved the following statement only for the Bradley–Terry model: The perturbed data matrix $YA\left(\epsilon \right)$ is evaluable if and only if $CG\left(A\right)$ is connected. This statement also holds for the case where $F\in \mathbb{F}$.

Theorem 3. 

Let $F\in \mathbb{F}$. The perturbed data matrix $YA\left(\epsilon \right)$ is evaluable if and only if $CG\left(A\right)$ is connected.

Proof. 

Perturbation Y makes every connection in $CG\left(A\right)$ into directed connections in both directions in $DG\left(A\right)$. If $CG\left(A\right)$ is connected, then there is an undirected path between any objects i and j; therefore, there is a directed path from i to j in $DG\left(YA\right(\epsilon \left)\right)$. If there is no path between i and j, where $i,j\in CG\left(A\right)$, then there is no directed path in both directions between them in $DG\left(YA\right(\epsilon \left)\right)$; therefore, $DG\left(YA\right(\epsilon \left)\right)$ is not strongly connected, i.e., $YA\left(\epsilon \right)$ is not evaluable.    □

This method also may insert unnecessary data; if there is a directed edge from i to j, and also from j to i, the added $\epsilon$ value does not improve evaluability.

To remove some unnecessary supplements from the perturbed matrix constructed by Yan, we can define the following perturbation method.

Definition 6 

(Modified Yan perturbation method denoted by M). 

$$M{A}_{i,j,k}\left(\epsilon \right)=\left\{\begin{array}{c}{A}_{i,j,k}+\epsilon ,k=1,2;\mathit{if}0<A_{i,j,1}+A_{i,j,2}\mathit{and}{A}_{i,j,1}·A_{i,j,2}=0\hfill \\ A_{i,j,k}\mathit{otherwise}.\hfill \end{array}\right.$$

(16)

The condition $0<A_{i,j,1}+A_{i,j,2}$, and $A_{i,j,1}·A_{i,j,2}=0$, expresses that there is at least one comparison between objects i and j, and all such comparisons are either consistently ‘better’ or consistently ‘worse’. In the case of perturbation method M, the amount of insertion is less than or equal to that in method Y. Now, we can state the following.

Theorem 4. 

Let $F\in \mathbb{F}$. The perturbed matrix $MA\left(\epsilon \right)$ can be evaluated if and only if the graph $CG\left(A\right)$ is connected.

Proof. 

Perturbation M makes additions if the compared pair has only one-directed connections; then, every connection becomes a directed edge in both directions in the graph $DG\left(MA\right(\epsilon \left)\right)$. As $CG\left(A\right)$ is connected, $DG\left(MA\right(\epsilon \left)\right)$ is strongly connected; therefore, $MA\left(\epsilon \right)$ is evaluable. If $CG\left(A\right)$ is not connected, then $DG\left(MA\right(\epsilon \left)\right)$ is not strongly connected; therefore, $MA\left(\epsilon \right)$ cannot be evaluated.    □

It is easy to see that the perturbation method M might insert perturbations even in cases where they are not necessary. Therefore, we aimed to develop an algorithm that adds supplements only when needed. To achieve this, we used the data structure analysis of the data matrix A.

4.2. Algorithm for Perturbation Based on Data Structure

Now, we can construct a new perturbation algorithm based on the data structure. It will be called the S algorithm. The steps for the algorithm are the following:

I.

Data structure analysis: Identify all maximal connected subgraphs of the graph $CG\left(A\right)$.

II.

Data structure analysis: Determine the strongly connected components of each maximal connected subgraph.

III.

Data structure analysis: construct the supergraph of each maximal connected subgraph. Determine the levels and identify the nodes that are placed at the top level. Then, identify the objects belonging to the super-nodes located at top level.

IV.

Evaluation of subsets: Evaluate each strongly connected component at the top level based exclusively on the data between its elements.

V.

Connection of the strongly connected components in the maximal connected subgraphs: For each leveled supergraph, construct two chains of unidirectional edges. An $\epsilon$-perturbation connects the element with the highest $\widehat{\pi }$-coordinate in the ith strongly connected component to the element with the lowest $\widehat{\pi }$-coordinate in the $(i+1)$th strongly connected component. The first chain contains the top level strongly connected components. The second chain contains those strongly connected components that do not have any outgoing edges. Arrange them in decreasing order according to their level. Finally, add an $\epsilon$-perturbation from the strongest object of last element of the second chain to the weakest object of the first element of the first chain. If the coordinates are not uniquely determined, choose among the appropriate ones the lowest number. Data modification involves the following: If the set of inserted directed edges is denoted by $INS$, we have

$$S{A}_{i,j,k}\left(\epsilon \right)=\left\{\begin{array}{c}\epsilon ,\mathrm{if}(i,j)\in INS\mathrm{and}k=2,\hfill \\ A_{i,j,k},\mathrm{otherwise}.\hfill \end{array}\right.$$

(17)

VI.

Evaluation of the maximal connected subgraphs: Step V makes every maximal connected subgraph evaluable; evaluate them based exclusively on the data between the elements within each subgraph.

VII.

Connection of the maximal connected subgraphs: Connect the maximal connected subgraphs with a chain of unidirectional $\epsilon$ perturbations into a cycle such that the element with the highest $\widehat{\pi }$-coordinate in the ith maximal connected subgraph has a perturbation to the element with the lowest $\widehat{\pi }$-coordinate in the $(i+1)$th maximal connected subgraph.

Theorem 5. 

The algorithm consisting of Steps I–VII renders any data set A evaluable in a finite number of steps. The original data set A cannot be made evaluable with less than the number of additions in the algorithm.

Proof. 

Since for any data set A, the graph $CG\left(A\right)$ has finitely many maximal connected subsets and finitely many strongly connected components, Steps I–VII terminate after a finite number of iterations. In Step I, the disjoint set union algorithm can be used. In Step II, Tarjan’s algorithm [53] can be used. In Step III, an algorithm presented at the bottom of Section 3, which assigns the super-nodes with no incoming super-edges to the top level, can be used.

The second part of the theorem is easy to see.    □

Note that Step VII can be applied in the case of the Y and M algorithms to connect the maximal connected subgraphs if $CG\left(A\right)$ is not connected.

In order to make it easier to reproduce the aforementioned S perturbation algorithm, we provide a flowchart and a pseudocode of the algorithm. The pseudocode is presented in Algorithm 1. The flowchart can be seen in Figure 2.

Algorithm 1 The pseudocode of the S perturbation.

1: procedure S-Perturbation(graph)  2:       Let maximal_ connected_components ← all maximal_connected components of graph  3:       for all component in maximal_ connected_components do  4:             Find all strongly connected components ($SCC$s) of component  5:             Build a supergraph where each node is an $SCC$  6:             Assign levels to the $SCC$s in the supergraph (top to bottom)  7:             for all $SCC$ with no incoming or no outgoing superedges in the supergraph do  8:                  Evaluate the $SCC$  9:             end for 10:             Sort these evaluated $SCC$s in decreasing order of level (top to bottom) 11:             for each pair of consecutive $SCC$s in this order do 12:                  Add $\epsilon$-perturbation from the best-ranked element of the current $SCC$ 13:                        to the worst-ranked element of the next $SCC$ 14:             end for 15:       end for 16:       for all component in maximal_connected_components do 17:             Evaluate the component 18:       end for 19:       Order the maximal_connected components by index 20:       for each pair of consecutive components (in cyclic order) do 21:             Add $\epsilon$-perturbation from the best-ranked element of the current component 22:                   to the worst-ranked element of the next component 23:       end for 24: end procedure

Comparing the C, Y, and M perturbation methods to S, we can state the following. While the former algorithms insert edges based on certain values in the data—often unnecessarily—the S algorithm, after analyzing the structure of the entire dataset, inserts only those edges that are strictly necessary to ensure evaluability. As we will see later, this leads to faster convergence to the optimal limit point and significantly reduces the number of artificial edges.

5. Examples

We present some simple data structures that are worth demonstrating because they are easy to follow and understand. The first example is as follows.

Example 1. 

The number of objects equals three. There is a comparison between any two objects. Moreover,

$$\begin{array}{c}\hfill A_{1,2,1}=0,A_{1,2,2}=1,A_{1,3,1}=0,A_{1,3,2}=1,A_{2,3,1}=0,A_{2,3,2}=1.\end{array}$$

(18)

The graph of Example 1 and the supergraph belonging to it can be seen in Figure 3.

The ranking of the objects intuitively is clear: object 1 is the best, then object 2 is the next, and finally, object 3 follows. Recalling Theorem 1, it is clear that (18) is non-evaluable. To gain deeper insight, we now examine the log-likelihood Function (6) in detail under the condition $m_1=0$.

$$LogL\left(A\right|0,m_2,m_3)=log(1-F\left(m_2\right))+log(1-F\left(m_3\right))+log(1-F(m_3-m_2)).$$

(19)

Let

$$\begin{array}{c}\hfill Z^{(0,1)}(A,0,m_2,m_3)=log(1-F\left(m_2\right)),\\ \hfill Z^{(0,2)}(A,0,m_2,m_3)=log(1-F\left(m_3\right)),\\ \hfill Z^{(1,2)}(A,0,m_2,m_3)=log(1-F(m_3-m_2)).\end{array}$$

(20)

Introducing $m_1^{\prime }=0$, $m_2^{\prime }=m_2-x$ and $m_3^{\prime }=m_3-2·x$, we have

$$\underset{x\to \infty }{lim}{Z}^{(s_1,s_2)}(A,0,m_2^{\prime },m_3^{\prime })=0,s_1<s_2,$$

(21)

strictly monotonically increasing if x moves strictly monotonically increasing. Although the maximal value is not attained, the log-likelihood Function (19) can get arbitrarily close to zero, so its limit is equal to its supremum. The boundary point corresponding to the supremum is ${\underline{m}}^{\left(L\right)}$=$(0,-inf,-inf)$, or ${\underline{\pi }}^{\left(L\right)}=(1,0,0)$. We will call it the optimal limit point, as it is the limit of a parameter setting along which the log-likelihood function tends to its supremum.

Whether we apply the C, Y, or M perturbation, the perturbed data matrices will be the same. The perturbed data matrices can be evaluated, and the results of the evaluations for different $\epsilon$ values are shown in

Table 1. The optimal parameter values in the case of different $\epsilon$ values and perturbation methods C, M, and Y using the Bradley–Terry model. Data are shown as in Example 1.

$\mathit{\epsilon }$	$\underline{\widehat{\mathit{\pi }}}{\left(\mathit{CA}\left(\mathit{\epsilon }\right)\right)}_1$	$\underline{\widehat{\mathit{\pi }}}{\left(\mathit{CA}\left(\mathit{\epsilon }\right)\right)}_2$	$\underline{\widehat{\mathit{\pi }}}{\left(\mathit{CA}\left(\mathit{\epsilon }\right)\right)}_3$
$0.1$	$0.84008$	$0.13744$	$0.02248$
$0.01$	$0.98040$	$0.01922$	$0.00038$
$0.001$	$0.99798$	$0.00202$	${10}^{-5}$

. It can be seen that ${lim}_{\epsilon \to 0+}\underline{\widehat{\pi }}\left(CA\left(\epsilon \right)\right)={\underline{\pi }}^{\left(L\right)}.$

The S algorithm inserts an edge from the object 3 to the object 1 in this direction.

Table 2. The optimal parameter values in the case of different $\epsilon$ values and perturbation method S using the Bradley–Terry model. Data are shown as in Example 1.

$\mathit{\epsilon }$	$\underline{\widehat{\mathit{\pi }}}{\left(\mathit{SA}\left(\mathit{\epsilon }\right)\right)}_1$	$\underline{\widehat{\mathit{\pi }}}{\left(\mathit{SA}\left(\mathit{\epsilon }\right)\right)}_2$	$\underline{\widehat{\mathit{\pi }}}{\left(\mathit{SA}\left(\mathit{\epsilon }\right)\right)}_3$
$0.1$	$0.90258$	$0.08870$	$0.00872$
$0.01$	$0.99000$	$0.00990$	$0.00001$
$0.001$	$0.99900$	$0.00100$	${10}^{-6}$

contains the estimated weights for different $\epsilon$ values.

Table 2 shows that ${lim}_{\epsilon \to 0+}\underline{\widehat{\pi }}\left(SA\left(\epsilon \right)\right)={\underline{\pi }}^{\left(L\right)}$ also holds in this case. Furthermore, the convergence is quicker compared to the other perturbation cases.

Example 2. 

In our second example, the number of objects equals three again. There is no connection between the objects 1 and 2. Moreover, let

$$\begin{array}{c}\hfill A_{1,3,1}=0,0<A_{1,3,2},A_{2,3,1}=0,0<A_{2,3,2}.\end{array}$$

(22)

The graph from Example 2 and its corresponding supergraph with its levels are shown in Figure 4.

In this case, we do not have any direct information about the relationship between objects 1 and 2. The only clue may be how many times they proved to be better than object 3. Based on this, the intuitive ranking is either $1,2,3$ or $2,1,3$, depending on which value of $A_{1,3,2}$ and $A_{2,3,2}$ is greater.

The log-likelihood Function (6) with $m_1=0$ is

$$LogL\left(A\right|0,m_2,m_3)=A_{1,3,2}·log(1-F\left(m_3\right))+A_{2,3,2}·log(1-F(-m_2+m_3)).$$

(23)

It can be easily seen that if $m_3\to -\infty$ and $m_3-m_2\to -\infty$, both strictly monotonically decreasing, then $LogL\left(A\right|0,m_2,m_3)\to 0$ is strictly monotonically increasing. We can again state that the maximum does not exist, but the supremum of the log-likelihood function is 0. This value can be approached arbitrarily closely, independently of $A_{1,3,2}$ and $A_{2,3,2}$. Condition $m_3-m_2\to -\infty$ does not determine the value of $m_2$; it can be zero, positive, a negative real number, ∞, or −∞. The optimal limit points are of the followings: ${\underline{m}}^{\left(L\right)}=(0,{\widehat{m}}_2,-\infty )$. Turning to $\underline{\pi }$, we get ${\underline{\pi }}^{\left(L\right)}=({\alpha }_1,{\alpha }_2,0)$, where $0\le {\alpha }_1,{\alpha }_2$, with ${\alpha }_1+{\alpha }_2=1$. There is no connection between the strongly connected components on the $0th$ level; therefore, we have flexibility in choosing their weights. This observation will also be used again later. By performing the C, Y, M, and S perturbations, we can see that $YA=MA$, but $CA$ and $SA$ differ from them. The S perturbation method inserts two edges: one from 3 to 1 and another from 1 to 2. As $CG\left(A\right)$ is connected, the perturbed data matrices can be evaluated. The results of the evaluations of the perturbed data $CA$, $YA(=MA)$, and $SA$ are contained in

Table 3. The estimated parameter values in the case of C perturbation method and different $\epsilon$ perturbation values using the Bradley–Terry model. Data are shown as in Example 2, with $A_{1,3,2}=1,A_{2,3,2}=2$.

$\mathit{\epsilon }$	$\underline{\widehat{\mathit{\pi }}}{\left(\mathit{CA}\left(\mathit{\epsilon }\right)\right)}_1$	$\underline{\widehat{\mathit{\pi }}}{\left(\mathit{CA}\left(\mathit{\epsilon }\right)\right)}_2$	$\underline{\widehat{\mathit{\pi }}}{\left(\mathit{CA}\left(\mathit{\epsilon }\right)\right)}_3$
$0.1$	$0.40938$	$0.55961$	$0.03101$
$0.01$	$0.41371$	$0.58290$	$0.00340$
$0.001$	$0.41416$	$0.58550$	$0.00034$

,

Table 4. The estimated parameter values in the case of Y and M perturbation methods and different $\epsilon$ perturbation values using the Bradley–Terry model. Data are shown as in Example 2, with $A_{1,3,2}=1,A_{2,3,2}=2$.

$\mathit{\epsilon }$	$\underline{\widehat{\mathit{\pi }}}{\left(\mathit{YA}\left(\mathit{\epsilon }\right)\right)}_1$	$\underline{\widehat{\mathit{\pi }}}{\left(\mathit{YA}\left(\mathit{\epsilon }\right)\right)}_2$	$\underline{\widehat{\mathit{\pi }}}{\left(\mathit{YA}\left(\mathit{\epsilon }\right)\right)}_3$
$0.1$	$0.33333$	$0.63636$	$0.03030$
$0.01$	$0.33335$	$0.66335$	$0.00330$
$0.001$	$0.33327$	$0.66640$	$0.00033$

, and

Table 5. The estimated parameter values in the case of S perturbation method and different $\epsilon$ perturbation values using the Bradley–Terry model. Data are shown as in Example 2, with $A_{1,3,2}=1,A_{2,3,2}=2$.

$\mathit{\epsilon }$	$\underline{\widehat{\mathit{\pi }}}{\left(\mathit{SA}\left(\mathit{\epsilon }\right)\right)}_1$	$\underline{\widehat{\mathit{\pi }}}{\left(\mathit{SA}\left(\mathit{\epsilon }\right)\right)}_2$	$\underline{\widehat{\mathit{\pi }}}{\left(\mathit{SA}\left(\mathit{\epsilon }\right)\right)}_3$
$0.1$	$0.41746$	$0.56578$	$0.01676$
$0.01$	$0.41458$	$0.58371$	$0.00171$
$0.001$	$0.41435$	$0.58548$	$0.00017$

, respectively.

Based on the numerical results, one can observe that in the case of the Y (and M) perturbation,

$${\displaystyle \frac{\underline{\widehat{\pi }}{\left(YA\left(\epsilon \right)\right)}_1}{\underline{\widehat{\pi }}{\left(YA\left(\epsilon \right)\right)}_2}}\approx {\displaystyle \frac{1}{2}}={\displaystyle \frac{A_{1,3,2}}{A_{2,3,2}}},$$

(24)

but this is not the case for the C perturbation method. The property

$$\underset{\epsilon \to 0^{+}}{lim}{\displaystyle \frac{\underline{\widehat{\pi }}{\left(YA\left(\epsilon \right)\right)}_1}{\underline{\widehat{\pi }}{\left(YA\left(\epsilon \right)\right)}_2}}={\displaystyle \frac{A_{1,3,2}}{A_{2,3,2}}}$$

(25)

can be analytically proven in this example if we use $Y/M$ perturbation. We draw the reader’s attention to the fact that the limit of $\underline{\widehat{\pi }}\left(CA\left(\epsilon \right)\right)$ differs from the limit of $\underline{\widehat{\pi }}\left(YA\left(\epsilon \right)\right)$, which equals the limit of $\underline{\widehat{\pi }}\left(MA\left(\epsilon \right)\right)$.

Here,

$$li{m}_{\epsilon \to 0+}\underline{\widehat{\pi }}\left(CA\left(\epsilon \right)\right)={\underline{\pi }}^{\left(L\right)}=({\alpha }_1,{\alpha }_2,0)$$

(26)

with ${\alpha }_1=0.41481$, ${\alpha }_2=0.58519$,

$$li{m}_{\epsilon \to 0+}\underline{\widehat{\pi }}\left(YA\left(\epsilon \right)\right)={\underline{\pi }}^{\left(L\right)}=({\alpha }_1,{\alpha }_2,0)$$

(27)

with ${\alpha }_1=1/3$ and ${\alpha }_2=2/3$, and

$$li{m}_{\epsilon \to 0+}\underline{\widehat{\pi }}\left(SA\left(\epsilon \right)\right)={\underline{\pi }}^{\left(L\right)}=({\alpha }_1,{\alpha }_2,0)$$

(28)

with ${\alpha }_1=0.41210$ and ${\alpha }_2=0.58790$.

We can conclude that the ranking of the objects coincides with our intuition in all four perturbation methods.

Example 3. 

The third example is taken from Yan’s paper ([51], Example 1) and consists of four objects to be evaluated. The data are the following:

$$A_{i,j,k}=\left\{\begin{array}{c}1,\mathit{if}(i,j,k)=(1,2,1)\\ 2,\mathit{if}(i,j,k)=(1,2,2)\\ 1,\mathit{if}(i,j,k)=(1,4,2)\\ 2,\mathit{if}(i,j,k)=(3,4,1)\\ 1,\mathit{if}(i,j,k)=(3,4,2)\\ 0,\mathit{otherwise}.\end{array}\right.$$

(29)

The graph of Example 3 and the corresponding supergraph with its levels can be seen in Figure 5.

Now we have two levels of the supergraph, and both levels have only one strongly connected component. Intuition suggests that the ranking is $1,2,4,3$ in decreasing order.

Fix $m_1=0$, as usual. The log-likelihood function given in Equation (6) is as follows:

$$\begin{array}{c}LogL\left(A\right|0,m_2,m_3,m_4)=logF\left(m_2\right)+2·log(1-F\left(m_2\right))+\hfill \\ +log(1-F\left(m_4\right))+2·logF(m_4-m_3)+log(1-F(m_4-m_3)).\hfill \end{array}$$

(30)

The sum contains three well-separated parts. The first consists exclusively of the terms that involve the parameters of the objects in the level 0, i.e., objects 1 and 2. Now,

$$\begin{array}{c}\hfill Z^{\left(0\right)}(A,0,m_2,m_3,m_4)=logF\left(m_2\right)+2·log(1-F(m_2-m_1)).\end{array}$$

(31)

The second part serves as a bridge between the elements of level 0 and level 1. It contains the terms that include parameters from objects in both levels.

$$Z^{(0,1)}(A,0,m_2,m_3,m_4)=log(1-F\left(m_4\right)).$$

(32)

Finally, the last part contains those terms in which the parameters corresponding to the elements of level 1 are included, i.e.,

$$Z^{\left(1\right)}(A,0,m_2,m_3,m_4)=2·logF(m_4-m_3)+log(1-F(m_4-m_3)).$$

(33)

One can observe that $Z^{\left(0\right)}(A,0,m_2,m_3,m_4)$ does not depend on the variables $m_3$ and $m_4$ and can be maximized in itself. Therefore, find the maximum value of $Z^{\left(0\right)}$; denote the first and second arguments $(0,{\widehat{m}}_2)$. After some analytical computations, we get ${\widehat{m}}_2=F^{-1}(1/3)<0$. Then, find the maximum value of $Z^{\left(1\right)}$. The argument is not unique in this case; the coordinates can be shifted. Let a possible argument of the third and fourth coordinates be $({\widehat{m}}_3,{\widehat{m}}_4)$. We can prove that ${\widehat{m}}_3<{\widehat{m}}_4$ holds.

The value of $Z^{\left(1\right)}$ does not change if we substitute $m_2^{\prime }={\widehat{m}}_2$, $m_3^{\prime }={\widehat{m}}_3-x$, and $m_4^{\prime }={\widehat{m}}_4-x$, where $x\in \mathbb{R}$. Consequently,

$$\underset{x\to \infty }{lim}{Z}^{(0,1)}(A,0,m_2^{\prime },m_3^{\prime },m_4^{\prime })=0.$$

(34)

The log-likelihood function tends to its supremum, and the optimal limit point is ${\underline{m}}^{\left(L\right)}=(0,{\widehat{m}}_2,-\infty ,-\infty )$. If we use the Bradley–Terry model, the optimal limit point is ${\underline{\pi }}^{\left(L\right)}=(\widehat{\pi }{\left(A^{\prime }\right)}_1,\widehat{\pi }{\left(A^{\prime }\right)}_2,0,0)$. $\widehat{\pi }{\left(A^{\prime }\right)}_1,\widehat{\pi }{\left(A^{\prime }\right)}_2$ are the estimated weights of objects 1 and 2 based only on the comparison results between these two objects. Formally, $A^{\prime }$ contains the comparison data between objects 1 and 2. It can be easily seen that ${\underline{\pi }}^{\left(L\right)}=(2/3,1/3,0,0)$.

In this example, all four perturbation methods produce different perturbed data matrices. In the case of perturbation method C,

Table 6. The estimated parameter values applying C perturbation method in the case of different $\epsilon$ perturbation values using the Bradley–Terry model. Data are shown as in Example 3.

$\mathit{\epsilon }$	$\underline{\widehat{\mathit{\pi }}}{\left(\mathit{CA}\left(\mathit{\epsilon }\right)\right)}_1$	$\underline{\widehat{\mathit{\pi }}}{\left(\mathit{CA}\left(\mathit{\epsilon }\right)\right)}_2$	$\underline{\widehat{\mathit{\pi }}}{\left(\mathit{CA}\left(\mathit{\epsilon }\right)\right)}_3$	$\underline{\widehat{\mathit{\pi }}}{\left(\mathit{CA}\left(\mathit{\epsilon }\right)\right)}_4$
$0.1$	$0.52804$	$0.24808$	$0.08542$	$0.13846$
$0.01$	$0.64640$	$0.31629$	$0.01271$	$0.02460$
$0.001$	$0.66455$	$0.33148$	$0.00133$	$0.00264$

presents the results.

In the Bradley–Terry model, applying the Y perturbation, the estimated weights in Example 3 can be computed analytically. They are as follows:

$$\underline{\widehat{\pi }}\left(YA\left(\epsilon \right)\right)=({\displaystyle \frac{1}{DY\left(\epsilon \right)}},{\displaystyle \frac{\frac{1+\epsilon }{2+\epsilon }}{DY\left(\epsilon \right)}},{\displaystyle \frac{\frac{\epsilon }{2+\epsilon }}{DY\left(\epsilon \right)}},{\displaystyle \frac{\frac{\epsilon }{1+\epsilon }}{DY\left(\epsilon \right)}})$$

(35)

with

$$DY\left(\epsilon \right)=1+{\displaystyle \frac{1+\epsilon }{2+\epsilon }}+{\displaystyle \frac{\epsilon }{2+\epsilon }}+{\displaystyle \frac{\epsilon }{1+\epsilon }}.$$

(36)

By applying the M perturbation, the estimated weights are the following:

$$\underline{\widehat{\pi }}\left(MA\left(\epsilon \right)\right)=({\displaystyle \frac{1}{DM\left(\epsilon \right)}},{\displaystyle \frac{0.5}{DM\left(\epsilon \right)}},{\displaystyle \frac{\frac{\epsilon }{2·(1+\epsilon )}}{DM\left(\epsilon \right)}},{\displaystyle \frac{\frac{\epsilon }{1+\epsilon }}{DM\left(\epsilon \right)}})$$

(37)

where

$$DM\left(\epsilon \right)=1+0.5+{\displaystyle \frac{\epsilon }{2·(1+\epsilon )}}+{\displaystyle \frac{\epsilon }{1+\epsilon }}$$

(38)

One can see that in all three perturbations,

$$\begin{array}{c}\hfill \underset{\epsilon \to 0+}{lim}\underline{\widehat{\pi }}{\left(CA\left(\epsilon \right)\right)}_3=\underset{\epsilon \to 0+}{lim}\underline{\widehat{\pi }}{\left(YA\left(\epsilon \right)\right)}_3=\underset{\epsilon \to 0+}{lim}\underline{\widehat{\pi }}{\left(MA\left(\epsilon \right)\right)}_3=0,\\ \hfill \underset{\epsilon \to 0+}{lim}\underline{\widehat{\pi }}{\left(CA\left(\epsilon \right)\right)}_4=\underset{\epsilon \to 0+}{lim}\underline{\widehat{\pi }}{\left(YA\left(\epsilon \right)\right)}_4=\underset{\epsilon \to 0+}{lim}\underline{\widehat{\pi }}{\left(MA\left(\epsilon \right)\right)}_4=0.\end{array}$$

(39)

Moreover,

$$\begin{array}{c}\hfill \underset{\epsilon \to 0^{+}}{lim}{\displaystyle \frac{\underline{\widehat{\pi }}{\left(CA\left(\epsilon \right)\right)}_1}{\underline{\widehat{\pi }}{\left(CA\left(\epsilon \right)\right)}_2}}=\underset{\epsilon \to 0^{+}}{lim}{\displaystyle \frac{\underline{\widehat{\pi }}{\left(YA\left(\epsilon \right)\right)}_1}{\underline{\widehat{\pi }}{\left(YA\left(\epsilon \right)\right)}_2}}={\displaystyle \frac{A_{1,2,2}}{A_{1,2,1}}}=2,\\ \hfill \underset{\epsilon \to 0+}{lim}{\displaystyle \frac{\underline{\widehat{\pi }}{\left(CA\left(\epsilon \right)\right)}_3}{\underline{\widehat{\pi }}{\left(CA\left(\epsilon \right)\right)}_4}}=\underset{\epsilon \to 0+}{lim}{\displaystyle \frac{\underline{\widehat{\pi }}{\left(YA\left(\epsilon \right)\right)}_3}{\underline{\widehat{\pi }}{\left(YA\left(\epsilon \right)\right)}_4}}={\displaystyle \frac{A_{3,4,2}}{A_{3,4,1}}}={\displaystyle \frac{1}{2}}.\end{array}$$

(40)

Moreover, in the case of the M perturbation method, the equalities

$$\begin{array}{c}\hfill {\displaystyle \frac{\underline{\widehat{\pi }}{(MA\left(\left(\epsilon \right)\right)}_1}{\underline{\widehat{\pi }}{\left(MA\left(\epsilon \right)\right)}_2}}={\displaystyle \frac{A_{1,2,2}}{A_{1,2,1}}}=2,\\ \hfill {\displaystyle \frac{\underline{\widehat{\pi }}{\left(MA\left(\epsilon \right)\right)}_3}{\underline{\widehat{\pi }}{(MA\left(\left(\epsilon \right)\right)}_4}}={\displaystyle \frac{A_{3,4,2}}{A_{3,4,1}}}={\displaystyle \frac{1}{2}}\end{array}$$

(41)

hold for any $0<\epsilon$.

Perturbation method S inserts a directed edge from object 4 to object 2. The results of the evaluation of these perturbed matrices for different $\epsilon$ values are contained in

Table 7. The estimated parameter values applying S perturbation method in the case of different $\epsilon$ perturbation values using the Bradley–Terry model. Data are shown as in Example 3.

$\mathit{\epsilon }$	$\underline{\widehat{\mathit{\pi }}}{\left(\mathit{SA}\left(\mathit{\epsilon }\right)\right)}_1$	$\underline{\widehat{\mathit{\pi }}}{\left(\mathit{SA}\left(\mathit{\epsilon }\right)\right)}_2$	$\underline{\widehat{\mathit{\pi }}}{\left(\mathit{SA}\left(\mathit{\epsilon }\right)\right)}_3$	$\underline{\widehat{\mathit{\pi }}}{\left(\mathit{SA}\left(\mathit{\epsilon }\right)\right)}_4$
$0.1$	$0.63436$	$0.27936$	$0.02870$	$0.05740$
$0.01$	$0.66334$	$0.32681$	$0.00328$	$0.00657$
$0.001$	$0.66633$	$0.33267$	$0.00033$	$0.00067$

.

From Table 7, it can be seen that ${lim}_{\epsilon \to 0+}\underline{\widehat{\pi }}\left(SA\left(\epsilon \right)\right)=(2/3,1/3,0,0)$, and the convergence is quicker than in the cases of C, Y, and M perturbations. One can prove analytically that the equality

$$\begin{array}{c}\hfill {\displaystyle \frac{\underline{\widehat{\pi }}{\left(SA\left(\epsilon \right)\right)}_3}{\underline{\widehat{\pi }}{\left(SA\left(\epsilon \right)\right)}_4}}={\displaystyle \frac{1}{2}}\end{array}$$

(42)

holds again.

In this case as well, the rankings agree with the intuition.

Finally,

Table 8. The numbers of inserted edges in the case of different perturbation methods.

Perturbation Method	C	Y	M	S
Example 1	6	6	6	1
Example 2	6	4	4	2
Example 3	12	6	2	1

contains the numbers of inserted edges in the case of different perturbation methods.

One can see that the S perturbation method inserts the fewest edges, always only as many as are absolutely necessary. In [54], the authors suggested applying Bayesian parameter estimation with gamma prior distributions. Example 3 was analyzed in [51] using the mentioned method, and the results are presented in Section 5, Table 6, of the paper [51]. Based on this, we can see that, for different parameters of the prior distributions, the results do not approach the optimal limit point—even in the case of the parameter setting suggested by the authors.

6. Characterization of the Optimal Limit Point Based on the Data Structure

Our observations on Examples 1, 2, and 3 help us to formulate a statement about the optimal limit points. In the presented examples, the log-likelihood function does not attain its maximum; however, its supremum can be approached arbitrarily closely. We can find trajectories along which the supremum can be approximated. The arguments of these trajectories have limits: these limit points are the optimal limit points. These points are not in the domain of the log-likelihood function, but they lie on the boundary of the domain (possibly including the value of $-\infty$). In Example 2, we realized that the supremum can be approached on different trajectories which result in more than one optimal limit points. At the same time, different perturbation methods lead to different evaluation results in the limit as $\epsilon$ approaches zero.

The main idea of finding these trajectories is the following: Separate the terms of the sum (6) into specific groups—those between objects belonging to strongly connected components on the same level and those between components on different levels. We show that the terms between objects from different levels are increasing and tend to zero, while the terms between objects on the same level can be optimized independently, since there is no connection between the strongly connected components on the same level.

Assume that $CG\left(A\right)$ is connected, but $DG\left(A\right)$ is not strongly connected.

We will need some notations. The structure of the notations is the following: Letter $S_l^{\left(s\right)}$ is for the strongly connected component itself, while $I_l^{\left(s\right)}$ the set of objects in it. The upper index refers to the level of the strongly connected component of the objects; the lower index is for the identifier of the strongly connected component at its level. The notations are detailed as follows:

• Let the number of strongly connected components of $DG\left(A\right)$ be $NS$, where $1<NS$.
• Let the number of the level of the supergraph constructed on the basis of $DG\left(A\right)$ be $NL+1$, where $0<NL$, and let the levels be indexed by $0,1,\dots ,NL$.
• Let the strongly connected components be denoted by $S_l^{\left(s\right)}$, where s refers to the level ($s=0,\dots ,NL)$, and l is the index of the strongly connected component being on the level s, i.e., $l=1,\dots ,n_s$.
• Let the index set of objects belonging to the objects in strongly connected component $S_l^{\left(s\right)}$ be $I_l^{\left(s\right)}$.
• Let $I^{\left(s\right)}$ be the indices of the objects in strongly connected components that belong to super-nodes at the sth level, i.e., $I^{\left(s\right)}={\cup }_{l=1}^{n_s}{I}_l^{\left(s\right)}$.
• Let the sum of the terms in the log-likelihood function be denoted by $Z_l^{\left(s\right)}(A,\underline{m})$, which exclusively contains those parameters belonging to the lth strongly connected component at level s, i.e.,

  $$Z_l^{\left(s\right)}(A,\underline{m})=\sum _{i,j\in I_l^{\left(s\right)},k=1,2}{A}_{i,j,k}·log{p}_{i,j,k},s=0,\dots ,NL;l=1,\dots ,n_s.$$

  (43)
• Let

  $$Z^{\left(s\right)}(A,\underline{m})=\sum _{l=1}^{n_s}{Z}_l^{\left(s\right)}(A,\underline{m}),s=0,\dots ,NL.$$

  (44)
• Let the sum of the terms be denoted by $Z^{(s_1,s_2)}(A,\underline{m})$, which depend exclusively on the parameters of the objects at level $s_1$ and at level $s_2$, i.e.,

  $$Z^{(s_1,s_2)}(A,\underline{m})=\sum _{i\in I^{\left(s_1\right)},j\in I^{\left(s_2\right)},k=1,2}{A}_{i,j,k}·log{p}_{i,j,k}{s}_1,s_2=0,\dots ,NL,s_1<s_2.$$

  (45)
• Let $Z^{\left(diff\right)}(A,\underline{m})$ denote the sum of the terms that depend on the parameters of the objects at different levels, i.e.,

  $$Z^{\left(diff\right)}(A,\underline{m})=\sum _{s_1<s_2,s_1=0,\dots ,NL,s_2=1,\dots ,NL}{Z}^{(s_1,s_2)}(A,\underline{m}).$$

  (46)

Lemma 1. 

The log-likelihood Function (6) equals

$$LogL\left(A\right|\underline{m})=\sum _{s=0}^{NL}{Z}^{\left(s\right)}(A,\underline{m})+Z^{\left(diff\right)}(A,\underline{m}).$$

(47)

Proof. 

Due to the construction of the levels of the supergraph, there is no comparison between objects on the same level but in different strongly connected components, i.e., $A_{i,j,k}=0$ for $k=1,2$ if $i\in I_{l_1}^{\left(s\right)}$ and $j\in I_{l_2}^{\left(s\right)}$, with $l_1\ne l_2$ and $s=0,\dots ,NL$. Therefore, the terms in the log-likelihood function can be grouped into two separate parts. One part contains terms that depend only on parameters included in the fixed strongly connected components; these are contained in $Z^{\left(s\right)}(A,\underline{m})$. The other part consists of terms involving parameters from strongly connected components on different levels; these are included in $Z^{\left(diff\right)}(A,\underline{m})$. □

We now turn to the optimization of $Z^{\left(s\right)}$.

Lemma 2. 

$Z^{\left(s\right)}(A,\underline{m})$ for $s=0,1,\dots ,NL$ can be optimized by maximizing $Z_l^{\left(s\right)}(A,\underline{m})$ for every index $s=0,1,\dots ,NL$ and $l=1,\dots ,n_s$ separately.

Proof. 

Taking into consideration (44), the maximum of $Z^{\left(s\right)}(A,\underline{m})$ corresponds to the maximum of separated terms. Since every object is in a unique strongly connected component, the sum achieves its maximum when each $Z_l^{\left(s\right)}$ reaches its maximum. For a fixed strongly connected component, the maximum of the log-likelihood function exists, and the argument is unique if we fix one parameter in each strongly connected component. Consequently, the arguments of the maximal values are only uniquely determined up to a translation in every strongly connected component. □

After maximizing the sum of the terms arising from the same strongly connected components, let us examine those belonging to different strongly connected components. Since there is no edge between two components at the same level, we only need to deal with the terms associated with objects belonging to components at different levels. It will be shown that the sum of these terms converges to zero along certain sequences, while the terms within the same strongly connected components remain maximal.

Lemma 3. 

Fixing a parameter setting $\underline{m}$ and making the transformation

$$m_i^{\prime }=m_i-s·x\mathit{if}i\in I^{\left(s\right)},0\le s,$$

(48)

then

$$\underset{x\to +\infty }{lim}{Z}^{\left(diff\right)}=0.$$

(49)

Proof. 

We show that every term (45) tends to zero if $x\to \infty$. If $s_1<s_2$, then, there are only ‘better’ results between the elements in $i\in I^{\left(s_1\right)}$ and $j\in I^{\left(s_2\right)}$, i.e., $A_{i,j,1}=0$, for $i\in I^{\left(s_1\right)}$ and $j\in I^{\left(s_2\right)}$, as the objects i and j belong to different levels. Moreover, the terms $1-F(m_j-s_2·x-(m_i-s_1·x))$ tend to 1 if $x\to \infty$, as the arguments of F tend to $-\infty$. Therefore, their logarithms tend to the largest value, which is zero. Consequently, the sum of them will tend to zero as well. □

Lemma 4. 

With the transformation of (48), we have

$$Z^{\left(s\right)}(A,{\underline{m}}^{\prime })=Z^{\left(s\right)}(A,\underline{m})\mathit{for}\mathit{any}0\le s.$$

(50)

Proof. 

It is easy to see that in the case of $i,j\in I^{\left(s\right)}$, $m_i^{\prime }-m_j^{\prime }=m_i-m_j$. Equations (3) and (4) show that the quantities $p_{i,j,k}$ depend only on the difference in expectations. □

Lemma 5. 

$$\underset{\underline{m}\in {\mathbb{R}}^n}{sup}LogL\left(A\right|\underline{m})=\sum _{s=0}^{NL}{Z}^{\left(s\right)}(A^{\prime },\underline{\widehat{m}}\left(S_l^{\left(s\right)}\right)),$$

(51)

where $\widehat{m}\left(S_l^{\left(s\right)}\right)$ is the maximum likelihood estimate of the expectations based on the comparison data between the objects in $S_l^{\left(s\right)}$. The appropriate data sets are denoted by $A^{\prime }$. The estimated values are unique only apart from translation.

Proof. 

Recalling Lemmas 1 and 2, and taking into account Lemmas 3 and 4, we get the statement. □

Now we turn to one of the main theorems in the section. It is for the determination of optimal limit point(s). Its essence can be summarized as follows. Suppose that $CG\left(A\right)$ is connected but $DG\left(A\right)$ is not strongly connected. Considering the original form, optimize the log-likelihood functions separately in each strongly connected component at level 0 by fixing one parameter to zero. These values can be shifted separately by the same value for a fixed strongly connected component. Use these shifted arguments’ coordinates for the appropriate coordinates of ${\underline{m}}^{\left(L\right)}$. All other coordinates have to be $-\infty$.

If we consider the exponential form (8), optimize the log-likelihood function separately within each strongly connected component at level 0. These yield estimated weights based on the local comparison results, which involve only the objects within the given strongly connected component. Take an arbitrary normalized vector with non-negative coordinates whose dimension equals the number of strongly connected components at level 0. These coordinates will serve as multipliers for the estimated weights within the strongly connected components. The multipliers are constant within each strongly connected component but may differ across components. The coordinates corresponding to the objects at level 0 will be the values obtained as a result of this multiplication. Assign zero to all remaining coordinates.

For the sake of the formal statement and the proof, we introduce a further notation: the function h defines a one-to-one correspondence between the indices of the objects in the strongly connected components and their original values. For example, suppose object 7 is in the third strongly connected component at level 0, and it is the fourth element in that component. Then, $i=7$, and the seventh coordinate of the optimal limit point corresponds to the fourth estimated parameter in $S_3^{\left(0\right)}$. This is expressed as $h(S_3^{\left(0\right)})\left(7\right)=4$.

Now, we can state the following.

Theorem 6. 

Suppose that $CG\left(A\right)$ is connected, and $DG\left(A\right)$ is not strongly connected.

I. 

The optimal limit points can be expressed as follows:

$${\underline{m}}_i^{\left(L\right)}=\left\{\begin{array}{c}\underline{\widehat{m}}{\left(S_l^{\left(0\right)}\right)}_{h(S_l^{\left(0\right)})\left(i\right)}+d_l,\mathit{if}i\in I_l^{\left(0\right)},l=1,\dots ,n_0,\hfill \\ -\infty ,\mathit{if}i\notin I^{\left(0\right)}.\hfill \end{array}\right.$$

(52)

Here, $\underline{\widehat{m}}\left(S_l^{\left(0\right)}\right)$ are the estimated expectations of the objects in the strongly connected components $I_l^{\left(0\right)}$. The estimation is performed exclusively based on the data of $S_l^{\left(0\right)}$ after fixing one parameter to 0. These values are shifted by an arbitrary constant $d_l$.

II. 

If we turn to the exponential form of (8), the optimal limit points are as follows:

$${\pi }_i^{\left(L\right)}=\left\{\begin{array}{c}{\alpha }_l·\underline{\widehat{\pi }}{\left(S_l^{\left(0\right)}\right)}_{h\left(S_l^{\left(0\right)}\left(i\right)\right)},\mathit{if}i\in I_l^{\left(0\right)},l=1,\dots ,n_0,\hfill \\ 0,\mathit{if}i\notin I^{\left(0\right)}\hfill \end{array}\right.$$

(53)

Here, $\underline{\widehat{\pi }}\left(S_l^{\left(0\right)}\right)$ are the estimated weights computed by exclusively applying the data of the strongly connected component $S_l^{\left(0\right)}$. ${\alpha }_l$ for $l=1,\dots ,n_0$ are arbitrary non-negative constant values belonging to the lth strongly connected component on the top level, with ${\sum }_{l=1}^{n_0}{\alpha }_l=1$.

III. 

If there is only one strongly connected component at the top level, then, by fixing one parameter in $I_1^{\left(0\right)}$ to zero, there is a unique optimal limit point, and it is

$${\underline{m}}_i^{\left(L\right)}=\left\{\begin{array}{c}\underline{\widehat{m}}{\left(S_1^{\left(0\right)}\right)}_{h(S_1^{\left(0\right)})\left(i\right)},\mathit{if}i\in I_1^{\left(0\right)},\hfill \\ -\infty ,\mathit{if}i\notin I_1^{\left(0\right)}\hfill \end{array}\right.$$

(54)

IV. 

Turning to the exponential form of (8), if there is only one strongly connected component at the top level, the optimal limit point, under the constraint ${\sum }_{i=1}^n{\widehat{\pi }}_i=1$, is unique and equals

$${\pi }_i^{\left(L\right)}=\left\{\begin{array}{c}\underline{\widehat{\pi }}{\left(S_1^{\left(0\right)}\right)}_{h\left(S_1^{\left(0\right)}\left(i\right)\right)},\mathit{if}i\in I_1^{\left(0\right)},\hfill \\ 0,\mathit{if}i\notin I_1^{\left(0\right)}.\hfill \end{array}\right.$$

(55)

Proof. 

I. Recall Lemma 5 and note that the Function (46) is strictly increasing as $x\to \infty$.

II. Take the exponential transformations, with $exp(-\infty )=0$, and normalize. The possible translations in the strongly connected components are incorporated into the ${\alpha }_l$ values.

III. From I, we know that the coordinates corresponding to the indices of the strongly connected components at the top level can be determined independently by fixing one expectation, and they can be shifted. From I, we know that the coordinates corresponding to the indices of the strongly connected components at the top level can be determined independently by fixing one expectation, and they can be shifted. Since there is only one strongly connected component at the top level, the possibility of shifting corresponds to fixing a single coordinate. All other coordinates, belonging to the strongly connected components below the top level, are equal to $-\infty$.

IV. From II, we know that the non-zero coordinates of the optimal limit points are the weighted averages of the normalized weights corresponding to the objects in the strongly connected components at the top level. Since there is only one strongly connected component, the weighted average equals the original weights, and the other coordinates are zero. □

Note that the optimal limit point may depend on the distribution of the differences ${\eta }_{i,j}$, but which coordinates are zero depends only on the data structure, i.e., the supergraph.

Theorem 6 clarifies the conditions under which the optimal limit point is unique or non-unique. In practice, non-uniqueness may allow the decision maker to assign weights to the objects within certain strongly connected components at level 0 at their own discretion. In such cases of non-evaluability, it is advisable to avoid relying on this possibility and instead request additional comparisons to ensure that decisions are based on reliable comparison results. The optimal pairs are explored by the S algorithm. This remark clearly also holds when the comparison graph is disconnected. In such a case, nothing can be said about the relationship between the separate components, so it is very important to perform comparisons between elements of different subsets.

An interesting question is the relationship between the optimal limit points and the evaluation results based on the perturbed data matrices for small values of $\epsilon$. In [50], the authors proved that, in the case of the Bradley–Terry model,

$$\underline{\widehat{\pi }}\left(CA\left(0\right)\right)=li{m}_{\epsilon \to 0+}\underline{\widehat{\pi }}\left(CA\left(\epsilon \right)\right)$$

(56)

exists. The same argumentation as in [50] is true for Y, M, and S perturbations. Therefore, we can conclude that $\underline{\widehat{\pi }}\left(YA\left(0\right)\right)=li{m}_{\epsilon \to 0+}\underline{\widehat{\pi }}\left(YA\left(\epsilon \right)\right)$, $\underline{\widehat{\pi }}\left(MA\left(0\right)\right)=li{m}_{\epsilon \to 0+}\underline{\widehat{\pi }}\left(MA\left(\epsilon \right)\right)$, and $\underline{\widehat{\pi }}\left(SA\left(0\right)\right)=li{m}_{\epsilon \to 0+}\underline{\widehat{\pi }}\left(SA\left(\epsilon \right)\right)$ exist. Although the proof cannot be directly generalized for the case of $F\in \mathbb{F}$, simulation results support that in this case the same statements hold. Moreover, based on Examples 1–3 and simulations, we conjecture that these limits coincide with one of the optimal limit points. Example 2 shows that these limits may differ from each other. Nevertheless, if the optimal limit point is unique (i.e., there is only one strongly connected component at the top level); then,

$$\underline{\widehat{m}}\left(CA\left(0\right)\right)=\underline{\widehat{m}}\left(YA\left(0\right)\right)=\underline{\widehat{m}}\left(MA\left(0\right)\right)=\underline{\widehat{m}}\left(SA\left(0\right)\right)={\underline{m}}^{\left(L\right)}.$$

(57)

If $CG\left(A\right)$ contains more than one maximal connected subgraph, we do not have any information about the relations of the subgraphs. Consequently, we cannot make a reliable evaluation. It may happen that two subgraphs have approximately the same distribution in terms of the strength of the objects within them, but it is also possible that every element in one group is stronger than all the elements in the other group. Think, for example, of teams playing in Serie A and Serie D in the Italian football league system.

Now, we turn to the relation between the optimal limit points and the limit of the evaluation results based on perturbed data if $\epsilon \to 0+.$

Theorem 7 (Assume that $CG\left(A\right)$ is connected).

Consider the Bradley–Terry model, and let $GA\left(\epsilon \right)$ be a general ε-perturbation of the data matrix A, i.e.,

$$G{A}_{i,j,k}\left(\epsilon \right)=\left\{\begin{array}{c}{A}_{i,j,k}+\epsilon ,\mathit{for}\mathit{some}(i,j)\mathit{pairs}\mathit{and}k\mathit{value},\hfill \\ A_{i,j,k},\mathit{otherwise},\hfill \end{array}\right.$$

(58)

keeping the property $GA{\left(\epsilon \right)}_{j,i,1}=GA{\left(\epsilon \right)}_{i,j,2}$. Let us assume that the perturbed data matrix $GA\left(\epsilon \right)$ is evaluable. Then,

$$\underset{\epsilon \to 0+}{lim}\underline{\widehat{\pi }}\left(GA\left(\epsilon \right)\right)=\underline{\widehat{\pi }}\left(0\right)$$

(59)

exists and is equal to an optimal limit point ${\underline{\pi }}^{\left(L\right)}$.

If there exists a unique optimal limit point, then the limits of the evaluations, $\epsilon \to 0+$, are the same when applying any perturbation method.

Proof. 

Let us carefully go through Conner and Grant’s proofs in [50]. On the basis of Theorem 1.1 in [50] one can see that the limit exists. A slight modification of the proof of Theorem 3.3 in [50] guarantees that if there is a directed path from i to j, then ${\displaystyle \frac{\widehat{\pi }{\left(GA\left(\epsilon \right)\right)}_j}{\widehat{\pi }{\left(GA\left(\epsilon \right)\right)}_i}}$ is bounded (simply replace $n(n-1)$ by 1 or by the number of the inserted edges). If there is a directed path from i to j but there is no directed path from j to i, then

$$\underset{\epsilon \to 0+}{lim}{\displaystyle \frac{\widehat{\pi }{\left(GA\left(\epsilon \right)\right)}_j}{\widehat{\pi }{\left(GA\left(\epsilon \right)\right)}_i}}=0.$$

(60)

The condition “there is a directed path from i to j but there is no directed path from j to i” means that i and j are in strongly connected components on different levels, and the level of i is less than the level of j. The optimal limit points have this property: their coordinates are equal to 0 for objects outside strongly connected components at the top level. Moreover, if objects i and j are in the same strongly connected component at the top level, Theorem 3.5 in [50] guarantees that

$$\underset{\epsilon \to 0}{lim}{\displaystyle \frac{\widehat{\pi }{\left(GA\left(\epsilon \right)\right)}_j}{\widehat{\pi }{\left(GA\left(\epsilon \right)\right)}_i}}={\displaystyle \frac{\widehat{\pi }{\left(A^{\prime }\right)}_j}{\widehat{\pi }{\left(A^{\prime }\right)}_i}}.$$

(61)

Here, $A^{\prime }$ is the data matrix reduced to the current strongly connected component. As the strongly connected components can be separately optimized to approach the supremum of the log-likelihood function, this property aligns with that observed in optimal limit points. Hence, the limit of the coordinates computed by an arbitrary $\epsilon$-perturbation method constitutes an optimal limit point. If there is only one optimal limit point, then the limit of the evaluation results obtained using different perturbation methods must be the same. This is the case in Examples 1 and 3. If there is more than one optimal limit point, the limits of the evaluation results obtained from different perturbation methods may differ. This is illustrated in Example 2 using perturbations C, $Y/M$, and S.

If we take the logarithm of the limit of different coordinates, we get the same statement for the additive form of the Bradley–Terry model. □

From the above, it follows that for each G perturbation method that makes evaluable data matrix A, one can construct $\underline{\widehat{\pi }}\left(GA\left(\epsilon \right)\right)$ sequences along which the log-likelihood function, which converges to its supremum.

We note that, although the proof is valid only for the Bradley–Terry model, the authors believe that the statement remains valid when a general c.d.f. is applied.

7. Comparison of the Perturbation Methods by Computer Simulations

We performed computer simulations to compare the properties of different perturbation methods, focusing on the similarity of rankings and the number of inserted data. We applied the Bradley–Terry model and randomly chose an initial weight vector. We compared the results to this vector, as we are interested in the extent of information retrieval compared to the original rankings. The steps of the simulations are the following:

ST_I

Generate uniformly distributed ${\pi }_i$ values for $i=1,\dots ,n$ in the interval (0, 1), and normalize them. These will be called the original weights. Determine the ranking defined by them; it is referred to as the original ranking.

ST_II

Compute the probabilities of the decisions ‘better’ and ‘worse’ according to (10) and (9), respectively, for each pair $(i,j)$.

ST_III

Randomly generate edges according to the number of comparisons and also generate data corresponding to the edges according to the probabilities computed in ST_II.

ST_IV

Check whether the data are evaluable or not, and consider only the cases when they are not evaluable, but the comparison graph is connected.

ST_V

Apply the perturbations C, Y, M, and S to the data. Compute the numbers of the inserted edges.

ST_VI

Evaluate the perturbed data matrices.

ST_VII

Compare the ranking defined by the estimates to the original ranking by computing Spearman [55] and Kendall rank correlations [56].

ST_VIII

Repeat the steps ST_I to ST_VII as many times as the simulation number indicates.

ST_IX

Take the averages over the simulations and determine the standard deviation of the quantities computed in steps ST_V and ST_VII.

$N={10}^6$ simulations were performed. We chose $n=10$ as the number of objects, and the number of comparisons was set to 20, 40, 60, and 80. The ratios of connected but not strongly connected graphs were 87.48%, 65.38%, 34.5%, and 23.61%, respectively, in these cases. We have investigated the mentioned cases. For the values $\epsilon =1$ and $0.001$,

Table 9. The number of inserted edges, as well as average values and standard deviations of Spearman’s $\rho$ and Kendall’s $\tau$ rank correlations, related to the initial parameter setting for random comparison data with 20, 40, 60, and 80 comparisons using C perturbation in the case of $n=10$ objects.

c. n.	n. of Inserted Edges	Spearman’s $\mathit{\rho }$ Average; St. Dev.		Kendall’s $\mathit{\tau }$ Average; St. Dev.
		$\epsilon =\mathbf{1}$	$\epsilon =\mathbf{0}.\mathbf{001}$	$\epsilon =\mathbf{1}$	$\epsilon =\mathbf{0}.\mathbf{001}$
20	90	0.518; 0.250	0.483; 0.270	0.393; 0.207	0.367; 0.221
40	90	0.663; 0.196	0.648; 0.208	0.521; 0.180	0.511; 0.189
60	90	0.734; 0.166	0.729; 0.172	0.594; 0.165	0.591; 0.170
80	90	0.770; 0.153	0.768; 0.155	0.634; 0.158	0.634; 0.160

,

Table 10. The average number and the standard deviation of inserted edges and the average values and standard deviations of Spearman’s $\rho$ and Kendall’s $\tau$ rank correlations related to the initial parameter setting for random comparison data with 20, 40, 60, and 80 comparisons using Y perturbation in the case of $n=10$ objects.

c. n.	n. of Inserted Edges Average; St. Dev.	Spearman’s $\mathit{\rho }$ Average; St. Dev.		Kendall’s $\mathit{\tau }$ Average; St. Dev.
		$\epsilon =\mathbf{1}$	$\epsilon =\mathbf{0}.\mathbf{001}$	$\epsilon =\mathbf{1}$	$\epsilon =\mathbf{0}.\mathbf{001}$
20	32.78; 2.84	0.481; 0.270	0.476; 0.271	0.366; 0.220	0.362; 0.221
40	53.18; 4.08	0.655; 0.204	0.647; 0.208	0.516; 0.186	0.510; 0.189
60	66.44; 4.29	0.733; 0.169	0.729; 0.172	0.594; 0.168	0.591; 0.170
80	74.97; 4.03	0.770; 0.154	0.768; 0.155	0.635; 0.159	0.634; 0.160

,

Table 11. The average number and the standard deviation of inserted edges, as well as the average values and standard deviations of Spearman’s $\rho$ and Kendall’s $\tau$ rank correlations, related to the initial parameter setting for random comparison data with 20, 40, 60, and 80 comparisons using M perturbation in the case of $n=10$ objects.

c. n.	n. of Inserted Edges Average; St. Dev.	Spearman’s $\mathit{\rho }$ Average; St.Dev.		Kendall’s $\mathit{\tau }$ Average; St. Dev.
		$\epsilon =\mathbf{1}$	$\epsilon =\mathbf{0}.\mathbf{001}$	$\epsilon =\mathbf{1}$	$\epsilon =\mathbf{0}.\mathbf{001}$
20	30.17; 4.18	0.480; 0.270	0.476; 0.271	0.365; 0.220	0.362; 0.221
40	44.95; 6.02	0.653; 0.204	0.647; 0.208	0.514; 0.186	0.510; 0.189
60	51.95; 6.6	0.730; 0.169	0.729; 0.172	0.591; 0.167	0.591; 0.170
80	54.06; 6.75	0.768; 0.154	0.768; 0.155	0.632; 0.159	0.634; 0.160

and

Table 12. The average number and the standard deviation of inserted edges, as well as the average values and standard deviations of Spearman’s $\rho$ and Kendall’s $\tau$ rank correlations, related to the initial parameter settings for random comparison data with 20, 40, 60, and 80 comparisons using S perturbation in the case of $n=10$ objects.

c. n.	n. of Inserted Edges Average, St. Dev.	Spearman’s $\mathit{\rho }$ Average; St. Dev.		Kendall’s $\mathit{\tau }$ Average; St. Dev.
		$\epsilon =\mathbf{1}$	$\epsilon =\mathbf{0}.\mathbf{001}$	$\epsilon =\mathbf{1}$	$\epsilon =\mathbf{0}.\mathbf{001}$
20	2.52; 1.12	0.478; 0.270	0.482; 0.270	0.361; 0.220	0.367; 0.221
40	1.2; 0.44	0.648; 0.207	0.647; 0.208	0.509; 0.188	0.510; 0.189
60	1.06; 0.24	0.729; 0.172	0.729; 0.172	0.590; 0.169	0.591; 0.170
80	1.03; 0.16	0.768; 0.154	0.768; 0.155	0.633; 0.160	0.634; 0.160

present the results for the C, Y, M, and S perturbation methods, respectively. The comparison numbers appear in the first column under “c.n.” The second column of the tables shows the average numbers of insertions (i.e., artificial data) along with the standard deviations, separated by semicolons. Following these, the tables report the average values and standard deviations of Spearman’s $\rho$ and Kendall’s $\tau$ rank correlations between the original and estimated rankings.

Looking at the correlation coefficients, we observe that as the number of comparisons increases, the average values increase in the case of all four perturbation methods. Meanwhile, the standard deviations decrease with the growing number of comparisons.

From Table 9, Table 10, Table 11 and Table 12, Spearman’s $\rho$ correlation coefficients show that, based on 20 comparisons, the original rankings can be reconstructed to a moderate extent on average, while with 60 and 80 comparisons, the reconstruction is efficient. The same conclusion can be drawn from the Kendall’s $\tau$ coefficients. A larger comparison number results in better information retrieval with lower uncertainty across all four perturbation cases.

Comparing the different methods with the same number of comparisons, we observe that the average values and dispersions are very similar across all four perturbation methods. Usually, the averages corresponding to $\epsilon =1$ are slightly larger than those corresponding to $\epsilon =0.001$. The examination of the $\epsilon =1$ perturbation case is interesting because it involves the insertion of data that could also occur in reality. From Table 9, Table 10, Table 11 and Table 12, we can see that both rank correlation coefficients across all four perturbations tend toward similar values as $\epsilon$ decreases. This trend is especially apparent in graphs based on 60 and 80. It is also evident that perturbation C yields the most favorable rank correlation coefficient values, followed by Y, M, and finally S. However, the differences among the correlation values are minimal.

The substantial difference is in the average number of inserted edges. Perturbation S inserts significantly less edges compared to the other perturbations. We can also see that perturbations Y and M seem to add more and more edges in graphs with higher edge counts, while S adds less edges as the number of original edges increases. The average values of around 1 and 2.5 indicate that the comparison data require only a few well-assigned new comparison data to become evaluable. The information retrieval is similarly effective as with the other perturbation methods that use a lot of artificial data. Therefore, it may be worthwhile to request real comparison data if possible. The S algorithm, which is based on data structure analysis, reveals which pairs should be compared.

Similar phenomena are observed for larger numbers of objects.

Table 13. The average number and the standard deviation of the numbers of inserted edges, as well as the average values and standard deviations of Spearman’s $\rho$ and Kendall’s $\tau$ rank correlations, related to the initial parameter setting for random comparison data with 200 comparisons using all investigated perturbations in the case of $n=50$ objects.

p. m.	n. of Inserted Edges Average, St. Dev.	Spearman’s $\mathit{\rho }$ Average; St. Dev.		Kendall’s $\mathit{\tau }$ Average; St. Dev.
		$\epsilon =\mathbf{1}$	$\epsilon =\mathbf{0}.\mathbf{001}$	$\epsilon =\mathbf{1}$	$\epsilon =\mathbf{0}.\mathbf{001}$
C	2450; 0	0.652; 0.074	0.629; 0.083	0.474; 0.063	0.458; 0.069
Y	369.19; 7.04	0.639; 0.081	0.627; 0.083	0.466; 0.067	0.456; 0.069
M	357.47; 10.33	0.638; 0.081	0.627; 0.083	0.465; 0.068	0.456; 0.069
S	3.51; 1.6	0.631; 0.083	0.627; 0.083	0.459; 0.068	0.456; 0.069

reports the average numbers of inserted edges and Spearman and Kendall correlation coefficients, together with the standard deviations, for $n=50$ objects and 200 comparisons. In this setting, 98% of the random graph of comparisons are connected, but the directed graph is not strongly connected. The standard deviations are considerably smaller than for $n=10$ objects. The averages and standard deviations are very close to each other for the different methods. The number of inserted edges for algorithm S is only a fraction (less than 1%) of that required by the other algorithms.

We also examined the convergence properties of the different methods. In

Table 14. The average values and standard deviations of the Euclidean distances between evaluation results obtained by perturbations and the optimal limit points for random data with 200 comparisons, considering all perturbations in the case where there are n = 50 objects.

Perturbation Method	Euclidean Distance Average; St. Dev.
	$\epsilon =\mathbf{1}$	$\epsilon =\mathbf{0}.\mathbf{1}$	$\epsilon =\mathbf{0}.\mathbf{01}$	$\epsilon =\mathbf{0}.\mathbf{001}$
C	0.531; 0.275	0.508; 0.283	0.405; 0.266	0.146; 0.132
Y	0.510; 0.280	0.399; 0.255	0.133; 0.109	0.017; 0.017
M	0.509; 0.280	0.399; 0.255	0.133; 0.109	0.017; 0.017
S	0.452; 0.347	0.228; 0.197	0.044; 0.057	0.005; 0.008

, we present the differences between the optimal limit point and the evaluation results obtained by different perturbation methods when the optimal limit point is unique. The number of objects was $n=50$, and the number of comparisons was 200. About 50% of the random comparison graphs were connected, but the directed graphs were not strongly connected, with only one strongly connected component at the top level. We can see that the S algorithm shows the best convergence property with the smallest average $l_2$ distances and with the smallest standard deviations. The same can be stated if we investigate the rank correlations between the rankings of the evaluations applying perturbed data and ranking of the optimal limit point. Regarding convergence in rank correlations, the best method is perturbation S, as shown in

Table 15. The average values and standard deviations of Spearman’s $\rho$ rank correlations when compared to the optimal limit point for random comparison data with 200 comparisons for each investigated perturbation in the case of $n=50$ objects.

Perturbation Method	Spearman’s $\mathit{\rho }$ Average; St. Dev.
	$\epsilon =\mathbf{1}$	$\epsilon =\mathbf{0}.\mathbf{1}$	$\epsilon =\mathbf{0}.\mathbf{01}$	$\epsilon =\mathbf{0}.\mathbf{001}$
C	0.907; 0.034	0.940; 0.017	0.991; 0.004	0.999; 0.001
Y	0.983; 0.007	0.996; 0.003	0.999; 0.002	0.999; 0.001
M	0.980; 0.008	0.996; 0.003	0.999; 0.002	0.999; 0.008
S	0.986; 0.022	0.999; 0.002	0.999; 0.001	0.999; 0.001

and

Table 16. The average values and standard deviations of Kendall’s $\tau$ rank correlations when compared to the optimal limit point for random optimal comparison data with 200 comparisons for each investigated perturbation in the case of $n=50$ objects.

Perturbation Method	Kendall’s $\mathit{\tau }$ Average; St. Dev.
	$\epsilon =\mathbf{1}$	$\epsilon =\mathbf{0}.\mathbf{1}$	$\epsilon =\mathbf{0}.\mathbf{01}$	$\epsilon =\mathbf{0}.\mathbf{001}$
C	0.744; 0.035	0.798; 0.030	0.935; 0.016	0.988; 0.007
Y	0.904; 0.022	0.961; 0.014	0.991; 0.007	0.995; 0.006
M	0.893; 0.024	0.959; 0.014	0.991; 0.007	0.995; 0.006
S	0.950; 0.030	0.991; 0.008	0.995; 0.006	0.996; 0.006

.

We also examined the runtime of the perturbation methods. The results can be seen in

Table 17. The average runtimes of all perturbations across multiple object and comparison numbers. The runtime values are for ${10}^6$ simulations and measured in seconds (s). (o.n. = object number; c.n. = number of comparisons.)

o. n.	c. n.	Perturbation Methods
		C	Y	M	S
10	20	0.62	0.74	0.85	236.74
	40	0.66	0.86	1.02	160.29
	60	0.65	0.79	0.98	139.13
	80	0.62	0.67	0.87	119.53
50	200	13.22	12.14	11.35	688.67

. The simulations were run on a laptop with an AMD Ryzen 3 5425U processor. We can see that perturbations C, Y, and M share similar average running times, with M having a slightly higher value than the other two. Despite perturbation S having the worst average runtime, we can confidently say that this perturbation is computable within a short amount of time, as the average value concerning one simulation is below ${10}^{-3}$ s.

8. An Application for Analyzing the Results of a Sports Tournament

Applications of pairwise comparison for evaluating tennis players can be found in the literature [57,58]. The presented method is based on a pairwise comparison matrix whose elements are constructed by dividing the number of wins by the number of defeats. However, this construction requires modification if only one match is played between any two players—which is typically the case in a tournament. Therefore, we will apply a Thurstone-motivated method to estimate the weights of the participants in the 2019 ATP Finals based on the match results of this tournament.

A tennis match ends with the victory of one of the players; therefore, the result can be considered the outcome of a comparison allowing two options, ‘better’ and ‘worse’. If Player 1 defeats Player 2, then Player 1 is considered the ‘better’ player in this comparison (match), and Player 2 is considered to be ‘worse’ in this comparison.

The 2019 ATP Finals was a men’s year-end tennis tournament featuring eight participants, which was held in London. The players were Matteo Berrettini (B), Novak Djokovic (D), Roger Federer (F), Daniil Medvedev (M), Rafael Nadal (N), Dominic Thiem (TH), Stefanos Tsitsipas (TS), and Alexander Zverev (Z). The match results can be found on the webpage [59].

Figure 6 represents the directed graph $DG\left(A\right)$ and the supergraph, along with its levels. It can be seen that the results cannot be fully evaluated, as Medvedev was defeated by all the other competitors he played against; consequently, the graph $DG\left(A\right)$ is not strongly connected. Nevertheless, the graph $CG\left(A\right)$ is connected. The data structure analysis reveals two strongly connected components: one at level 0 containing seven players and another at level 1 consisting of a single player—Medvedev. Therefore, according to Theorem 6 IV, we can obtain the optimal limit point by evaluating the results of the 7 players among themselves and setting Medvedev’s coordinate to 0. Perturbation C and $Y/M$ insert 56 data and 30 data, respectively, while perturbation S only inserts 1: a win of Medvedev over Berrettini.

Applying the Bradley–Terry model, the optimal limit point, and the estimated weights evaluated by perturbations C, $Y/M$, and S with values $\epsilon =1$ and $\epsilon =0.001$, the results are contained in

Table 18. The optimal limit point and the estimated weights of the tennis players of 2019 ATP Finals based on the results of the matches evaluated by different perturbation methods applying $\epsilon =1$ using Bradley–Terry model.

r.	Player	Group Phase		opt.l.p.	C	Y/M	S
		group	r.	${\underline{\pi }}^{\left(L\right)}$	$\widehat{\underline{\pi }}\left(CA\left(1\right)\right)$	$\widehat{\underline{\pi }}\left(YA\left(1\right)\right)$	$\widehat{\underline{\pi }}\left(SA\left(1\right)\right)$
1	TS	group A	1	0.392	0.164	0.185	0.403
2	N	group A	3	0.205	0.137	0.157	0.276
3	TH	group B	1	0.136	0.136	0.138	0.091
4	Z	group A	2	0.107	0.125	0.137	0.131
5	F	group B	2	0.089	0.124	0.122	0.047
6	D	group B	3	0.036	0.111	0.093	0.016
7	B	group B	4	0.036	0.111	0.093	0.011
8	M	group A	4	0	0.091	0.078	0.025

and

Table 19. The optimal limit point and the estimated weights of the tennis players of 2019 ATP Finals based on the results of the matches evaluated by different perturbation methods applying $\epsilon =0.001$ using Bradley–Terry model.

r.	Player	Group Phase		opt.l.p.	C	Y/M	S
		group	r.	${\underline{\pi }}^{\left(L\right)}$	$\widehat{\underline{\pi }}\left(CA\left(0.001\right)\right)$	$\widehat{\underline{\pi }}\left(YA\left(0.001\right)\right)$	$\widehat{\underline{\pi }}\left(SA\left(0.001\right)\right)$
1	TS	group A	1	0.392	0.378	0.382	0.392
2	N	group A	3	0.205	0.200	0.204	0.207
3	TH	group B	1	0.136	0.138	0.137	0.134
4	Z	group A	2	0.107	0.110	0.109	0.108
5	F	group B	2	0.089	0.092	0.090	0.087
6	D	group B	3	0.036	0.039	0.038	0.035
7	B	group B	4	0.036	0.039	0.038	0.035
8	M	group A	4	0	0.004	0.002	0.001

, respectively. To obtain evaluable data, the inserted artificial comparison result is a Medvedev’s win over Berrettini when the S algorithm is applied. If a new match were to be scheduled, the most advisable pairing would be Medvedev versus Berrettini. The results of the evaluation in this case (assuming Medvedev’s win over Berrettini) are presented in the last column of Table 18. It would have been a conceivable result if they had played each other, since so far Medvedev had won three out of three matches against Berrettini.

Although the C and $Y/M$ perturbations do not alter the ranking based on the optimal limit point, the S perturbation results in a modified ranking. In this case, Medvedev moves ahead of Berrettini and Djokovic, which is understandable, since he has a win over Berrettini. Nadal’s strength increases due to the rise in Medvedev’s strength. Thiem’s strength decreases as a result of Berrettini’s weakened performance due to the assumed loss—so much so that he falls behind the strengthening Zverev in the rankings.

The rankings by the different methods coincide, and the weights computed by $\epsilon =0.001$ perturbation are close to the optimal limit point. The highest-ranked and the lowest-ranked player are the tournament winner and the last-place finisher, respectively. Thiem can be considered the runner-up in the tournament, as he reached the final but was defeated there, but applying Thurstone-motivated methods, he is only the third in weights. This can be explained by the fact that his wins, except for the one against Zverev, were against lower-ranked players: Djokovic and Berrettini. Apart from Tsitsipas, Thiem was also overtaken by Nadal, whose weights turned out to be surprisingly high. This is because Nadal was able to defeat the strongest player, Tsitsipas, and was defeated only by a mid-weight player, Zverev. The method’s ability to take into account the opponent’s strengths—not just the result achieved against them—is clearly demonstrated through this example as well.

The results obtained using this method can be used by the organizers when distributing the prize money, even though the tournament players cannot be ranked using the usual scoring method, or when determining the players to be placed in each seed group for organizing a new tournament.

We also attempted to evaluate the above match results using RankNet neural network. We conducted an extremely large number of experiments, but in many cases, the result was NaN, meaning the neural network did not produce an evaluation output. With an extremely low learning rate (${10}^{-7}$), we did obtain results, but the outcome still depended on which matches were placed in the training, validation, and testing sets. Therefore, evaluation using a neural network is not recommended at this stage.

9. Summary

In this paper, non-evaluable data were investigated in terms of their information content in two-option Thurstone-motivated models by applying a general cumulative distribution function. Since the maximum of the log-likelihood function was not attained in this case, we instead considered its supremum.

By finding trajectories that approach the supremum, we regarded the limits of their arguments as optimal limit points. As a theoretical result, we proved that these can be determined by analyzing the data structure, and their uniqueness can also be characterized on the basis of the data structure (Theorem 6).

Furthermore, we proved another statement (Theorem 7) that reveals a connection between perturbation methods and optimal limit points.

Based on knowledge of the data structure, we developed a new perturbation algorithm that inserts as few data points as possible to make the dataset evaluable. This knowledge also allows us to identify pairs for which real comparison data should be collected in order to make the available dataset evaluable.

Numerical experiments show that the convergence of the newly developed algorithm S to the optimal limit point was the best among the investigated perturbation methods.

This research has been conducted for two-option Thurstone-motivated models. It would be worthwhile to investigate the characteristics of non-evaluable data in multi-option models, to develop perturbation methods for them, and to compare these methods in terms of information content under different distance metrics.