A Weakly Pareto Compliant Quality Indicator

In multiand many-objective optimization problems, the optimization target is to obtain a set of non-dominated solutions close to the Pareto-optimal front, well-distributed, maximally extended and fully filled. Comparing solution sets is crucial in evaluating the performance of different optimization algorithms. The use of performance indicators is common in comparing those sets and, subsequently, optimization algorithms. Therefore, an effective performance indicator must encompass these features as a whole and, above all, it must be Pareto dominance compliant. Unfortunately, some of the known indicators often fail to properly reflect the quality of a solution set or cost a lot to compute. This paper demonstrates that the Degree of Approximation (DOA) quality indicator is a weakly Pareto compliant unary indicator that gives a good estimation of the match between the approximated front and the Pareto-optimal front.


Introduction
The optimized design of industrial applications is often problematic because of the simultaneous occurrence of many conflicting targets [1][2][3].In real-world optimization problems, the decision maker needs to have a wide range of solutions to choose from [4].Some optimization methods solve a single function obtained by aggregating different objective functions [5][6][7].The choice of weights is the major weakness to this approach [5].Other multi-and many-objective optimization algorithms (MOOAs) search for a non-dominated solution set [8][9][10], i.e., a set of multiple alternative solutions.This set is the Approximation Set in the decision space and the Approximated Pareto Front (APF) in the objective functions space.The main goal of such algorithms is to provide an APF matching the Pareto-optimal front (POF).The problem is to assess how well the approximated front fits the optimal one [11].The notion of optimization algorithms performance involves evaluating the quality of the solution and the required computational effort [12].This proves troublesome in the case of multiand many-objective optimization problems (MOOPs): a good approach would be to use a quality indicator (QI), i.e., a function of the APF that simplifies the quantitative performance comparison of different optimization algorithms.The simplest comparison method would be to check whether one APF is better than another with respect to the Pareto dominance relations [11].Thus, a QI must be able to account for Pareto dominance to properly compare two different algorithms.This is known as completeness with respect to Pareto dominance relations, and is the most desired property of a QI.More specifically, assuming there are two MOOAs providing, respectively, two non-dominated set A and B, a (weakly) Pareto compliant QI must comply with the following relation: if A dominates B then QI(A) must be better than QI(B).On the other hand, if there exists a scenario where A dominates B, but the QI(A) is worse than QI(B), then the QI is not Pareto compliant, therefore it could provide misleading results when it is used to compare MOOAs.Moreover, when APFs are incomparable with respect to Pareto dominance relations, more information is needed to compare the APFs provided by different MOOAs.Generally, a good MOOA should [13][14][15]: minimize the APF distance from the POF; 2.
obtain a good (usually uniform) distribution of the solutions found; 3.
maximize the APF extension i.e., for each objective the non-dominated solutions should cover a wide range of values (best case: the global optimum of each objective function must be found); 4. maximize the APF density, i.e., high cardinality for the approximation set is desirable.
Each goal represents a desired feature of the APF: in the following we refer to them as closeness, distribution, extension and cardinality, respectively.It is worth noticing that, given a scenario where: A is closer to the POF than B; 2.
the solutions in A are better distributed than the ones in B; 3.
A is more extended than B; 4.
the size of A is greater than the size of B, it is very probable that A dominates B, or, at least B does not dominate A.
Although it is improbable that the two APFs will be incomparable with respect to Pareto dominance relations, when this event occurs (an example is shown in Figure 1) QI(A) should be better than QI(B).In other words, the QI should reveal that A outperforms B with respect to all features and, consequently, A is preferable to B even though the two APFs are incomparable.
Math.Comput.Appl.2017, 22, 25 2 of 28 then the QI is not Pareto compliant, therefore it could provide misleading results when it is used to compare MOOAs.Moreover, when APFs are incomparable with respect to Pareto dominance relations, more information is needed to compare the APFs provided by different MOOAs.Generally, a good MOOA should [13][14][15]: 1. minimize the APF distance from the POF; 2. obtain a good (usually uniform) distribution of the solutions found; 3. maximize the APF extension i.e., for each objective the non-dominated solutions should cover a wide range of values (best case: the global optimum of each objective function must be found); 4. maximize the APF density, i.e., high cardinality for the approximation set is desirable.
Each goal represents a desired feature of the APF: in the following we refer to them as closeness, distribution, extension and cardinality, respectively.It is worth noticing that, given a scenario where: 1.A is closer to the POF than B; 2. the solutions in A are better distributed than the ones in B; 3. A is more extended than B; 4. the size of A is greater than the size of B, it is very probable that A dominates B, or, at least B does not dominate A.
Although it is improbable that the two APFs will be incomparable with respect to Pareto dominance relations, when this event occurs (an example is shown in Figure 1) QI(A) should be better than QI(B).In other words, the QI should reveal that A outperforms B with respect to all features and, consequently, A is preferable to B even though the two APFs are incomparable.In this case A and B are incomparable in terms of Pareto dominance, but A is preferable to B because A is closer to the Pareto-optimal front (POF), more extended, more populated, and better distributed.
A unary QI (UQI) estimates a non-dominated solution set quality by means of a real number [16]; then it is useful to estimate the effectiveness of a MOOA.Several known UQIs have no or limited completeness as regards Pareto dominance relations and are unable to take into account all the features listed previously.Few UQIs overcome these limitations despite needing much computational effort.
This paper demonstrates the ≻-completeness of the UQI, called Degree of Approximation (DOA) [17].Moreover, it proves its ability to take into account all four features.
To our knowledge, Hypervolume [1] is the only ⊳-complete UQI (then Hypervolume is a Pareto compliant UQI), and for this reason it is considered the best UQI for comparing optimization algorithms.Nevertheless, the relation A⊲B differs from A≺B since the former accounts for the case in which A contains some solutions of B but the probability of this specific event is very low, and it can be considered null when the objective functions' space belongs to the set of real numbers.Therefore, DOA can be used to evaluate the performance of optimization algorithms instead of the Hypervolume since it is proven that DOA is ≻-complete (then DOA is a weakly Pareto compliant UQI).A unary QI (UQI) estimates a non-dominated solution set quality by means of a real number [16]; then it is useful to estimate the effectiveness of a MOOA.Several known UQIs have no or limited completeness as regards Pareto dominance relations and are unable to take into account all the features listed previously.Few UQIs overcome these limitations despite needing much computational effort.
This paper demonstrates the -completeness of the UQI, called Degree of Approximation (DOA) [17].Moreover, it proves its ability to take into account all four features.
To our knowledge, Hypervolume [1] is the only -complete UQI (then Hypervolume is a Pareto compliant UQI), and for this reason it is considered the best UQI for comparing optimization algorithms.Nevertheless, the relation A B differs from A≺B since the former accounts for the case in which A contains some solutions of B but the probability of this specific event is very low, and it can be considered null when the objective functions' space belongs to the set of real numbers.Therefore, DOA can be used to evaluate the performance of optimization algorithms instead of the Hypervolume since it is proven that DOA is -complete (then DOA is a weakly Pareto compliant UQI).Note that the calculation of Hypervolume is difficult as the number of objective functions increases, while the calculation of DOA is usually very simple and fast even in the case of many-objective optimization.DOA calculation needs the knowledge of the POF, while the Hypervolume indicator needs to know the reference point.However, it is not a limitation when they are applied to compare the performance of different MOOAs.In fact, the best way to perform the comparisons is using properly designed benchmark problems for which the POF is known [18]: Schaffer's benchmark [19], Fonseca's benchmark [20], Poloni's benchmark [21], Kursawe's benchmark [22], ZDT1-6 [13], and DTLZ1-7 [23].
In the Evolutionary Multi-Objective community, UQIs are also adopted as the guidance mechanism during the optimization process, e.g., SMS-EMOEA [24] exploits the Hypervolume indicator during optimization.The use of a given UQI in indicator-based optimization algorithms enables one to implicitly scalarize the multi-objective problem.In this case, the compliance with the dominance relation of the indicator is one reason for the slow convergence rates of elitist multi-and many-objective evolutionary algorithms (MOEAs) [25].On the other hand, DOA is devised to evaluate the performance of the final set of non-dominated solutions and it is not devised to be included as guidance mechanism of a MOOA.Hence, in the DOA's application field, Pareto compliance is a suitable and necessary feature.
The paper is organized as follows.Section 2 recalls the definitions and terminology typically used in multi-objective optimization related to the Pareto dominance concept.Section 3 outlines the characteristics of a QI and presents a review of the most common UQIs.Section 4 describes DOA in detail, while Sections 5 and 6 mathematically demonstrate its -completeness and -completeness, respectively.Section 7 proves DOA compatibility with respect to the not better dominance relation.Finally, Section 8 validates DOA with some examples to highlight its accounting for closeness, distribution, extension and cardinality.Conclusions and future works are drawn in Section 9 and minor details of the proof in Section 6 have been reported in the appendices.

Multi-and Many-Objective Optimization Problem
Solving a MOOP means finding the optimal and feasible parameter configurations.A feasible solution (configuration) is called decision vector (x = x 1 ,x 2 ,...,x m ) and is a point in the decision space(X).An objective vector(y = y 1 ,y 2 ,...,y m ), that is a point in the objective space (Y), is linked to each decision vector by means of evaluating function f.So, a MOOP, with m decision variables (parameters to be set), n targets (objective functions to be optimized), and c constraints ( equality and care inequality constraints), can be mathematically represented as follows.
Maximize or minimize: subject to: where: Without loss of generality, in the following it is assumed that each objective function has to be minimized.

Pareto Dominance
Usually, in real-world MOOPs, there is no single parameter configuration that simultaneously optimizes all objective functions, i.e., a point does not exist in the decision space that is a global optimum.Thus, solving a real-world MOOP means offering the designer a set of alternative optimal solutions in the Pareto dominance sense.
Pareto dominance-A decision vector x 1 dominates another decision vector x 2 iff: This relation is denoted as When one or more of these relations is not satisfied, x 1 does not dominate x 2 ; this condition is denoted as x 1 ≺ x 2 .It is worth noticing that, for a single objective function, the standard relation less than is generally used to define the corresponding minimization problem, while the symbol ≺ represents a natural extension of < in the case of MOOPs [26].
Pareto optimality-A decision vector x' is said to be Pareto-optimal iff: The set that groups this kind of solution is known as a Pareto-optimal set, and all the solutions of this set are alternative, with no one solution being dominated by the other solutions.
In addition to dominance, other types of relation between the solutions can be defined: strictly dominance: a decision vector x 1 strictly dominates another decision vector x 2 (denoted as weakly dominance: a decision vector x 1 weakly dominates the decision vector x 2 (denoted as x 1 x 2 ) iff: Finally, when x 1 is better than x 2 with respect to a subset of objective functions but x 2 is better than x 1 with respect to another subset, the two solutions are said incomparable, denoted as x 1 ||x 2 (or x 2 ||x 1 ): Table 1 resumes the dominance relations.It is worth to note that a relation may imply other relations: Table 1.Dominance relations between two solutions [11].

Symbol Relation Description
x 1 ≺≺x 2 strictly dominance x 1 strictly dominates x 2 x 1 is better than x 2 with respect to each objective function x 1 is not worse than x 2 with respect to each objective function and x 1 is better than x 2 by at least one objective function x 1 x 2 weakly dominance x 1 weakly dominates x 2 x 1 is not worse than x 2 with respect to each objective function x 1 ||x 2 incomparability x 1 and x 2 are incomparable x 1 and x 2 do not weakly dominate each other By relating the solutions of one APF A to those of another APF B it is possible to extend the dominance relations between two solutions to two APFs.Table 2 shows the relations between two APFs.

Definitions
A quality indicator QI is a function q: S→ R, where S is the objective functions space, that assigns a real value to a set of APFs belonging to S related to a MOOP.When the function q has just one argument (i.e., one APF), the quality indicator is called unary, when it has two arguments (i.e., two APFs) it is called binary, and so on.
The aim of a QI is to compare APFs and so QIs are mainly used to indicate if a MOOA works any better than others.Some QIs can also be applied as the acceptance criterion to the selection operator of the stochastic search algorithms [27], but DOA is not devised for such a scope.

Comparison Methods
This paper focuses on the use of QIs for evaluating the performances of different optimization algorithms.To do this the QI results must be interpreted by means of an interpretation function E: R q → Bool, where q depends on the size of the QI set. Figure 2 shows some examples of interpretation functions (A and B are two APFs).

Definitions
A quality indicator QI is a function q: S→ , where S is the objective functions space, that assigns a real value to a set of APFs belonging to S related to a MOOP.When the function q has just one argument (i.e., one APF), the quality indicator is called unary, when it has two arguments (i.e., two APFs) it is called binary, and so on.
The aim of a QI is to compare APFs and so QIs are mainly used to indicate if a MOOA works any better than others.Some QIs can also be applied as the acceptance criterion to the selection operator of the stochastic search algorithms [27], but DOA is not devised for such a scope.

Comparison Methods
This paper focuses on the use of QIs for evaluating the performances of different optimization algorithms.To do this the QI results must be interpreted by means of an interpretation function E: q → Bool, where q depends on the size of the QI set. Figure 2 shows some examples of interpretation functions (A and B are two APFs).
Finally, the combination of a quality indicator, I, and an interpretation function, E, is called a comparison method [11], and is referred to as CI,E:

CI,E(A,B) = E(I(A),I(B)).
Comparison method in case of one QI (a) and in case of a vector of q QIs (b).

Compatibility and Completeness
Usually, one or a set of QIs can be useful to compare different optimization algorithms to figure out which works better on a particular class of problems.
Non-dominated solutions are preferred to the dominated ones from the designer's point of view.Then, when a comparison method shows that APF A is preferable to APF B, A must be better than B. In a similar way, when A is better than B, a comparison method must indicate that A is preferable to B. Such features are known as ⊳-compatibility and ⊳-completeness [11].
Let ►be an arbitrary dominance relation among those defined in Table 2 (≻≻ or ≻ or ⊳).A comparison method CI,E is said ►-compatible if for each possible pair of APFs A and B:   , , is true Finally, the combination of a quality indicator, I, and an interpretation function, E, is called a comparison method [11], and is referred to as C I,E : C I,E (A,B) = E(I(A),I(B)).

Compatibility and Completeness
Usually, one or a set of QIs can be useful to compare different optimization algorithms to figure out which works better on a particular class of problems.
Non-dominated solutions are preferred to the dominated ones from the designer's point of view.Then, when a comparison method shows that APF A is preferable to APF B, A must be better than B. In a similar way, when A is better than B, a comparison method must indicate that A is preferable to B. Such features are known as -compatibility and -completeness [11].
Let be an arbitrary dominance relation among those defined in Table 2 ( or or ).A comparison method C I,E is said -compatible if for each possible pair of APFs A and B: A comparison method C I,E is said -complete if for each possible pair of APFs A and B: It has been demonstrated [11] that a comparison method based on a UQI (or on a finite combination of UQIs) that is both -compatible and -complete cannot exist.Moreover, Pareto dominance is sufficient but not necessary to consider an APF preferable to another: there are pairs of APFs with considerable quality difference which are considered, by Pareto dominance relations, as not comparable [16].Hence, if a comparison method based on UQI were -compatible, the indicator could not provide any preference in the case of two incomparable APFs.Therefore, it would be better if the UQI were only compatible with [28] and it should take into account all the features (closeness, distribution, extension, cardinality) that are desirable for an APF.
Finally, while a comparison method -complete is necessary (i.e., when APF A is better than APF B the comparison method must highlight it), when a comparison method shows that A is preferable to B, one of the following two cases must hold:

•
A is better than B (A B); A and B are incomparable and A outperforms B with respect to closeness, distribution, extension and cardinality.

Closeness, Distribution, Extension, and Cardinality
The main target of an optimization algorithm to solve a MOOP is to find an APF as similar as possible to the POF.Hence, as said before, the APF must be: • close to the POF; Figure 3 represents the extreme cases: an APF exhibiting good closeness only, and an APF with all good features but not close to the POF; • well distributed (usually uniform); Figure 4 shows an APF exhibiting a uniform distribution only and an APF with all good features but not uniformly distributed; • very extended (in the best case the global optimum of each objective function belongs to the APF); Figure 5 shows an APF with only a good extension and one with all good features but not extended; • of high cardinality; Figure 6 shows an APF with good cardinality only and an APF with all good features but poor cardinality.
Figure 7 shows an APF with all the desired features.A good QI must take into account all these features to give a correct measure of APF quality.


very extended (in the best case the global optimum of each objective function belongs to the APF); Figure 5 shows an APF with only a good extension and one with all good features but not extended;  of high cardinality; Figure 6 shows an APF with good cardinality only and an APF with all good features but poor cardinality.
Figure 7 shows an APF with all the desired features.A good QI must take into account all these features to give a correct measure of APF quality.Table 3 points out if a specific feature partially  or totally  affects the value of some UQIs.A heuristic approach has been applied to determine whether a feature (closeness, distribution, extension, cardinality) affects the QI value.In particular, an APF B obtained by improving a given feature of another APF A is expected to have an indicator value better than that of A when the indicator is sensitive to this feature.For example, if an APF is gradually moved towards the POF and the indicator increasingly improves, then the indicator is influenced by the closeness feature.An indicator is partially affected by a feature when it sometimes improves and other times does not change.
The Inverted Generational Distance indicator (IGD) [29] and a similar indicator using a weighted-scalar distance function called Average Distance from Reference Set indicator (ADRS) [30], the Completeness indicator [31,32], the Potential Function indicator [27] and the Hypervolume indicator [1] account for all the features but they present some drawbacks.The IGD and ADRS indicators have the same complexity of DOA, but they are ≻≻-complete only [11].The Δp indicator [39] can be considered as the combination of slight variations of the Generational Distance and the IGD indicator [25].Therefore, it presents the advantage and limitations of these indicators, that is it accounts for all the features but it is ≻≻-complete only.
The Completeness and Potential Function indicators are as ≻-complete as the DOA indicator.Nevertheless, the Completeness indicator cannot be directly computed, but can be estimated by drawing samples from the feasible set and computing completeness for these samples.The confidence interval for the true value can be evaluated with any reliability value, given sufficiently large samples [26].For the Potential Function indicator similar considerations hold.Hence, the drawback of both indicators is the high computational cost.
To our knowledge, Hypervolume is the only ⊳-complete UQI, and so is considered the best UQI for comparing optimization algorithms.Nevertheless, the relation A⊲B differs from A≺B since the former accounts for the case in which B contains some solutions of A but the probability of this specific event is very low, and it can be considered null when the objective functions' space belongs to the set of real numbers.
or totally affects the value of some UQIs.A heuristic approach has been applied to determine whether a feature (closeness, distribution, extension, cardinality) affects the QI value.In particular, an APF B obtained by improving a given feature of another APF A is expected to have an indicator value better than that of A when the indicator is sensitive to this feature.For example, if an APF is gradually moved towards the POF and the indicator increasingly improves, then the indicator is influenced by the closeness feature.An indicator is partially affected by a feature when it sometimes improves and other times does not change.
Table 3. Summary of selected UQIs and features that influence their value.

Closeness Distribution Extension Cardinality
Average Distance from Reference Set [30] Chi-Square-Like Deviation Measure [33] Math.Comput.Appl.2017, 22, 25 8 of 28 Table 3 points out if a specific feature partially  or totally  affects the value of some UQIs.A heuristic approach has been applied to determine whether a feature (closeness, distribution, extension, cardinality) affects the QI value.In particular, an APF B obtained by improving a given feature of another APF A is expected to have an indicator value better than that of A when the indicator is sensitive to this feature.For example, if an APF is gradually moved towards the POF and the indicator increasingly improves, then the indicator is influenced by the closeness feature.An indicator is partially affected by a feature when it sometimes improves and other times does not change.
The Inverted Generational Distance indicator (IGD) [29] and a similar indicator using a weighted-scalar distance function called Average Distance from Reference Set indicator (ADRS) [30], the Completeness indicator [31,32], the Potential Function indicator [27] and the Hypervolume indicator [1] account for all the features but they present some drawbacks.
The IGD and ADRS indicators have the same complexity of DOA, but they are ≻≻-complete only [11].The Δp indicator [39] can be considered as the combination of slight variations of the Generational Distance and the IGD indicator [25].Therefore, it presents the advantage and limitations of these indicators, that is it accounts for all the features but it is ≻≻-complete only.
The Completeness and Potential Function indicators are as ≻-complete as the DOA indicator.Nevertheless, the Completeness indicator cannot be directly computed, but can be estimated by drawing samples from the feasible set and computing completeness for these samples.The confidence interval for the true value can be evaluated with any reliability value, given sufficiently large samples [26].For the Potential Function indicator similar considerations hold.Hence, the drawback of both indicators is the high computational cost.
To our knowledge, Hypervolume is the only ⊳-complete UQI, and so is considered the best Math.Comput.Appl.2017, 22,25 Table 3 points out if a specific feature partially  or totally  affects the value of some heuristic approach has been applied to determine whether a feature (closeness, distr extension, cardinality) affects the QI value.In particular, an APF B obtained by improving feature of another APF A is expected to have an indicator value better than that of A w indicator is sensitive to this feature.For example, if an APF is gradually moved towards the P the indicator increasingly improves, then the indicator is influenced by the closeness feat indicator is partially affected by a feature when it sometimes improves and other times d change.
The Inverted Generational Distance indicator (IGD) [29] and a similar indicator weighted-scalar distance function called Average Distance from Reference Set indicator (AD the Completeness indicator [31,32], the Potential Function indicator [27] and the Hype indicator [1] account for all the features but they present some drawbacks.The IGD and ADRS indicators have the same complexity of DOA, but they are ≻≻-c only [11].The Δp indicator [39] can be considered as the combination of slight variation Generational Distance and the IGD indicator [25].Therefore, it presents the advanta limitations of these indicators, that is it accounts for all the features but it is ≻≻-complete on The Completeness and Potential Function indicators are as ≻-complete as the DOA in Nevertheless, the Completeness indicator cannot be directly computed, but can be estim drawing samples from the feasible set and computing completeness for these samp confidence interval for the true value can be evaluated with any reliability value, given suf large samples [26].For the Potential Function indicator similar considerations hold.He drawback of both indicators is the high computational cost.
To our knowledge, Hypervolume is the only ⊳-complete UQI, and so is considered Completeness Indicator [31,32] Enclosing Hypercube [11] Math. Comput.Appl.2017, 22, 25 8 of 28 Table 3 points out if a specific feature partially  or totally  affects the value of some UQIs.A heuristic approach has been applied to determine whether a feature (closeness, distribution, extension, cardinality) affects the QI value.In particular, an APF B obtained by improving a given feature of another APF A is expected to have an indicator value better than that of A when the indicator is sensitive to this feature.For example, if an APF is gradually moved towards the POF and the indicator increasingly improves, then the indicator is influenced by the closeness feature.An indicator is partially affected by a feature when it sometimes improves and other times does not change.
The Inverted Generational Distance indicator (IGD) [29] and a similar indicator using a weighted-scalar distance function called Average Distance from Reference Set indicator (ADRS) [30], the Completeness indicator [31,32], the Potential Function indicator [27] and the Hypervolume indicator [1] account for all the features but they present some drawbacks.The IGD and ADRS indicators have the same complexity of DOA, but they are ≻≻-complete only [11].The Δp indicator [39] can be considered as the combination of slight variations of the Generational Distance and the IGD indicator [25].Therefore, it presents the advantage and limitations of these indicators, that is it accounts for all the features but it is ≻≻-complete only.
The Completeness and Potential Function indicators are as ≻-complete as the DOA indicator.Nevertheless, the Completeness indicator cannot be directly computed, but can be estimated by drawing samples from the feasible set and computing completeness for these samples.The confidence interval for the true value can be evaluated with any reliability value, given sufficiently large samples [26].For the Potential Function indicator similar considerations hold.Hence, the Generational Distance [34] Hypervolume [1] Inverted Generational Distance [29] M 1 * [13] M 2 * [13] Math.Comput.Appl.2017, 22,25 Table 3 points out if a specific feature partially  or totally  affects the v heuristic approach has been applied to determine whether a feature (cl extension, cardinality) affects the QI value.In particular, an APF B obtained feature of another APF A is expected to have an indicator value better than indicator is sensitive to this feature.For example, if an APF is gradually moved the indicator increasingly improves, then the indicator is influenced by the indicator is partially affected by a feature when it sometimes improves and change.
The Inverted Generational Distance indicator (IGD) [29] and a simil weighted-scalar distance function called Average Distance from Reference Set i the Completeness indicator [31,32], the Potential Function indicator [27] an indicator [1] account for all the features but they present some drawbacks.

Spacing [37]
 Unary ε-Indicator [11,26]  Uniform Distribution [38]  Worst Distance from Reference Set [30]  Δ [8]  The IGD and ADRS indicators have the same complexity of DOA, but t only [11].The Δp indicator [39] can be considered as the combination of sl Generational Distance and the IGD indicator [25].Therefore, it presents limitations of these indicators, that is it accounts for all the features but it is ≻≻ The Completeness and Potential Function indicators are as ≻-complete a Maximum Pareto Front Error [34] Outer Diameter [26] Overall Nondominated Vector Generation [34] Overall Pareto Spread [35] Potential Function [27] Seven Points Average Distance [36] Math.Comput.Appl.2017, 22, 25 8 of 28 Table 3 points out if a specific feature partially  or totally  affects the value of some UQIs.A heuristic approach has been applied to determine whether a feature (closeness, distribution, extension, cardinality) affects the QI value.In particular, an APF B obtained by improving a given feature of another APF A is expected to have an indicator value better than that of A when the indicator is sensitive to this feature.For example, if an APF is gradually moved towards the POF and the indicator increasingly improves, then the indicator is influenced by the closeness feature.An indicator is partially affected by a feature when it sometimes improves and other times does not change.
The Inverted Generational Distance indicator (IGD) [29] and a similar indicator using a weighted-scalar distance function called Average Distance from Reference Set indicator (ADRS) [30], the Completeness indicator [31,32], the Potential Function indicator [27] and the Hypervolume indicator [1] account for all the features but they present some drawbacks.∆ p [39,40] The Inverted Generational Distance indicator (IGD) [29] and a similar indicator using a weighted-scalar distance function called Average Distance from Reference Set indicator (ADRS) [30], the Completeness indicator [31,32], the Potential Function indicator [27] and the Hypervolume indicator [1] account for all the features but they present some drawbacks.
The IGD and ADRS indicators have the same complexity of DOA, but they are -complete only [11].The ∆ p indicator [39] can be considered as the combination of slight variations of the Generational Distance and the IGD indicator [25].Therefore, it presents the advantage and limitations of these indicators, that is it accounts for all the features but it is -complete only.The Completeness and Potential Function indicators are as -complete as the DOA indicator.Nevertheless, the Completeness indicator cannot be directly computed, but can be estimated by drawing samples from the feasible set and computing completeness for these samples.The confidence interval for the true value can be evaluated with any reliability value, given sufficiently large samples [26].For the Potential Function indicator similar considerations hold.Hence, the drawback of both indicators is the high computational cost.
To our knowledge, Hypervolume is the only -complete UQI, and so is considered the best UQI for comparing optimization algorithms.Nevertheless, the relation A B differs from A≺B since the former accounts for the case in which B contains some solutions of A but the probability of this specific event is very low, and it can be considered null when the objective functions' space belongs to the set of real numbers.
Moreover, Hypervolume running time grows exponentially with the number of objective functions [41][42][43][44].The most obvious method for calculating Hypervolume is the inclusion-exclusion algorithm, with complexity O(n2 m ), where n is the number of objectives and m is the number of APF points.The fastest methods for calculating Hypervolume (e.g., LebMeasure [45], Hypervolume by Slicing Objectives algorithm [46]) lead to a O(m 2 n 3 )complexity.The DOA indicator has a lower computational cost, presenting a O(nMm) complexity, where M is the number of POF points.

The Weakly Pareto Compliant Quality Indicator
The comparison method based on DOA and its associated interpretation function is -complete.In detail, for a given POF and an APF A, DOA is computed as follows.
First, given a solution i belonging to the POF, D i,A , is determined from the sub-set of A containing the solutions dominated by i (Figure 8).Hence, if the number of solutions belonging to D i,A is not null, for each approximated solution a∈D i,A the Euclidean distance df i,a between a and i is computed as: with: n number of objective functions, f k,a value of k-th objective function of the approximated solution a, f k,i value of k-th objective function of optimal solution i. si,A = min(di,A,ri,A) (18) the DOA indicator for the APF A is computed as:  Euclidean distance d i,A (Figure 9) between i and the nearest approximated solution belonging to D i,A is computed in the objective function space as: where |D i,A | is the number of solutions belonging to D i,A .Another quantity r i,A (similarly to d i,A ) is computed for i considering the solutions of A not dominated by i (i.e., A\D i,A ): where rf i,a is a reduced distance (Figure 10) between i and a non-dominated solution a of A (i.e., ∀a∈ A: i || a), i.e., computed only for objectives k with f k,a ≥ f k,i : Note that, rf i,a is equal to df i,a when a ∈ D i,A .Moreover, defining n a (n a < n) as the number of functions for which the f k,a − f k,i ≥ 0 (f k,a ≥ f k,i , k = 1,..,n a and f k,a < f k,i , k = n a + 1,..,n) expression ( 16) can be rewritten as: Finally, defining the DOA indicator for the APF A is computed as: DOA and IGD present some similarities.More specifically, if d i,A , Equation ( 14), is computed considering all solutions in A regardless of the dominance relation with i, and only this quantity is considered then the IGD indicator is obtained, that is: The IGD indicator is not -complete while DOA is -complete, as will be proved in the following.Therefore, the use of r i,A is the enabling key for DOA -completeness.While the Hypervolume indicator needs to know the reference point, the DOA calculation needs the knowledge of the POF, like the IGD indicator.This is not a drawback in MOOA benchmarking which is usually carried out for problems with known POF.More specifically, the Pareto optimal solutions and the analytical expression of the objective functions are usually known.Therefore, the POF is obtained by points uniformly discretizing the decision parameter intervals where the Pareto optimal solutions are located.The value of DOA does not depend on the number of points in the POF if |POF|>>|APF|.
Considering two APFs A and B, the proposed quality indicator needs interpreting [11] to affirm either that A is preferable to B or B is preferable to A or A and B are equivalent: the proposed obvious interpretation function is illustrated in Figure 11.Moreover, DOA changes with arbitrary scaling of the objective functions, since the DOA indicator is a distance-based metric, while the relationship between DOA(A) and DOA(B) does not change.DOA and IGD present some similarities.More specifically, if di,A, Equation ( 14), is computed considering all solutions in A regardless of the dominance relation with i, and only this quantity is considered then the IGD indicator is obtained, that is: The IGD indicator is not ≻-complete while DOA is ≻-complete, as will be proved in the following.Therefore, the use of ri,A is the enabling key for DOA ≻-completeness.
While the Hypervolume indicator needs to know the reference point, the DOA calculation needs the knowledge of the POF, like the IGD indicator.This is not a drawback in MOOA benchmarking which is usually carried out for problems with known POF.More specifically, the Pareto optimal solutions and the analytical expression of the objective functions are usually known.Therefore, the POF is obtained by points uniformly discretizing the decision parameter intervals where the Pareto optimal solutions are located.The value of DOA does not depend on the number of points in the POF if |POF|>>|APF|.
Considering two APFs A and B, the proposed quality indicator needs interpreting [11] to affirm either that A is preferable to B or B is preferable to A or A and B are equivalent: the proposed obvious interpretation function is illustrated in Figure 11.Moreover, DOA changes with arbitrary scaling of the objective functions, since the DOA indicator is a distance-based metric, while the relationship between DOA(A) and DOA(B) does not change.
In the following, it is demonstrated that A≺B implies A is preferable to B, that is DOA(A) < DOA(B), in order to affirm that DOA is a ≻-complete quality indicator.
For the sake of clarity, the ≻≻-completeness of DOA is demonstrated before proving its ≻-completeness.

≻≻-Completeness
DOA is a ≻≻-complete quality indicator if DOA(A)<DOA(B) for any pair of APF A and B, with A≺≺B.
In the hypothesis that A≺≺B, each solution of B is strictly dominated by, at least, one solution of A. To demonstrate that DOA(A)<DOA(B) is sufficient to prove that si,A is always lesser than si,B for each point i ∊ POF.In other words, if Considering a point i ∊POF, in the following, b indicates the solution belonging to B which provides si,B and a a solution of A that strictly dominates b (a≺≺b); only four scenarios are possible (see Figure 12): In the following, it is demonstrated that A≺B implies A is preferable to B, that is DOA(A) < DOA(B), in order to affirm that DOA is a -complete quality indicator.
For the sake of clarity, the -completeness of DOA is demonstrated before proving its -completeness.

-Completeness
DOA is a -complete quality indicator if DOA(A)<DOA(B) for any pair of APF A and B, with A≺≺B.
In the hypothesis that A≺≺B, each solution of B is strictly dominated by, at least, one solution of A. To demonstrate that DOA(A)<DOA(B) is sufficient to prove that s i,A is always lesser than s i,B for each point i ∈ POF.In other words, then Considering a point i ∈POF, in the following, b indicates the solution belonging to B which provides s i,B and a a solution of A that strictly dominates b (a≺≺b); only four scenarios are possible (see Figure 12): Note that the other scenarios i≼b Λ i≼a and i||b Λ i≼a are not possible because a≺≺b: in fact, from either i≼b Λ a≺≺b and i||b Λ a≺≺b follows i⋠a.
Moreover, it is worth mentioning the following: Remark 1.
i≼a implies that si,A ≤ dfi,a, in detail: a (where a*∊A and a*≠a) or if si,A = ri,A (this implies that ri,A < di,A ≤ dfi,a).

Remark 2. i||a implies that si,A ≤ rfi,a, in detail:
 si,A = rfi,a iff si,A = ri,A Λ ri,A = rfi,a;  si,A < rfi,a either if si,A = ri,A Λ ri,A = rfi,a* < rfi,a (where a*∊A and a*≠a) or if si,A = di,A (this implies that di,A < ri,A ≤ rfi,a).
Finally, the inequality si,A < si,B will be proved for the four scenarios A1-D1: this inequality naturally implies the ≻≻-completeness of the DOA indicator.Note that the other scenarios i b Λ i a and i||b Λ i a are not possible because a≺≺b: in fact, from either i b Λ a≺≺b and i||b Λ a≺≺b follows i a.

A1. i≺≺b Λ i≼a
Moreover, it is worth mentioning the following: Remark 1. i a implies that s i,A ≤ df i,a , in detail: (where a*∈A and a* =a) or if s i,A = r i,A (this implies that r i,A < d i,A ≤ df i,a ).
Remark 2. i||a implies that s i,A ≤ rf i,a , in detail: • s i,A = rf i,a iff s i,A = r i,A Λ r i,A = rf i,a ; • s i,A < rf i,a either if s i,A = r i,A Λ r i,A = rf i,a* < rf i,a (where a*∈A and a* =a) or if s i,A = d i,A (this implies that d i,A < r i,A ≤ rf i,a ).
Finally, the inequality s i,A < s i,B will be proved for the four scenarios A1-D1: this inequality naturally implies the -completeness of the DOA indicator.

A1. i≺≺b Λ i a
In this case, i strictly dominates b then s i,B = d i,B = df i,b , because b is the solution which provides s i,B .Moreover, i a implies that s i,A ≤ df i,a (see Remark 1).
So, in order to demonstrate that s i,A < s i,B it is sufficient to demonstrate that df i,a < df i,b .
Recalling that the following inequalities hold:

B1. i≺≺b Λ i||a
In this case, i strictly dominates b then s i,B = d i,B = df i,b , because b is the solution which provides s i,B .Moreover, i||a implies that s i,A ≤ rf i,a (see Remark 2).
So, in order to demonstrate that s i,A < s i,B it is sufficient to demonstrate that rf i,a < df i,b .Proof is given in the next section because scenario C encompasses scenario B.

C1. i≺b Λ i||a
In this case, i dominates b then s i,B = d i,B = df i,b , because b is the solution which provides s i,B .Moreover, i || a implies that s i,A ≤ rf i,a (see Remark 2).
So, in order to demonstrate that s i,A < s i,B it is sufficient to demonstrate that rf i,a < df i,b .Ordering the n objective functions of solution a in such a way that the first n a (with n a < n) are greater than those of i and recalling that: then the following inequalities hold:

D1. i||b Λ i||a
In this case, i and b are incomparable then s i,B = r i,B = rf i,b , because b is the solution which provides s i,B .Moreover, i||a implies that s i,A ≤ rf i,a (see Remark 2).
So, in order to demonstrate that s i,A < s i,B it is sufficient to demonstrate that rf i,a < rf i,b .
Ordering the n objective functions of solution a in such a way that the first n a are greater than those of i, ordering the n objective functions of solution b in such a way that the first n b are greater than those of i (with n a ≤ n b < n, since a b Λ i||a implies n a ≤ n b , while i||b implies n b < n) and recalling that: then the following inequalities hold:

-Completeness
In this section is proved that DOA is a -complete quality indicator.Consider any pair of APF A and B, with A≺B, the -completeness of DOAis demonstrated by proving that s i,A is never greater than s i,B (for each point i∈POF) and always exists a point i*∈ POF for which s i*,A is lesser than s i*,B : then In the following, b indicates the solution belonging to B which provides s i,B for a point i ∈ POF and a a solution of A that dominates b (a≺b).Moreover, the -completeness of DOA is proved in the worst and most general case, i.e., when ∀ b∈B a∈A: a≺≺b (i.e., a≺b Λ a ≺ ≺b, limit case); only five scenarios are possible (see Figure 13):  So, in order to demonstrate that si,A < si,Bit is sufficient to demonstrate that dfi,a < dfi,b.
Recalling that: Note that the scenario i||b Λ i a is not possible because a≺b: in fact, from i||b Λ a≺b follows i a.Moreover, scenario A2 does not include i = a, differently from section 5, because in this case i≺≺b while a≺b Λ a ≺ ≺b.Analogously, differently from section 5, when i≺b the scenario i a is possible.Finally, the scenario i b is not considered because A≺B, in fact i = b would lead to the absurd a≺b=i.
In order to demonstrate the -completeness of DOA, it is proved that the inequality s i,A < s i,B is verified ∀i for the three scenarios A2, B2 and C2.While for the remaining two scenarios D2 and E2 we will prove that the following two sufficient conditions hold: For the sake of simplicity, the proof of β will be given in Appendix A.

A2. i≺≺b Λ i≺a
In this case, i strictly dominates b then s i,B = d i,B = df i,b , because b is the solution which provides s i,B .Moreover, i≺a implies that s i,A ≤ df i,a (see Remark 1).
So, in order to demonstrate that s i,A < s i,B it is sufficient to demonstrate that df i,a < df i,b .

B2. i≺≺b Λ i||a
In this case, i strictly dominates b then s i,B = d i,B = df i,b , because b is the solution which provides s i,B .Moreover, i||a implies that s i,A ≤ rf i,a (see Remark 2).
So, in order to demonstrate that s i,A < s i,B it is sufficient to demonstrate that rf i,a < df i,b .
Ordering the n objectives f of solution a in such a way that the first n a (with n a < n) are greater than those of i and recalling that: (29)

C2. i≺b Λ i a
In this case, i dominates b then s i,B = d i,B = df i,b , because b is the solution which provides s i,B .Moreover, i a implies that s i,A ≤ df i,a (see Remark 1).
So, in order to demonstrate that s i,A < s i,B it is sufficient to demonstrate that df i,a < df i,b .
Recalling that: then the following inequalities hold:

D2. i≺b Λ i||a
In this case, i dominates b then s i,B = d i,B = df i,b , because b is the solution which provides s i,B .Moreover, i||a implies that s i,A ≤ rf i,a (see Remark 2).
So, in order to demonstrate that s i,A ≤ s i,B it is sufficient to demonstrate that rf i,a ≤ df i,b .
Ordering the n objectives f of solution a in such a way that the first n a (with n a < n) are greater than those of i and recalling that: Note that rf i,a is strictly lesser than df i,b when 1 ≤ j ≤ n a , since , . . .,n.In this case, obviously, f h,i < f h,a = f h,b with 1 ≤ h ≤ n a and f j,a < f j,i = f j,b with j > n a .As said before, the proof that ∃ i*∈POF: s i*,A < s i*,B has been reported in Appendix A.

E2. i||b Λ i||a
In this case, i and b are incomparable then s i,B = r i,B = rf i,b , because b is the solution which provides s i,B .Moreover, i||a implies that s i,A ≤ rf i,a (see Remark 2).
So, in order to demonstrate that s i,A ≤ s i,B, it is sufficient to demonstrate that rf i,a ≤ rf i,b .Ordering the n objectives f of solution a in such a way that the first n a are greater than those of i, ordering the n objectives f of solution b in such a way that the first n b are greater than those of i (with n a ≤ n b < n, since a≺b Λ i||a implies n a ≤ n b , while i||b implies n b < n) and recalling that: then the following inequalities hold: It is worth noting that: In the last case, rf i,a = rf i,b iff f k,a = f k,b ∀ k = 1,..,n a .In this case, obviously, f j,a < f j,b ≤ f j,i whit j > n a .As said before, the proof that ∃ i*∈POF: s i*,A < s i*,B has been reported in Appendix A.

-Compatibility
In this section it is proved that DOA is a -compatible quality indicator.The following remarks are necessary.Remark 3. A solution a' belonging to A does not influence DOA(A) if, ∀ i∈POF: • s i,A < df i,a' when a' is dominated by i; • s i,A < rf i,a' when a' is not dominated by i.
In fact, in this case s i,A = df i,a' and s i,A = rf i,a' , by which follows that DOA(A) does not change its value if a' is moved off from A. These means that an APF B = A a'} has the same quality indicator value, i.e., DOA(A) = DOA(B).Using DOA seems B equivalent to A while A, having one more solution, is better than B [11].Then the proposed method is not a -complete quality indicator.On the other hand, DOA together with its interpretation function is a -complete comparison method, as it is demonstrated before.

Remark 4.
The relation A B is equivalent to assume that B = C D, where C A, D A = Ø and ∀ b∈D ∃ a∈A: a≺b.Note that, when C = Ø then A≺B.Hence, the difference between case A≺B and A B is that in the latter B could contains some solutions of A. Moreover, obviously, only the last case includes B A (i.e., D = Ø).For these reasons when A B, DOA(A) is never greater than DOA(B).

Remark 5. The relation A||B involves that DOA(A) can be greater or less than or equal to DOA(B).
For two generic APF A and B, the -completeness of DOA, together with Remarks 3-5, imply that if DOA(A) < DOA(B) then A≺B or A B or A||B, i.e., it is sure that B A. Then DOA is a -compatible unary quality indicator.

DOA Validation
To demonstrate that DOA takes into account all features (closeness, distribution, extension, and cardinality) three typical POFs have been considered:

•
convex and connected; • non-convex and connected; • convex and disconnected.
In particular, the DOA unary quality indicator has been computed for different examples of APFs.Both POFs and APFs are drawn in Figures 14-16.Table 4 shows the values of DOA for the APFs considered.
Let us examine the strategy adopted to choose the examples of APF for each POF: the APF B obtained by improving a given feature of APF A is expected to have a better indicator value than that of A if the indicator is sensitive to this feature.For example, if an APF is gradually moved towards the POF and the indicator increasingly improves, then the indicator is influenced by the closeness feature.The validation was performed according to these considerations.Hence, from an APF (APF1, indicated in the figures by symbol♦) with only poor features, the second APF (APF2, indicated by +) is created by improving APF1 closeness that is by converging APF1 on the POF.Therefore, APF2 has better closeness than APF1 but the same distribution, extension and cardinality.The third APF (APF3, indicated by ) is obtained by improving the distribution of APF2 by uniformly distributing the solutions of APF2.APF3 has better distribution than APF2 but the same closeness, extension and cardinality.The fourth APF (APF4, indicated by ) is created by improving the extension of APF3, preserving the other features.Finally, the fifth APF (APF5, indicated by ) is created by adding new points to APF4, and hence improving the cardinality of APF5 with respect to the fourth APF.It is worth noting that it does not matter in which order the different features are added to the initial APF1, and the order used in Table 4 simply follows the features' description given in the Introduction.

9.Conclusions and Future Work
Evaluating the performance of MOOAs is very difficult because their comparison involves comparing APFs.QIs are used to measure the goodness of the APF provided by different optimization algorithms to highlight which works better.Whatever the characteristics of the POF, such a strategy highlights that the value of DOA decreases when one feature improves.
The results in Table 4 highlight the -completeness of DOA too.In fact, the first APF is dominated by the others it has a DOA value worse than those of the other APFs.

Conclusions and Future Work
Evaluating the performance of MOOAs is very difficult because their comparison involves comparing APFs.QIs are used to measure the goodness of the APF provided by different optimization algorithms to highlight which works better.
Therefore, a QI must be able to account for Pareto dominance to properly compare two different algorithms.Moreover, when APFs are incomparable, further data (closeness, distribution, extension and cardinality) must be taken into account to compare the APFs provided by different MOOAs.Few UQIs are -complete and able to account for the aforementioned features, yet they require great computational effort.
This paper has described the DOA unary quality indicator, which could be very useful in assessing the performance of an MOOA by estimating the match between the approximation front found by the MOOA and the optimal one.It has been proven that it is -complete, -compatible, and requires little computational cost.Moreover, a numerical validation was carried out to demonstrate that it accounts for closeness, distribution, extension, and cardinality.An implementation of the DOA indicator is available online: http://wwwelfin.diees.unict.it/esg/DOA.html.
The major drawback of the proposed indicator is that it is not -complete.Even though the dominance relation A B differs from A ≺ B since the former accounts for the case in which B contains some solutions of A, the probability of this specific event is next to nil.Nevertheless, a future development of this work is to properly modify DOA to make it -complete, maintaining low computational complexity.In this perspective, the underlying reasons leading from a not -complete indicator, IGD, to the similar one proposed in this paper, DOA, having -completeness can be the enabling keys.
Moreover, an integral formulation of DOA can be developed to deal with continuous POFs that can be expressed by an analytical function.
Finally, innovative studies have proposed interesting strategies to use UQIs requiring the knowledge of the POF as the guidance mechanism for indicator-based MOEAs.Therefore, the use of DOA for indicator-based evolutionary algorithms is another attractive development of the work.
Figure 1.In this case A and B are incomparable in terms of Pareto dominance, but A is preferable to B because A is closer to the Pareto-optimal front (POF), more extended, more populated, and better distributed.

Figure 1 .
Figure 1.In this case A and B are incomparable in terms of Pareto dominance, but A is preferable to B because A is closer to the Pareto-optimal front (POF), more extended, more populated, and better distributed.

Figure 2 .
Figure 2. Comparison method in case of one QI (a) and in case of a vector of q QIs (b).

8 Figure 8 .
Figure 8.Di,A of a point i belonging to the POF (example with n = 2).

Figure 8 .
Figure 8. D i,A of a point i belonging to the POF (example with n = 2).

Figure 10 .
Figure 10.ri,A of a point i belonging to the POF.

Figure 9 .
Figure 9. d i,A of a point i belonging to the POF.

Figure 10 .
Figure 10.ri,A of a point i belonging to the POF.Figure 10. r i,A of a point i belonging to the POF.

Figure 10 .
Figure 10.ri,A of a point i belonging to the POF.Figure 10. r i,A of a point i belonging to the POF.

EFigure 11 .
Figure 11.Interpretation function: pseudo-code to compare A and B by means of DOA(A) and DOA(B).

Figure 11 .
Figure 11.Interpretation function: pseudo-code to compare A and B by means of DOA(A) and DOA(B).

Figure 13 .
Figure 13.Possible scenarios for A≻B.(a) scenario A2 or C2; (b) scenario B2 or D2; (c) scenario C2: limit case; (d) scenario D2: limit case; (e) scenario E2. 6.1.A2. i≺≺b Λ i≺a In this case, i strictly dominates b then si,B = di,B = dfi,b, because b is the solution which provides si,B.Moreover, i≺a implies that si,A ≤ dfi,a (see Remark 1).So, in order to demonstrate that si,A < si,Bit is sufficient to demonstrate that dfi,a < dfi,b.Recalling that:
An APF (blue dots) with all good features but not close to the POF and another (red triangles) that is only close to the POF.An APF (blue dots) with all good features but not close to the POF and another (red triangles) that is only close to the POF.An APF blue dots) with all good features but not extended and another (red triangles) that is only extended.Figure 6.An APF (blue dots) with all good features but with poor cardinality and another (red triangles) that has only high cardinality.
f2 f1 Figure 4.An APF (blue dots) with all good features but not uniformly distributed and another (red triangles) that is only uniformly distributed.Figure 5.An APF blue dots) with all good features but not extended and another (red triangles) that is only extended.Figure 6.An APF (blue dots) with all good features but with poor cardinality and another (red triangles) that has only high cardinality.2 Figure 4.An APF (blue dots) with all good features but not uniformly distributed and another (red triangles) that is only uniformly distributed.f2 f1 Figure 4.An APF (blue dots) with all good features but not uniformly distributed and another (red triangles) that is only uniformly distributed.Figure 5.An APF blue dots) with all good features but not extended and another (red triangles) that is only extended.Figure 6.An APF (blue dots) with all good features but with poor cardinality and another (red triangles) that has only high cardinality.2 Figure 5.An APF blue dots) with all good features but not extended and another (red triangles) that is only extended.f2 f1 Figure 4.An APF (blue dots) with all good features but not uniformly distributed and another (red triangles) that is only uniformly distributed.Figure 6.An APF (blue dots) with all good features but with poor cardinality and another (red triangles) that has only high cardinality.2 Figure 6.An APF (blue dots) with all good features but with poor cardinality and another (red triangles) that has only high cardinality.

Table 3 .
Summary of selected UQIs and features that influence their value.

Table 3 .
Summary of selected UQIs and features that influence their value.

Table 3 .
Summary of selected UQIs and features that influence their value.

Table 3 .
Summary of selected UQIs and features that influence their value.

Table 3 .
Summary of selected UQIs and features that influence their value.
Figure 9.di,A of a point i belonging to the POF. i||a

Table 4 .
DOA Evaluation for some typical POF (symbol in brackets is the marker associated to the APF in Figures14-16).