A Game-Theoretic Kendall’s Coefficient Weighting Framework for Evaluating Autonomous Path Planning Intelligence

Abstract

Accurately evaluating the intelligence of autonomous path planning remains challenging, primarily due to the interdependencies among evaluation metrics and the insufficient integration of subjective and objective weighting methods. This paper proposes Game-Theoretic Kendall’s Coefficient (GTKC) weighting framework for evaluating autonomous path planning intelligence. The framework specifies a safety–efficiency–comfort metric system with observable, reproducible, and quantifiable metrics. To account for intermetric dependence, subjective weights are elicited via an improved Analytic Network Process (ANP), while objective weights are derived using the CRITIC method to capture contrast intensity and intercriteria conflict. The credibility of the subjective and objective weights is evaluated using Kendall’s coefficient and the coefficient of variation, respectively. Subsequently, based on the principle that higher credibility should receive greater weight, a game-theoretic optimization model is employed to dynamically derive optimal combination coefficients. Experimental results on three case scenarios demonstrate that the GTKC framework significantly outperforms existing weighting approaches in terms of effectiveness (achieving a lowest Mean Absolute Error (MAE) of 0.15 and a perfect Spearman’s correlation coefficient ($\overline{\rho }=1.0$) with ground-truth rankings), stability (Mean Standard Deviation (MSD) = 0.023), and ranking consistency (Kendall’s coefficient W = 0.924). These findings validate GTKC as a theoretically grounded and practically robust mechanism that explicitly models metric interdependencies and integrates expert knowledge with empirical evidence, enabling reliable and reproducible evaluation of autonomous path planning intelligence.

1. Introduction

Autonomous path planning systems (APPSs) play a critical role in the unmanned ground vehicle (UGV) decision-making procedure [1,2,3,4,5]. APPS is primarily manifested in their capability to recognize traffic signals and obstacles [6], understand traffic rules and dynamic environmental changes [4,7], and dynamically adjust their operational states to select safe, comfortable, energy-efficient [8,9], and high-performance paths toward designated destinations [10]. However, given the direct implications for human life and traffic safety, there is increasing public and academic interest in accurately evaluating the intelligence of autonomous path planning and comparing intelligence levels across different systems [11,12,13,14,15,16,17,18,19,20,21].

The current research landscape on quantitative evaluation of autonomous path planning intelligence is characterized by the following aspects. First, regarding the selection of evaluation metrics, many scholars typically employ performance metrics based on a system’s internal parameters. For instance, Ref. [11] evaluates system performance using measurable parameters like trajectory tracking accuracy and computational real-time performance. Ref. [12] establishes a “white-box” evaluation model by incorporating control stability indices and sensor noise characteristics, while Ref. [13] employs dedicated metrics such as energy consumption models and motion smoothness to classify system intelligence levels. Although these methods offer precise insights into specific system performances, their effectiveness heavily relies on a comprehensive understanding of the internal architecture, assuming “white-box” access. This inherent dependency, however, severely limits their applicability for evaluating proprietary or architecturally complex systems where such internal knowledge is inaccessible. Second, concerning the selection of weighting methods, prevailing approaches largely utilize quantitative techniques that assume independence among metrics. Ref. [14] applies the Analytic Hierarchy Process (AHP) to construct judgment matrices through expert questionnaires. Ref. [15] employs the entropy weight method to calculate objective weights based on the dispersion degree of metric data, and Ref. [16] determines core indicator weights through factor analysis for dimensionality reduction. While these methods enhance the objectivity of evaluations, their foundational assumption of metric independence is often unrealistic. This neglect of inherent correlations can introduce bias into the weighting results, thereby compromising the validity of the final evaluation. Finally, addressing the combination of subjective and objective weights, researchers have proposed various combination weighting schemes. Ref. [17] proposed combining subjective AHP weights and objective entropy weights through linear weighting. Ref. [18] adopted a multiplicative synthesis rule to fuse weights derived from fuzzy AHP and the CRITIC method, and Ref. [19] balanced expert experience with data-driven results by setting fixed proportional coefficients. However, these methods lack a rigorous theoretical basis for determining the combination coefficients, often relying on heuristic or experiential settings. This reliance introduces subjectivity and arbitrariness into the process, ultimately undermining the robustness and scientific credibility of the final weights.

To overcome the limitations mentioned above, this paper proposes a Game-Theoretic Kendall’s Coefficient (GTKC) weighting framework that addresses three critical challenges in autonomous path planning intelligence evaluation through a triple-pronged approach.

(1)

We construct a three-dimensional evaluation metric system encompassing safety, efficiency, and comfort, explicitly modeling interdependencies among these metrics while ensuring observability and applicability across both “white-box” and “black-box”.

(2)

Since intelligence evaluation metrics are often interdependent, we accordingly select the improved Analytic Network Process (ANP) for subjective weighting due to its capacity to model metric correlations and feedback loops, and employ the CRITIC method for objective weighting as it effectively quantifies data contrast and conflict through standard deviation and correlation analysis.

(3)

We introduce a game-theoretic optimization model that dynamically balances subjective and objective weights by minimizing deviations through Nash equilibrium, while rigorously evaluating internal consistency using Kendall’s coefficient (for expert consensus) and coefficient of variation (for data stability). This framework harmonizes expert knowledge with data-driven insights, enhances robustness through credibility-weighted vector fusion, and significantly improves ranking consistency compared to conventional combination methods.

The remainder of this paper is structured as follows: Section 2 formally defines the problem of autonomous path planning intelligence evaluation. Section 3 presents the construction of the three-dimensional evaluation metric system and introduces the GTKC weighting framework. Section 4 validates GTKC against comparative methods through progressive experiments. Finally, Section 5 summarizes the key findings, highlights the main contributions of this research, and discusses potential avenues for future work.

2. Problem Description

To evaluate the intelligence of mobile applications, the research methodology should follow the evaluation framework illustrated in Figure 1.

Let $A=a_1,a_2,\dots ,a_m$ be a set of m APPS algorithms under evaluation, $M=m_1,m_2,\dots ,m_n$ be n evaluation metrics, and $X\in {\mathbb{R}}^{m\times n}$ be the normalized performance matrix derived from observable behavioral data. The intelligence evaluation problem aims to construct a reliable ranking $\mathcal{R}$ of algorithms based on a weighted aggregation of metric scores:

$$\mathrm{Score}\left(a_i\right)=\sum _{j=1}^n{w}_j·X_{ij},\mathrm{with}\sum _{j=1}^n{w}_j=1,w_j\ge 0$$

(1)

where $w={[w_1,w_2,\dots ,w_n]}^T$ represents the metric weight vector, which quantifies the relative importance of each metric in the overall intelligence evaluation. The determination of an optimal weight vector w is crucial, as it directly influences the reliability and validity of the evaluation results.

Current approaches for determining w can be broadly categorized into three paradigms:

Subjective weighting methods derive weights based on expert knowledge and preferences:

$$w_s=f_s\left(E\right)$$

(2)

where E represents expert judgment matrices and $f_s$ denotes subjective weighting functions such as Analytic Hierarchy Process (AHP) or Analytic Network Process (ANP).

Objective weighting methods compute weights directly from the data characteristics:

$$w_o=f_o\left(X\right)$$

(3)

where $f_o$ represents objective weighting functions such as Entropy Weight Method (EWM) or Criteria Importance Through Intercriteria Correlation (CRITIC).

Combination weighting methods integrate both subjective and objective perspectives through linear combination:

$$w_c=\alpha w_s+\beta w_o,\mathrm{with}\alpha +\beta =1,\alpha ,\beta \ge 0$$

(4)

For complex evaluation problems such as APPS intelligence evaluation, combination weighting approaches are generally preferred as they balance expert domain knowledge with empirical data patterns. However, the critical challenge lies in determining the optimal combination coefficients $(\alpha ,\beta )$ that properly reflect the relative credibility of subjective and objective components. The selection of appropriate weighting methods and the determination of combination coefficients should be guided by the specific characteristics of the evaluation metrics and the inherent limitations of existing approaches.

3. Solution

3.1. Evaluation Metrics

We develop a hierarchical evaluation metric system for Autonomous Path Planning System (APPS) intelligence, structured around three core dimensions: safety, efficiency, and comfort. As illustrated in Figure 2, this system includes eight specific metrics, each designed to capture an essential aspect of path planning intelligence. Its design is grounded in the foundational principles of observability (metrics derivable from external outputs), quantifiability (numerically representable for objective comparison), and strong relevance (directly related to APPS operational objectives), and synthesizes key insights from existing literature [22,23,24,25].

(1) Safety. This dimension quantifies the APPS fundamental capability to safeguard occupants, pedestrians, and surrounding vehicles during navigation, measured by:

Obstacle Collision Rate ($C_r$): The ratio of collision occurrences to total navigation attempts (dimensionless, expressed as a decimal):

$$C_r={\displaystyle \frac{N_c}{N_t}}$$

(5)

where $N_c$ is the number of collisions and $N_t$ is the total number of navigation attempts.

Average Lateral Acceleration ($A_l$): The mean absolute value of lateral acceleration during navigation ($\mathrm{m}/{\mathrm{s}}^2$):

$$A_l={\displaystyle \frac{1}{T}}\sum _{t=1}^T\left|a_l\left(t\right)\right|$$

(6)

where $a_l\left(t\right)$ is the lateral acceleration at time t and T is the total navigation duration.

(2) Efficiency. This dimension captures the performance of APPS in resource optimization and task execution timeliness, evaluated through:

Task Completion Time ($T_c$): The time required to complete the navigation task ($\mathrm{s}$):

$$T_c=t_e-t_s$$

(7)

where $t_s$ and $t_e$ are the start and end times, respectively.

Average Speed ($V_a$): The mean velocity maintained during navigation ($\mathrm{m}/\mathrm{s}$):

$$V_a={\displaystyle \frac{1}{T}}{\int }_0^{T}v\left(t\right)dt$$

(8)

Total Trajectory Length ($L_t$): The cumulative distance traveled along the actual path  ($\mathrm{m}$):

$$L_t=\sum _{k=1}^{N-1}\sqrt{{(x_{k+1}-x_k)}^2+{(y_{k+1}-y_k)}^2}$$

(9)

where $(x_k,y_k)$ represents the position coordinates at waypoint k.

Energy Consumption ($E_c$): The total energy expenditure during navigation ($\mathrm{kWh}$):

$$E_c={\int }_0^{T}P\left(t\right)dt$$

(10)

where $P\left(t\right)$ denotes the instantaneous power consumption at time t. For the purpose of comparing relative energy performance across algorithms under consistent vehicle dynamics, environment, and scenario, it can be simplified to the sum of squared accelerations ${\int }_0^T{a}^2\left(t\right)dt$ owing to the direct correlation between actuation power and acceleration.

(3) Comfort. This dimension reflects the quality of the passenger experience generated by the maneuvers of APPS, evaluated by:

Average Acceleration ($A_a$): The root mean square of acceleration magnitude ($\mathrm{m}/{\mathrm{s}}^2$):

$$A_a=\sqrt{{\displaystyle \frac{1}{T}}{\int }_0^T[a_x^2\left(t\right)+a_y^2\left(t\right)]dt}$$

(11)

Average Curvature (${\kappa }_a$): The mean curvature of the traversed path ($1/\mathrm{m}$):

$${\kappa }_a={\displaystyle \frac{1}{L_t}}{\int }_0^{L_t}\left|\kappa \left(s\right)\right|ds$$

(12)

where $\kappa \left(s\right)$ is the path curvature at arc length s.

3.2. Game-Theoretic Kendall’s Coefficient Weighting Framework

In the process of combination weighting, the allocation of combination coefficients should be dynamically adjusted based on the internal consistency levels of both subjective and objective weights. Specifically, the higher the internal consistency of either subjective or objective weights, the larger the corresponding combination coefficient should be. Thus, the combination coefficients must be positively correlated with the internal consistency levels of the respective weights, so as to ensure that the more reliable weights exert greater influence in the combination weights.

Based on the above, the following section elaborates on the framework and implementation.

3.2.1. Framework

The GTKC weighting framework, schematically illustrated in Figure 3, is structured around three sequential stages to achieve a scientifically grounded fusion of subjective and objective perspectives.

(1)

Subjective Weighting Method: Expert judgments are aggregated and refined to derive subjective weights, explicitly accounting for interdependencies among evaluation metrics. This stage employs an enhanced group Analytic Network Process incorporating outlier filtering and optimization-based consensus fusion to enhance credibility.

(2)

Objective Weighting Method: Data-driven weights are computed by quantifying both the inherent contrast intensity (dispersion) within each metric and the conflict (redundancy) between metrics based on their correlation structure.

(3)

Combination Weighting Method: The subjective and objective weight vectors are optimally combined using a Nash equilibrium model minimizing deviation. Crucially, Kendall’s coefficient and the coefficient of variation are introduced to dynamically adjust the combination coefficients, assigning greater weight to the vector demonstrating higher internal consistency.

This tripartite framework facilitates the principled integration of expert knowledge and empirical data characteristics, dynamically balancing their influence based on inherent agreement.

3.2.2. Implementation

(1) Determining Subjective Weight Vector

ANP is well-suited for evaluating systems with internal feedback and complex interdependencies among metrics. Unlike AHP, which assumes metric independence, ANP enables a more realistic modeling of relationships, yielding more accurate results. However, when ANP is applied in group decision-making scenarios, expert judgments often exhibit inconsistencies. Simply discarding outlier opinions may lead to information loss and distorted evaluations.

To address this issue, this study adopts an improved group decision-making approach based on ANP. The implementation involves the following steps [26,27]:

(a) Determine Weight Ranges. Let the expert evaluation set be $F_m=\left\{f_{m1},f_{m2},\dots ,f_{mn}\right\}$, and let the weight set for metric $c_j$ be $W_j=\left\{f_{1j},f_{2j},\dots ,f_{mj}\right\}$. The range of expert opinions is defined as

$${\omega }_j^{-}=min\left(f_{1j},f_{2j},\dots ,f_{mj}\right)$$

(13)

$${\omega }_j^{+}=max\left(f_{1j},f_{2j},\dots ,f_{mj}\right)$$

(14)

(b) Identify Valid Intervals. Outliers are often located at the range boundaries. The interval length $d_j={\omega }_j^{+}-{\omega }_j^{-}$ reflects expert consensus. A large $d_j$ implies strong disagreement. So, the singularity test uses $\delta =d_j/2$ the precision threshold.

(c) Eliminate Singular Points. The endpoint is considered an outlier and removed. This step is iteratively performed until all remaining values are within a valid interval.

(d) Construct Filtered Expert Matrix. Let $m^{\prime }$ be the number of experts remaining after outlier elimination. The filtered expert matrix is denoted as $F_{m^{\prime }}=\left\{f_{m^{\prime }1},f_{m^{\prime }2},\dots ,f_{m^{\prime }n}\right\}$.

(e) Optimize Subjective Weight. The final group-consensus weight ${\omega }_j^{*}$ is obtained by minimizing the deviation between expert opinions and the group weight:

$$min\sum _{j=1}^n\sum _{m=1}^{m^{*}}{\left(f_{mj}-{\omega }_j^{*}\right)}^2$$

(15)

Subject to:

$$\sum _{j=1}^n{\omega }_j^{*}=1$$

(16)

$${\omega }_j^{-}<{\omega }_j^{*}<{\omega }_j^{+}.$$

(17)

The resulting subjective weight vector is

$${\omega }_s=\left\{{\omega }_1^{*},{\omega }_2^{*},\dots ,{\omega }_j^{*}\right\}.$$

(18)

(2) Determining Objective Weight Vector

The CRITIC method is a well-established objective weighting technique that quantifies metric weights based on both data dispersion and inter-metric conflict. Its five-step procedure is as follows [28]:

(a) Data Normalization. To eliminate the influence of differing measurement scales, the raw data are normalized using min–max scaling.

For the cost-type metrics, the normalization formula is

$$x^{\prime }={\displaystyle \frac{x_{\max }-x}{x_{\max }-x_{\min }}}$$

(19)

For the benefit-type metric, the normalization formula is

$$x^{\prime }={\displaystyle \frac{x-x_{\min }}{x_{\max }-x_{\min }}}$$

(20)

Thus, after normalization, a score of 1 always represents the best possible performance for any metric, and 0 the worst, ensuring a consistent and correct interpretation during the linear weighting in Equation (1).

(b) Calculate Standard Deviation $s_j$. The variability of each metric j is measured by its standard deviation $S_j$:

$$S_j=\sqrt{\sum _{i=1}^n{\left(X_{ij}^{\prime }-\overline{X_j}\right)}^2/\left(n-1\right)}$$

(21)

where $X_{ij}^{\prime }$ is the normalized value of the i-th sample for metric $j,\overline{X_j}$ is the mean, and n is the sample size.

(c) Calculate Conflict Between Metrics. The conflict or redundancy of metric j with respect to other metrics is given by

$$R_j=\sum _{i=1}^p\left(1-r_{ij}\right)$$

(22)

where $r_{ij}$ is the Pearson correlation coefficient between metric j and metric i, and p is the total number of metrics.

(d) Calculate Information Content. The amount of information carried by each metric is

$$C_j=S_j{R}_j$$

(23)

(e) Derive Objective Weight. The final objective weight for each metric is calculated as

$${\omega }_j^{\prime }=C_j/\sum _{j=1}^p{C}_j$$

(24)

This yields the objective weight vector:

$${\omega }_o=\left\{{\omega }_1^{\prime },{\omega }_2^{\prime },\dots ,{\omega }_j^{\prime }\right\}.$$

(25)

(3) Determining Combination Weight Vector

(a) Game-Theoretic Optimization Model

Let ${\omega }_1$ and ${\omega }_2$ denote the subjective weight and objective weight vectors, respectively. The combination weight vector is given by

$$\omega ={\alpha }_1{\omega }_s+{\alpha }_2{\omega }_o$$

(26)

The optimal values of ${\alpha }_1$ and ${\alpha }_2$ are determined by minimizing the total deviation from both components under a Nash equilibrium formulation:

$$minf\left({\alpha }_1,{\alpha }_2\right)={∥\begin{array}{c}\omega -{\omega }_s\end{array}∥}_2^2+{∥\begin{array}{c}\omega -{\omega }_o\end{array}∥}_2^2$$

(27)

that is also subject to

$${\alpha }_1+{\alpha }_2=1,{\alpha }_1,{\alpha }_2\in \left[0,1\right]$$

(28)

The initial combination coefficients ${\alpha }_1$ and ${\alpha }_2$ are derived by solving the optimization problem defined in Equations (27) and (28). Substituting the linear combination $\omega ={\alpha }_1{\omega }_s+{\alpha }_2{\omega }_o$ and the constraint ${\alpha }_2=1-{\alpha }_1$ into the objective function, we obtain

$$f\left({\alpha }_1\right)=\parallel ({\alpha }_1{\omega }_s+(1-{\alpha }_1){\omega }_o)-{\omega }_s{\parallel }_2^2+{\parallel ({\alpha }_1{\omega }_s+(1-{\alpha }_1){\omega }_o)-{\omega }_o\parallel }_2^2.$$

Simplifying the two terms,

$$f\left({\alpha }_1\right)=\parallel ({\alpha }_1-1)({\omega }_s-{\omega }_o){\parallel }_2^2+\parallel {\alpha }_1({\omega }_s-{\omega }_o){\parallel }_2^2={(1-{\alpha }_1)}^2\parallel {\omega }_s-{\omega }_o{\parallel }_2^2+{\alpha }_1^2{\parallel {\omega }_s-{\omega }_o\parallel }_2^2.$$

Factoring out the common term,

$$f\left({\alpha }_1\right)=\parallel {\omega }_s-{\omega }_o{\parallel }_2^2\left[{(1-{\alpha }_1)}^2+{\alpha }_1^2\right]={\parallel {\omega }_s-{\omega }_o\parallel }_2^2(2{\alpha }_1^2-2{\alpha }_1+1).$$

This is a univariate quadratic function in ${\alpha }_1$. The first and second derivatives are

$${\displaystyle \frac{df}{d{\alpha }_1}}=\parallel {\omega }_s-{\omega }_o{\parallel }_2^2(4{\alpha }_1-2),{\displaystyle \frac{d^{2}f}{d{\alpha }_1^2}}=4{\parallel {\omega }_s-{\omega }_o\parallel }_2^2.$$

Since ${\displaystyle \frac{d^{2}f}{d{\alpha }_1^2}}>0$ for any ${\omega }_s\ne {\omega }_o$, the function $f\left({\alpha }_1\right)$ is strictly convex and has a unique minimum. Setting the first derivative to zero,

$$\parallel {\omega }_s-{\omega }_o{\parallel }_2^2(4{\alpha }_1-2)=0\Rightarrow {\alpha }_1={\displaystyle \frac{1}{2}}.$$

Thus, the initial optimal coefficients are ${\alpha }_1^{\prime }={\alpha }_2^{\prime }=0.5$.

(b) Measurement Consistency

To comprehensively reflect the reliability of subjective and objective weights, different consistency measurement methods are adopted based on their inherent characteristics:

For subjective weights (derived from expert consensus), Kendall’s coefficient of concordance is used to quantify the consistency of expert judgments [29,30]:

$$W_1=12S/m^2\left(n^3-n\right)$$

(29)

where $S={\displaystyle \sum _{j=1}^n}{\left(X_j-\overline{X}\right)}^2;W_1$ is the consistency levels of the subjective weights; $X_j$ is the total rank score for metric $j;\overline{X}$ is the mean rank; m is the number of experts; n is the number of metrics.

For objective weights (derived from data-driven calculation), the coefficient of variation (CV, denoted as $W_2$) is used to measure consistency:

$$W_2={\sigma }_{{\omega }_2}/{\mu }_{{\omega }_2}$$

(30)

where ${\sigma }_{{\omega }_2}$ is the standard deviation of the objective weight vector ${\omega }_2;{\mu }_{{\omega }_2}$ is the mean of the objective weight vector ${\omega }_2$.

Owing to the inherent “lower-is-better” nature of $W_2$, it is imperative to transform it into a “higher-is-better” measure normalized to the interval (0,1]. This transformation is critical for establishing syntactic comparability with $W_1$, as measures with aligned evaluation criteria (i.e., both “higher-is-better”) and standardized scales are prerequisite for meaningful integration and subsequent analysis. The specific functional form employed to achieve this dual objective (semantic inversion and scale standardization) is explicitly defined as follows:

$$W_2^{\prime }=1/(1+W_2)$$

(31)

(c) Optimal Combination Coefficients

Subsequently, the optimal combination coefficients are adjusted by assigning higher weights to the component with better consistency:

$$\alpha =W_1{\alpha }_1^{\prime }/W_1{\alpha }_1^{\prime }+W_2^{\prime }{\alpha }_2^{\prime }$$

(32)

$$\beta =W_2^{\prime }{\alpha }_2^{\prime }/W_1{\alpha }_1^{\prime }+W_2^{\prime }{\alpha }_2^{\prime }$$

(33)

It is obvious that the adjusted combination coefficients satisfy $\alpha +\beta =1$.

(d) Optimal Combination Weight Vector

The optimal combined weight vector is then computed as follows:

$${\omega }^{*}=\alpha {\omega }_s+\beta {\omega }_o$$

(34)

This formulation ensures that the more consistent component receives proportionally greater influence in the final combination, enhancing the credibility and robustness of the overall evaluation.

Algorithm 1 summarizes the computational procedure of the proposed combination weighting framework based on GTKC.

Algorithm 1 GTKC

Require: Expert evaluation matrix $F_{sw}$, Objective index data $\mathrm{X}$ Ensure: Optimal combination weights ${\omega }^{*}$   1: //Stage 1: Determine subjective weights${\omega }_s$via improved ANP   2: Initialize empty weight set for each metric.   3: for each metric j do   4:    Determine weight range $\left[{\omega }_j^{-},{\omega }_j^{+}\right]$ using Equations (13) and (14)   5:    Identify valid weight interval via singularity test   6:    Remove outlier opinions and update expert set   7:    Formulate filtered expert matrix $F_{sw}^{\prime }$   8: end for   9: Solve optimization model via Equations (15)–(18) to get ${\omega }_s$ 10: //Stage 2: Determine objective weights ${\omega }_o$via CRITIC method 11: Normalize objective data matrix $\mathrm{X}$ using Equations (19) and (20) 12: Calculate standard deviation $S_j$ and correlation matrix for all metrics via Equation (21) 13: Compute conflict index $R_j$ and information content $C_j$ via Equations (22) and (23) 14: Derive ${\omega }_o$ by normalizing $C_j$ across all metrics via Equations (24) and (25) 15: //Stage 3: Determine optimal combination weights ${\omega }^{*}$ 16: Establish game model and compute initial ${\alpha }_1^{\prime }$, ${\alpha }_2^{\prime }$ via Equations (26)–(28) 17: Evaluate consistency: compute $W_1$ (expert consensus) via Equation (29) and $W_2$ (data variation) via Equation (30) 18: Transform $W_2$ to $W_2^{\prime }$ via Equation (31) to align with “higher-is-better” principle 19: Compute final coefficients $\alpha$, $\beta$ via credibility-weighted fusion in Equations (32) and (33) 20: Determine ${\omega }^{*}=\alpha {\omega }_s+\beta {\omega }_o$ via Equation (34) 21: return ${\omega }^{*}$

4. Experiment

To systematically validate the proposed GTKC framework, this study designs three case scenarios with a progressive experimental logic: starting with verifying whether the framework can accurately evaluate intelligence (effectiveness), then moving to testing whether results are reliable for the same object (stability), and finally exploring whether it can stably distinguish between multiple objects (ranking consistency). This layered design ensures each case builds on the previous one, gradually deepening the verification of the framework’s performance. Detailed experimental design and implementation are presented below:

4.1. Experimental Design

The primary goal of the designed testing scenario is not to evaluate the performance of the algorithms under evaluation, but to generate quantifiable behavioral data within the testing scenario for analyzing the effectiveness, stability, and ranking consistency of the proposed GTKC framework. If the testing scenario is excessively complex, the algorithms under evaluation will frequently fail—this will prevent the collection of sufficient successful trial data, making it impossible to conduct meaningful statistical comparisons of the various evaluation methods and ultimately undermining the aforementioned goal.

4.1.1. Test Scenarios

This experiment focuses on comparing the performance of multiple evaluation methods under identical conditions. To eliminate interference from environmental variables and road complexity, a 2D simulation environment was established to ensure consistent environmental conditions across all rounds, as illustrated in Figure 4.

(1) Simulation Environment Terrain Configuration: A flat rectangular area of 20 m × 20 m was constructed with a ground friction coefficient of 0.8, eliminating terrain irregularities that could affect kinematic modeling.

Obstacle: All obstacles were fully observable. Three static obstacles with radii of 1.5 m, 1 m, and 0.5 m were positioned at coordinates (15,15), (8,3), and (5,5), respectively. One dynamic obstacle (radius: 1 m) performed a linear reciprocating motion along the x-axis at a constant speed of 1m/s between (5,10) and (15,10).

Vehicle: The vehicle was modeled with dimensions of 1 m (length) × 0.5 m (width), a maximum acceleration of ±1 m/s², and a differential drive system featuring a maximum steering angle of ±30° and a safety distance of 4m.

While deliberately simplified to ensure data collection viability across all algorithms under evaluation, this environment retains core challenges such as static/dynamic obstacle interactions, requisite path planning, and the need for timely decision-making. This ensures that the generated performance metrics are meaningful for intelligence evaluation while guaranteeing the reproducibility and generalizability of our comparative findings on evaluation methods.

(2) Task

The vehicle was required to navigate from the start point (0,0) to the endpoint (20,20) from a stationary state, prioritizing path planning while minimizing energy consumption and travel time.

(3) Experimental Protocol

Algorithms in Groups A and B underwent independent repeated trials under identical conditions. To maintain fixed strategies and consistency, the self-learning functionality of all RL algorithms was disabled, retaining only their decision execution capabilities.

To mitigate random fluctuations, testing was terminated when the variance of all metrics fell below a predefined threshold $ϵ$.

Five domain experts were invited for this experiment (two in intelligent vehicle engineering, one each in operations research and optimization, control theory, and human factor engineering) to provide multi-dimensional support for evaluation metric weight setting, covering simulation-reality adaptability, multi-objective optimization-based mathematical logic, simulation feasibility, and user experience consistency, thereby ensuring its scientificity and rationality.

4.1.2. Logic of Progressive Verification

To evaluate the performance of an evaluation method, a progressive verification of its results must be conducted through three critical aspects: effectiveness, stability, and ranking consistency [31]. The logic of progressive validation is outlined as follows:

(1) Case 1: Effectiveness. Addresses “Can the method accurately evaluate the intelligence level of APPS?” This is the basic prerequisite -only if the evaluation results are consistent with the actual intelligence level can subsequent reliability and discriminability verification be meaningful.

(2) Case 2: Stability. Addresses “Are the evaluation results of the same method stable across repeated trials?” This is a deepening of effectiveness—even if a method is accurate in a single trial, large fluctuations in repeated tests will reduce its practical value.

(3) Case 3: Ranking Consistency. Addresses “Can the method stably rank multiple APPS across repeated trials?” This is the application-oriented extension-APPS evaluation often requires comparing multiple systems, and stable ranking results are key to supporting decision-making.

4.1.3. Comparative Methods

To comprehensively verify the advantages of GTKC, three representative weighting methods covering different paradigms are selected as comparative methods. These methods are used in all three case scenarios of experiments to ensure consistent cross-method comparisons:

(1) AHP [14]: A purely subjective weighting approach based on expert judgment, representing traditional subjective evaluation (weak in objectivity and metric dependency).

(2) AHP-EWM [17]: A hybrid method combining AHP (subjective) and EWM (objective), representing early combination weighting (ignores metric interdependencies).

(3) ANP-CRITIC [19]: A correlation-aware combination method integrating ANP (subjective, handles dependencies) and CRITIC (objective, reflects data characteristics), representing improved combination weighting (lacks dynamic consistency-driven coefficient adjustment).

4.1.4. Algorithms Under Evaluation

To meet the different objectives of the three case scenarios, two groups of algorithms under evaluation are designed, with clear division of labor:

(1) Group A: SAC Algorithm Variants with Multi-Level Stepwise Interventions

Used in Case 1 (effectiveness verification). By introducing controllable interventions (decision delay, Gaussian noise) to SAC, we construct algorithms with explicit intelligence differences (higher delay/noise → lower intelligence). This provides clear ground-truth rankings, enabling quantitative measurement of evaluation accuracy. For this purpose, two categories of algorithm variants were deployed in the simulation environment, with 600 experimental trials conducted. Experimental data revealed the following:

(a) Decision delay and task execution efficiency exhibited significant negative correlations. As shown in Figure 5, when system decision delay increased to 400 ms, task completion time surged approximately to 200 s (a 300% increase) This nonlinear response indicates rapid intelligence degradation beyond the 300 ms latency threshold.

(b) Gaussian noise intensity and algorithmic intelligence demonstrated marked negative correlations. This noise was injected into the control outputs of the algorithms to simulate actuation or control uncertainty. The injected noise was defined as zero-mean Gaussian white noise, with its standard deviation serving as the intervention magnitude (see

Table 1. Parameter configurations and reference intelligence rankings of Group A.

Group	Algorithm	Intervention Magnitude	Intelligence Ranking
${\mathrm{A}}_1$	SAC	0	1
	SAC-1	+60 ms	2
	SAC-2	+160 ms	3
	SAC-3	+380 ms	4
${\mathrm{A}}_2$	SAC	0	1
	SAC-N1	0.01	2
	SAC-N2	0.05	3
	SAC-N3	0.10	4

Group ${\mathrm{A}}_1$: SAC algorithms with controlled decision delay interventions; Group ${\mathrm{A}}_2$: SAC algorithms with controlled Gaussian noise amplitude interventions.

). As depicted in Figure 6, when injected Gaussian noise standard deviation exceeded 0.2, collision rates escalated to 100% (a 100% decline relative to baseline conditions). This suggests that excessive noise (standard deviation > 0.1) destabilizes action outputs, triggering rapid intelligence deterioration.

Based on these findings, baseline algorithm variants exhibiting significant disparities in intelligence were selected from the experimental dataset to form Group ${\mathrm{A}}_1$ and Group ${\mathrm{A}}_2$. Their parameter configurations and reference rankings are detailed in Table 1. Specifically, the reference rankings provide a foundational basis for the subsequent comparative analysis of evaluation methods, thereby ensuring the reproducibility and comparability of the experimental conclusions.

(2) Group B: 5 Representative Path Planning Algorithms

Used in Case 2 (stability verification) and Case 3 (ranking consistency verification). For Group B, algorithms under evaluation include five widely adopted path planning algorithms. These algorithms are categorized into two paradigms: Traditional Algorithms (DWA and Rapidly exploring Random Tree (RRT)) and RL Algorithms (A2C, Proximal Policy Optimization (PPO), and SAC).

DWA: Local path planning with real-time path planning but limited global optimization [32].

RRT: Sampling-based global planning suitable for high-dimensional spaces but with suboptimal trajectory smoothness [33].

A2C: Balances exploration and exploitation via policy gradients and value functions [34].

PPO: Enhances training stability through clipped policy updates [34].

SAC: A model-free RL algorithm using maximum entropy optimization to balance exploration and policy stability through reward-entropy maximization [35].

4.1.5. Experimental Procedure

All three case scenarios follow the same data collection and processing workflow to ensure result comparability:

Phase 1: Data Acquisition. For each algorithm, collect raw data of 8 metrics (covering safety: obstacle collision rate, average lateral acceleration; efficiency: task completion time, average speed, total trajectory length, energy consumption; comfort: average acceleration, average curvature across multiple trials.

Phase 2: Data Preprocessing. Normalize all metrics using min-max scaling to eliminate dimensional differences.

Phase 3: Weight Calculation. Compute weights for the four methods detailed in Section 4.1.3. AHP-EWM and ANP-CRITIC use empirical 0.5 combination coefficients [36,37,38,39,40]; GTKC uses dynamic coefficients described in Section 3.2.2.

Phase 4: Scoring and Ranking. Calculate overall intelligence scores via linear weighting of normalized metrics, and generate algorithm rankings based on scores.

4.2. Case 1: Effectiveness

This case aims to confirm whether the evaluation results of GTKC align with the actual intelligence level of APPS. For this purpose, Group A (SAC variants with clear ground-truth rankings) is used as the test subject, and we quantify the deviation between each method’s evaluation results and the reference ranking.

4.2.1. Experimental Setup

(1) Test Object: Group A $\left({\mathrm{A}}_1\right.$: decision delay intervention, ${\mathrm{A}}_2$: Gaussian noise intervention).

(2) Trial Design: Set the global precision threshold to $\epsilon =0.05$; conduct 13 consecutive trials for Group $A_1$ and 20 trials for Group ${\mathrm{A}}_2$.

(3) Evaluation Metrics: Use Mean Absolute Error (MAE, measures deviation from reference ranking) and mean Spearman’s correlation coefficient $\overline{\rho }$ (measures ranking similarity, range $\left[-1,1\right]$, higher = more consistent) for quantitative analysis:

$$MAE={\displaystyle \frac{1}{n}}\sum _{i=1}^n\left|o_{ij}-o_i\right|$$

(35)

where $o_{ij}$ is the rank of the i-th algorithm under the $\mathrm{j}$-th evaluation method, $o_i$ is the reference rank assigned by human experts, and n is the number of algorithms.

$$\overline{\rho }={\displaystyle \frac{1}{m}}\sum _{k=1}^m\left(1-6\sum _{i=1}^n{d}_{ik}^2/n\left(n^2-1\right)\right)$$

(36)

where $d_{ik}$ is the ranking difference between the evaluation ranking of the i-th algorithm in the k-th evaluation and the standard ranking, m is the number of experiments, and n is the number of algorithms.

4.2.2. Results and Analysis

Four comparative methods were applied to evaluate these algorithms, yielding the metric weights under each method. For key metrics (e.g., Task Completion Time $T_c$), GTKC’s weight allocation is more in line with the actual impact of intelligence.

(1) Analysis of Group ${\mathrm{A}}_1$ (Decision Delay Intervention)

The derived metric weights for Group ${\mathrm{A}}_1$ are presented in

Table 2. Evaluation metric weights for Group ${\mathrm{A}}_1$ under four comparative methods.

Method	Metric Weights
	Obstacle Collision Rate	Average Lateral Acceleration	Task Completion Time	Average Speed	Total Trajectory Length	Energy Consumption	Average Acceleration	Average Curvature
AHP	0.0747	0.0890	0.0506	0.2967	0.2877	0.0926	0.0485	0.0601
AHP-EWM	0.2792	0.0631	0.0602	0.1813	0.2474	0.0659	0.0530	0.0499
ANP-CRITIC	0.0811	0.0889	0.0925	0.1216	0.1473	0.1212	0.1690	0.1785
GTKC	0.0975	0.1060	0.1047	0.1348	0.2074	0.1349	0.1004	0.1143
$\Delta$ Weight (%)	20.22%	19.24%	13.19%	10.86%	40.80%	11.30%	−40.59%	−35.97%

$\Delta$ Weight (%), is calculated as $[W_{\mathrm{GTKC}}-W_{\mathrm{ANP}\text{-}\mathrm{CRITIC}})/W_{\mathrm{ANP}\text{-}\mathrm{CRITIC}}]\times 100 \%$.

. These metric weights are calculated using four weighting methods (AHP, AHP-EWM, ANP-CRITIC, and GTKC). Examining the weight allocation for $T_c$ reveals significant differences across methods:

• Comparison between AHP-EWM (0.0602) and ANP-CRITIC (0.0925) indicates that accounting for inter-metric relationships substantially influences weight allocation, leading to a 53.65% increase in weight for $T_c$.
• Further comparison between ANP-CRITIC (0.0925) and GTKC (0.1047) shows that the GTKC weight exceeds the ANP-CRITIC weight by 13.19%. This difference arises from the GTKC framework’s dynamic determination of combination coefficients (subjective-to-objective ratio = 1.27), prioritizing the more credible subjective weights derived via the improved ANP process.

After normalizing the experimental data, linear weighting was applied to derive evaluation scores and corresponding rankings for each algorithm under the different evaluation methods. The rank order comparison against the reference ranking (Table 1) is illustrated in Figure 7. The dashed lines represent the ground-truth reference rankings.

(2) Analysis of Group ${\mathrm{A}}_2$ (Gaussian Noise Intervention)

The derived metric weights for Group ${\mathrm{A}}_2$ are presented in

Table 3. Evaluation metric weights for Group ${\mathrm{A}}_2$ under four comparative methods.

Method	Metric Weights
	Obstacle Collision Rate	Average Lateral Acceleration	Task Completion Time	Average Speed	Total Trajectory Length	Energy Consumption	Average Acceleration	Average Curvature
AHP	0.0747	0.0890	0.0506	0.2967	0.2877	0.0926	0.0485	0.0601
AHP-EWM	0.1682	0.0593	0.0800	0.2029	0.2958	0.0664	0.0728	0.0546
ANP-CRITIC	0.1033	0.0766	0.1061	0.1306	0.1356	0.1055	0.1751	0.1673
GTKC	0.1211	0.0868	0.1289	0.1424	0.1872	0.1204	0.1108	0.1025
$\Delta$ Weight (%)	17.23%	13.32%	21.49%	9.04%	38.05%	14.12%	−36.72%	−38.73%

$\Delta$ Weight (%), is calculated as $[W_{\mathrm{GTKC}}-W_{\mathrm{ANP}\text{-}\mathrm{CRITIC}})/W_{\mathrm{ANP}\text{-}\mathrm{CRITIC}}] \times 100\%$.

. These metric weights are calculated using four weighting methods (AHP, AHP-EWM, ANP-CRITIC, and GTKC). Similar to Group ${\mathrm{A}}_1$, the weight allocation for $T_c$ highlights methodological differences:

• The ANP-CRITIC weight (0.1061) is 32.63% higher than the AHP-EWM weight (0.08), again underscoring the impact of considering metric interdependencies.
• The GTKC weight (0.1289) surpasses the ANP-CRITIC weight (0.1061) by 21.49%. This increase is attributed to GTKC’s consistency-driven adjustment (subjective-to-objective ratio = 1.29), favoring the more reliable subjective weights.

The evaluation scores and rankings derived for Group ${\mathrm{A}}_2$ across the 20 trials are compared against the reference ranking in Figure 8.

MAE and $\overline{\rho }$ of each evaluation method are derived from Equations (35) and (36), as shown in

Table 4. Performance comparison of evaluation methods based on $MAE$ and $\overline{\rho }$.

		AHP	AHP-EWM	ANP-CRITIC	GTKC
${\mathrm{A}}_1$	SAC	1.77	1.08	1.54	1.31
	SAC-1	2.69	2.77	2.54	2.54
	SAC-2	2.54	2.85	2.46	2.69
	SAC-3	3.00	3.31	3.46	3.46
	MAE	0.73	0.42	0.54	0.42
	$\overline{\mathit{\rho }}$	0.80	1.00	0.80	1.00
${\mathrm{A}}_2$	SAC	1.00	1.10	1.20	1.00
	SAC-N1	2.25	2.30	1.90	2.10
	SAC-N2	3.50	3.30	3.10	3.20
	SAC-N3	3.25	3.30	3.80	3.70
	MAE	0.38	0.35	0.15	0.15
	$\overline{\mathit{\rho }}$	0.80	0.95	1.00	1.00

$MAE$: Mean Absolute Error; $\overline{\rho }$: mean Spearman’s correlation coefficient.

. Consequently, across both algorithm groups subjected to distinct intelligence-diminishing interventions (decision delay and Gaussian noise), the GTKC framework consistently demonstrates superior effectiveness in approximating the ground-truth intelligence rankings, as evidenced by its significantly lower MAE and higher $\overline{\rho }$ compared to the comparative methods (AHP, AHP-EWM, ANP-CRITIC).

This confirms that GTKC can accurately evaluate APPS intelligence, laying the foundation for Case 2.

4.3. Case 2: Stability

Following the confirmation of effectiveness, this case aims to verify stability—specifically, whether the evaluation results of the same algorithm remain stable across repeated trials. We use Group B (five representative algorithms) as the test object, and measure the fluctuation of evaluation values to evaluate the method’s reliability.

4.3.1. Experimental Setup

(1) Test Object: Group B (DWA, RRT, A2C, PPO, SAC).

(2) Trial Design: Conduct 17 consecutive independent trials; disable the self-learning function of RL algorithms to ensure fixed decision strategies.

(3) Evaluation Metrics: Use the mean of the Standard Deviations (MSD, the standard deviations of each algorithm’s evaluation values across trials-lower MSD = better stability):

$$MSD={\displaystyle \frac{1}{n}}\sum _{i=1}^n{\sigma }_i$$

(37)

where ${\sigma }_i$ is the standard deviation of the i-th evaluation values, and n is the number of algorithms.

4.3.2. Results and Analysis

The resulting metric weights for each method are summarized in

Table 5. Evaluation metric weights for Group B under four comparative methods.

Method	Metric Weights
	Obstacle Collision Rate	Average Lateral Acceleration	Task Completion Time	Average Speed	Total Trajectory Length	Energy Consumption	Average Acceleration	Average Curvature
AHP	0.0747	0.0890	0.0506	0.2967	0.2877	0.0926	0.0485	0.0601
AHP-EWM	0.0764	0.0676	0.0329	0.1876	0.1980	0.0531	0.1825	0.2021
ANP-CRITIC	0.1290	0.0940	0.0630	0.1199	0.1049	0.0872	0.1993	0.2026
GTKC	0.1394	0.1107	0.0787	0.1334	0.1706	0.1051	0.1268	0.1353
$\Delta$ Weight (%)	8.06%	17.77%	24.92%	11.26%	62.63%	20.53%	−36.38%	−33.21%

$\Delta$ Weight (%), is calculated as $[W_{\mathrm{GTKC}}-W_{\mathrm{ANP}\text{-}\mathrm{CRITIC}})/W_{\mathrm{ANP}\text{-}\mathrm{CRITIC}}]\times 100\%$.

. The influence of considering inter-metric relationships on weight allocation is evident. For instance, comparing the weight assigned to $T_c$ by AHP-EWM (0.0329) and ANP-CRITIC (0.063) reveals a significant 91.49% increase under the latter method. Further comparison between ANP-CRITIC (0.063) and GTKC (0.0787) for the same metric indicates that the dynamic adjustment based on subjective–objective consistency (with a coordination coefficient ratio of 1.29 favoring the more credible subjective weights) resulted in GTKC assigning a weight 24.92% higher than ANP-CRITIC.

Subsequently, intelligence evaluation scores were computed for all Group B under each method. The distribution of these scores across the seventeen trials is presented in Figure 9:

• AHP: Algorithm scores demonstrated clear stratification but exhibited significant fluctuations, particularly for $\mathrm{A}2\mathrm{C}$ and $\mathrm{PPO}$.
• AHP-EWM: Evaluation values showed substantial discrepancies between algorithms, with SAC displaying pronounced upward score volatility.
• ANP-CRITIC: Moderate differences were observed between algorithm scores, and A2C showed considerably reduced fluctuations compared to AHP.
• GTKC: Algorithm scores were relatively well-differentiated and exhibited patterns broadly similar to those under ANP-CRITIC, suggesting comparable stability.

The MSD of the evaluation values, calculated using Equation (37), provides a quantitative measure of stability across methods. Figure 10 visualizes these results. AHP exhibited the highest MSD, indicating the poorest stability. Although GTKC (MSD = 0.023) demonstrated a slightly higher MSD than ANP-CRITIC (MSD = 0.022), a paired t-test revealed that this difference was not statistically significant ($t\left(4\right)=0.477$, $p=0.658>0.05$). The 95% confidence interval for the difference was $[-0.005, 0.007]$, which includes zero, further confirming no significant difference in stability between the two methods. Both GTKC and ANP-CRITIC significantly outperformed AHP-EWM and AHP. Consequently, the stability ranking of the evaluation methods, ordered from most to least stable, is ANP-CRITIC ≈ GTKC ≻ AHP-EWM ≻ AHP.

This indicates that GTKC has good stability—its results for the same algorithm are reliable, providing support for the third case scenario of ranking consistency verification.

4.4. Case 3: Ranking Consistency

Building on Case 2, this case aims to verify ranking consistency—specifically, whether the ranking order of multiple algorithms remains consistent across repeated trials. Leveraging the 17 trials of Group B (from Case 2), we use Kendall’s coefficient to quantify the agreement of ranking results, which reflects the method’s discriminative ability for multiple algorithms.

4.4.1. Experimental Setup

(1) Test Object: Group B (DWA, RRT, A2C, PPO, SAC).

(2) Trial Design: Conduct 17 consecutive independent trials (from Section 4.3) to ensure consistency with stability verification.

(3) Evaluation Metrics: Use Kendall’s coefficient W (range $\left[0,1\right]$, higher $W=$ more consistent ranking across trials):

$$W=12S/M^2\left(N^3-N\right)$$

(38)

where S is the sum of squared deviations from the mean rank, M is the number of trials, and N is the number of algorithms.

4.4.2. Results and Analysis

As defined in Equation (38) and illustrated in Figure 11, the GTKC framework achieved the highest concordance $\left(W=0.924\right)$, indicating superior stability in the produced ranking orders compared to the alternative methods. The AHP-EWM combination method yielded a slightly lower ($W=0.910$). In contrast, the purely subjective AHP approach demonstrated significantly lower reliability $\left(W=0.812\right)$, consistent with its inherent susceptibility to evaluator variability. The ANP-CRITIC method ($W=0.892$) also exhibited lower concordance than GTKC and AHP-EWM. Consequently, the ranking consistency performance of the evaluated methods is ordered as GTKC ≻ AHP-EWM ≻ ANP-CRITIC ≻ AHP.

This confirms that GTKC can stably rank multiple APPS, making it suitable for practical scenarios where multiple systems need to be compared.

5. Conclusions

This study proposes the Game-Theoretic Kendall’s Coefficient (GTKC) weighting framework to address three core limitations in autonomous path planning intelligence evaluation: the lack of architecture-agnostic metrics, neglect of metric interdependencies, and non-rigorous weight fusion. Our solution integrates a three-dimensional metric system (Safety, Efficiency, Comfort) with improved ANP for subjective weights and CRITIC for objective weights, dynamically fused via a Game-Theoretic Kendall’s Coefficient that prioritizes high-consistency weight vector.

Experimental validation demonstrates GTKC’s superiority over comparative methods (AHP, AHP-EWM, ANP-CRITIC). In effectiveness tests, GTKC achieved the lowest MAE (0.15 for Group A₂) and perfect Spearman’s correlation coefficient ($\overline{\rho }$ = 1.0) with ground-truth rankings, correctly classifying SAC variants (Table 4). While ANP-CRITIC showed a marginally lower MSD (0.022) than GTKC (0.023) in Group B, a paired t-test confirmed this difference was not statistically significant ($t\left(4\right)=0.477$, $p=0.658>0.05$), indicating comparable stability between the two methods. GTKC exhibited superior ranking consistency (W = 0.924) by dynamically balancing expert knowledge and data patterns (Figure 11). The framework advances APPS evaluation theory through its rigorous weight fusion mechanism and generalizable paradigm applicable to both traditional (DWA, RRT) and RL-based (A2C, PPO, SAC) algorithms.

This study has limitations including the use of a 2D simulation environment, insufficient real-world data validation, and the absence of statistical significance testing. To address these limitations, future work will extend experiments to real-world scenarios (e.g., sensor noise, delayed execution), validate method robustness through ablation studies, and employ statistical tests to support the generalizability of conclusions.