Multicriteria Machine Learning Model Assessment—Residuum Analysis Review

Abstract

The use of machine learning (ML) and its applications is one of the leading research areas nowadays. Neural networks have recently gained enormous popularity and many works in various fields use them in the hope of improving previous results. The application of the artificial intelligence (AI) methods and the rationale for this decision is one issue, but the assessment of such a model is a completely different matter. People mostly use mean square error or less often mean absolute error in the absolute or percentage versions. One should remember that an error does not equal an error and a single value does not provide enough knowledge about the causes of some behavior. Proper interpretation of the results is crucial. It leads to further model improvement. It might be challenging, but allows us to obtain better and more robust solutions, which ultimately solve real-life problems. The ML model assessment is the multicriteria task. A single measure delivers only a fraction of the picture. This paper aims at filling that research gap. Commonly used integral measures are compared with alternative measures like factors of the Gaussian and non-Gaussian statistics, robust statistical estimators, tail index and the fractional order. The proposed methodology delivers new single-criteria indexes or the multicriteria approach, which extend the statistical concept of the moment ratio diagram (MRD) into the index ratio diagram (IRD). The proposed approach is validated using real data from the Full Truck Load cost estimation example. It compares 35 different ML regression algorithms applied to that task. The analysis gives an insight into the properties of the selected methods, enables their comparison and homogeneity analysis and ultimately leads towards constructive suggestions for their eventual proper use. The paper proposes new indexes and concludes that correct selection of the residuum analysis methodology makes the assessment and the ML regression credible.

1. Introduction

Full truck loads (FTL) is a common transportation method, where the goods fill an entire truck. It perfectly suits a large volume of goods, where a load covers the whole truck space. Apart from the FTL, there exists an alternative method called less-than truckload (LTL), in which a truck takes several partial loads to different contract load/unload locations within a single journey. This work focuses on the FTL, however from a rarely addressed perspective.

In case of the external fleet contract pricing, the contracts are priced according to the varying contractor policy, which takes into account several objective and subjective market and non-market factors [1,2], such as contract0dependent, economic, regulatory, general and purely ambivalent factors. Those factors potentially reflect, in the opinion of the decision maker, the shipping cost for a given commodity, along some determined route within a certain time period. Dynamic pricing shows increasing shipping business dynamics [3] and determines it simultaneously [4]. Next, it is assumed the contractor may use some custom dynamic pricing model, which is associated with serious challenges [5].

The shipping cost estimations start to play an even more important role in case of short routes, when common relations between the price, fuel costs and the driver time is not straightway included. The pricing of the FTL long-range contracts is frequently solved using deterministic analytical fright calculators [6], or with the use of algorithmic estimators. It is worth noting that the AI and ML methods [7] lead in that area. The literature mostly focuses on the blind machine learning approaches [2,8,9,10] or hybrid ones [11].

On the contrary, the task of the short-range FTL shipment cost estimation, i.e., for routes shorter than 50 km, is seldom addressed in the literature. One may try to determine the reasons for that. Firstly, this task is of a smaller order and is often hidden in all data, or omitted due to the lower absolute cost values of such routes. Secondly, these routes often play a complementary or secondary role in relation to the long-distance ones. Thirdly, it is a thankless and simply difficult task. The fact that we undertake this task is related to its difficulty and observations made while dealing with the general task of estimating FTL costs [11], where the largest relative errors occur precisely for short routes, which spoils the overall picture of the modeling and estimation task.

The multi-criteria assessment methodology is the second contribution of this work. It is a well-known fact that each performance index, such as means square error, absolute error or various residuum statistical factors, exhibit different properties and are sensitive towards different estimation error aspects. Despite that knowledge, there is a significant deficiency in the literature, because multi-criteria residuum assessment approaches are hardly reported [12]. The researchers mostly use the mean square error (MSE), mean absolute error (MAE) or the relative mean absolute percentage error (MAPE). Each of them has different properties and puts more attention on various data features. Square errors are sensitive to large residua, while even often-occurring but small errors are neglected. Absolute errors equalize these differences and reflect small residua as well.

During this research, different measures are investigated: classical (normal), robust and L-moments, tail index and the Geweke-Porter-Hudak estimator of the fractional order of the ARFIMA filter. They are presented following the statistical approach named moment ratio diagrams (MRD) or L-moments ratio diagram (LMRD). Our work plans to supplement the estimation residuum analysis with the multi-criteria approach named the IRD—(Index Ratio Diagrams) using various measures.

The main contribution of this work lies in the proposal of the multicriteria residuum analysis concept, as this aspect is hardly existent in the research. The FTL estimation task is considered as the representative example for the assessment methodology.

General FTL cost estimation task formulation is introduced in Section 2, while Section 3 describes the utilized assessment methods and estimation algorithms. Various estimators are compared in Section 4. Section 5 presents the results of the multi-criteria residuum analysis, while Section 6 concludes the paper.

2. Estimation Case Study

The analysis uses the data from selected Polish shipping companies [9]. The original database consists of approximately 414,000 records. Once the data are limited only to the short-range contracts, the number of records is limited to 20,239 from 1 January 2016 till 30 April 2022. These data are used for training. Contracts from 1 May 2022 till 1 August 2022 (703 records) are used for validation, as shown in

Table 1. Number of records used during the analysis.

Data	Raw	Preprocessed
training	414,404	20,239
validation	14,968	703

.

Data Preprocessing and Features Selection

Each contract included in the database is characterized by 22 independent features. We limit this number to the 12 most important variables listed in

Table 2. The list of selected features.

Time-Related	Distance-Related	Route-Related	Freight-Related
date of payment	total distance	number of loads	usage of cold storage
min transport time	total empty distance	number of unloadings
max transport time
time interval
date of transport
lead time

. This table does not include an important feature like the fuel cost, which is highly volatile due to varying geopolitical and economic situations. The following descriptors are excluded from the estimation process:

• ID number—a sequence number;
• Maximum weight and tonne-kilometers, as they are frequently incomplete;
• Geographical clusters [11], the latitudes and longitudes of the loading and unloading places, which are used only in the selection of the short-range routes.

Python programming language (scikit_learn and torch libraries) and MATLAB (Statistics and Machine Learning Toolbox) are used during data processing and the estimation process.

3. Methods and Algorithms

This research uses quite a large scope of possible methods, which are included in the proposed IRD framework: integral indexes, classical, robust and L-moments, tail index and the Geweke–Porter–Hudak fractional order estimator of the ARFIMA filter. Methods used during calculations are described below.

3.1. Integral Measures

The MSE measure is calculated as the mean integral of the squared residua over some time period $k=1,\dots ,N$

$$\mathrm{MSE}=\frac{1}{n}\sum {\left(y-\widehat{y}\right)}^2.$$

(1)

It penalizes large errors, neglecting the smaller ones. This measure is significantly affected by outlying occurrences and exhibits the zero-breakdown point [13]. The MAE index sums absolute residua values

$$\mathrm{MAE}=\frac{1}{n}\sum \left|y-\widehat{y}\right|.$$

(2)

The MAE is less conservative as it penalizes continuing small residua. Though its breakdown point is zero as well, it is robust against a portion of outliers. The MAPE is defined in a relative way:

$$\mathrm{MAPE}=\frac{100\%}{n}\sum \left|\frac{y-\widehat{y}}{y}\right|.$$

(3)

Generally, it is difficult to define what error value is good and which model is proper enough. Lewis, in [14], proposed the interpretation of typical MAPE values, which is presented in

Table 3. Interpretations of MAPE values.

MAPE [%]	Interpretation
<10	highly accurate forecasting
10–20	good forecasting
20–50	reasonable forecasting
>50	inaccurate forecasting

.

3.2. Statistical Moments

This research follows a theoretical approach that assumes some distribution, which correctly represents the underlying process. Such probabilistic density function (PDF) is utilized through their factors and moments (if they exist).

Let us assume that ${\left\{X_i\right\}}^T$ is a given time series with the mean $\mu$ and the r-th central moment ${\gamma }_r=E{\left(X-\mu \right)}^r$, $E\left(·\right)$ denotes the expectation. The mean $\mu$ is the first moment ${\gamma }_1$, and the variance ${\sigma }^2$ is the second one denoted as ${\gamma }_2$, where $\sigma$ denotes the standard deviation. These moments are often used together with the third one, i.e., the skewness ${\gamma }_3$ and the fourth—the kurtosis ${\gamma }_4$. The skewness reflects data asymmetry and kurtosis its concentration.

$$\begin{array}{cc}\hfill {\gamma }_3& =\frac{1}{N{\sigma }^3}\sum _{i=1}^N{\left(x_i-x_0\right)}^3\hfill \end{array}$$

(4)

$$\begin{array}{cc}\hfill {\gamma }_4& =\frac{1}{N{\sigma }^4}\sum _{i=1}^N{\left(x_i-x_0\right)}^4-3\hfill \end{array}$$

(5)

The existence of outlying observations in the time series causes its distributions to start to be fat-tailed [15]. This feature biases the moments estimation. The use of statistical factors in the residuum analysis has quite a long history following the legacy of Gauss [16,17]. They are strictly connected with the assumption about data normality and normality tests.

3.3. L-Moments

The theory of L-moments was proposed by Hosking [18] as a linear combination of order statistics. The theory of L-moments includes new descriptions of the distribution shape, helps to estimate factors of an assumed statistical function and allows the testing of hypotheses about theoretical distributions. We may define L-moments for any random variable, whose expected value exists. The L-moments give almost unbiased statistics, even for a small sample. They are less sensitive to the distribution tails [19]. These properties are appreciated in the life sciences, although they might be also used in control engineering. Their calculation is done as follows. The data $\left\{x_1,\dots ,x_N\right\}$, N—number of samples, are ranked in ascending order from 1 to N. Next, the sample L-moments ($l_1,\dots ,l_4$), the sample L-skewness ${\tau }_3$ and L-kurtosis ${\tau }_4$ are evaluated as:

$$\begin{array}{cc}\hfill l_1& ={\beta }_0,l_2=2{\beta }_1-{\beta }_0,l_3=6{\beta }_2-6{\beta }_1+{\beta }_0,\hfill \\ \hfill l_4& =20{\beta }_3-30{\beta }_2+12{\beta }_1-{\beta }_0,\hfill \\ \hfill {\tau }_2& =\frac{l_2}{l_1},{\tau }_3=\frac{l_3}{l_2},{\tau }_4=\frac{l_4}{l_2},\hfill \end{array}$$

(6)

where

$${\beta }_j=\frac{1}{N}\sum _{i=j+1}^N{x}_i\frac{(i-1)(i-2)\cdots (i-j)}{(N-1)(N-2)\cdots (N-j)}$$

(7)

Statistical properties are reflected in L-shift $l_1$, L-scale $l_2\in (0,1)$, L-covariance (L-Cv) ${\tau }_2$, L-skewness ${\tau }_3\in (-1,1)$ and L-kurtosis ${\tau }_4\in (-\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$4$}\right.,1)$. They help to fit a distribution to a dataset. L-skewness and L-kurtosis work as the goodness-of-fit measure. They can be calculated for theoretical PDFs [20] and normal distribution has: $l_1=\mu$, $l_2=\raisebox{1ex}{$\sigma $}\!\left/ \!\raisebox{-1ex}{$\pi $}\right.$, ${\tau }_3=\raisebox{1ex}{$l_3$}\!\left/ \!\raisebox{-1ex}{$l_2$}\right.=0$ and ${\tau }_4=\raisebox{1ex}{$l_4$}\!\left/ \!\raisebox{-1ex}{$l_2$}\right.=0.1226$.

The L-moments deliver reliable estimates, especially for small samples and fat-tailed distributions. They form a backbone for the L-moments ratio diagrams, which support the distribution fitting to empirical samples. The most common diagram uses L-kurtosis (${\tau }_4$) versus L-skewness (${\tau }_3$) relationship [19]. Apart from that, L-moments diagrams are used to compare various samples originating from different sources in search for the homogeneity [21,22]. These features constitute the research idea for the proposal of the IRDs.

3.4. Robust Statistics

Robust statistics is taken into consideration to address the impact of outliers. Robust estimators acquired popularity with works of Huber [23]. Robust estimators allow to evaluate the shift, the scale and the regression coefficients for data impacted by outliers. This work utilizes the M-estimators with logistic psi-function implemented in the LIBRA toolbox [24].

M-estimators consider the maximum likelihood (ML) estimator that uses the log-likelihood formula for a given distribution $F_{\mu ,\sigma }$ is

$$\sum _{i=1}^N\left\{log{f}_0\left(\frac{x_i-\mu }{\sigma }-log\sigma \right)\right\},$$

(8)

The location M-estimator $\widehat{\mu }$ is defined as a solution of:

$$\frac{1}{n}\sum _{i=1}^n\psi \left(\frac{x_i-\widehat{\mu }}{{\sigma }_0}\right)=0,$$

(9)

where $\psi (.)$ is an influence function, $\widehat{\mu }$ is a location estimator and ${\sigma }_0$ is an assumed scale. In a similar way we define the scale M-estimator ${\sigma }_R=\widehat{\sigma }$

$$\frac{1}{n}\sum _{i=1}^n\rho \left(\frac{x_i-{\mu }_0}{\widehat{\sigma }}\right)=1,$$

(10)

where $\rho (.)$ is a loss function, $\sigma$ is a location estimator and ${\mu }_0$ is a preliminary location. The work utilizes logistic functions ${\rho }_{\mathrm{L}}\left(\xi \right)$ and ${\psi }_{\mathrm{L}}\left(\xi \right)$ given by

$$\begin{array}{cc}\hfill {\rho }_{\mathrm{L}}\left(\xi \right)& =k_{\mathrm{L}}^{2}ln\left[cosh\left(\frac{\xi }{k_{\mathrm{L}}}\right)\right],\hfill \end{array}$$

(11)

$$\begin{array}{cc}\hfill {\psi }_{\mathrm{L}}\left(\xi \right)& =k_{\mathrm{L}}tanh\left(\frac{\xi }{k_{\mathrm{L}}}\right).\hfill \end{array}$$

(12)

The utilization of robust statistics is just straightforward, as they form the natural extension of the statistical scale measures (variance and standard deviation) in case o outliers [25], which occur frequently in real-life applications [26]. With their use, we are not biasing our assessment by anomalies or erroneous records.

3.5. Moment Ratio Diagrams

Moment ratio diagrams graphically show the statistical properties of the considered time series in a plane. The MRD is a graphical representation in Cartesian coordinates of a pair of standardized moments. Actually, there are two versions [27]. The MRD(${\gamma }_3,{\gamma }_4$) shows the third standardized moment ${\gamma }_3$ (or its square ${\gamma }_3^2$) as abscissa and the fourth moment ${\gamma }_4$ as ordinate, plotted upside down. There exists a theoretical limitation of the accessible area, as ${\gamma }_4-{\gamma }_3^2-1\ge 0$. The locus corresponding to PDF can be a point, curve or region. It depends on the number of shape parameters. PDFs lacking shape factor (like Gauss or Laplace) are represented by a point, and functions with one shape coefficient are represented by a curve. Regions reflect functions with two shape factors. The second type of the diagram MRD(${\gamma }_2,{\gamma }_3$) represents variance ${\gamma }_2$ as the abscissa and skewness ${\gamma }_3$ as the ordinate.

Moment ratio diagrams initially have a formulated multicriteria assessment approach, though in the statistical context. This research uses this idea in the residual analysis.

3.6. L-Moment Ratio Diagrams

L-moments have been introduced by Hosking [18]. The LMRDs are popular in the extreme analysis. They allow the identification of proper distribution for empirical observations. The LMRD(${\tau }_3,{\tau }_4$) is the most common and it shows the L-kurtosis ${\tau }_4$ versus L-skewness ${\tau }_3$. Similarly to MRDs, one can confront the empirical data with the theoretical PDF candidate [19]. A blank diagram with shapes (points or curves) for some theoretical PDFs is presented in Figure 1.

Similarly to the MRD(${\gamma }_2,{\gamma }_3$), there are two LMRD versions: LMRD(${\tau }_2,{\tau }_3$) and LMRD($l_2,{\tau }_3$). They combine in a single plot the scale and skewness. As the CPA analyzes frequently use kurtosis, it is proposed to investigate a new formulation: the LMRD($l_2,{\tau }_4$). The LMRDs are the successor of the MRDs and predecessor of the IRDs and they should be considered from that perspective.

3.7. The $\alpha$-Stable Distribution

Apart from the specific robust estimators or L-moments one may use other distributions. Stable functions deliver an alternative set of the statistical measures [28]. The $\alpha$-stable distribution is expressed by the characteristics equation

$$F_{\alpha ,\beta ,\delta ,\gamma }^{\mathrm{stab}}\left(x\right)=exp\left\{i\delta x-{\left|\gamma x\right|}^{\alpha }\left(1-i\beta l\left(x\right)\right)\right\},$$

(13)

where

$$l\left(x\right)=\left\{\begin{array}{cc}\mathrm{sgn}\left(x\right)tan\left(\frac{\pi \alpha }{2}\right)\hfill & \mathrm{for}\alpha \ne 1\hfill \\ \mathrm{sgn}\left(x\right)\frac{2}{\pi }ln\left|x\right|\hfill & \mathrm{for}\alpha =1\hfill \end{array}\right.$$

(14)

The factor $0<\alpha \le 2$ is called the index of stability (stability exponent), the $\left|\beta \right|\le 1$ is the skewness factor, $\delta \in \mathbb{R}$ the shift and $\gamma >0$ the scale. Thus, the $\alpha$-stable distribution has one shift factor, one scale and two shape coefficients: $\alpha$ and $\beta$.

The $\alpha$-stable distribution has an increasing potential in the assessment approaches [29], as it allows to measure the data diversity (scale factor $\gamma$) and other shaping factors such as skewness ($\beta$) and the tailedness ($\alpha$). These features nominate them as the potential measures in the residual analysis as well. Their application might be constrained in case of data, which does not fall into the stable families, which should be validated before their use.

3.8. Tail Index

Statistics frequently use the law of large numbers and the central limit theorem. Once data exhibits outliers, which is revealed in the form of tails, the majority of the assumptions made are not met. In such a case the knowledge of where the tail starts and which observations are located in the tail plays an important role [30,31]. There are many methods to estimate it and the tail index, denoted as $\widehat{\xi }$, is the most promising one [32]. There are quite a few tail index estimation approaches, with two leading ones: the Hill [33] and Huisman estimator [34]. This work uses the second one.

Tail index as such is an extension of the $\alpha$-stable distribution’s stability exponent and it measures where the distribution tail starts. This perspective nominates the tail index as the potential measure of the data properties and their contamination with anomalies.

3.9. ARFIMA Models and Fractional Order

The ARFIMA time series is treated as an extension to the classical ARIMA regression models, see [35]. The process $x_k$ is denoted as ARFIMA$(p,d,q)$

$$A_p\left(z^{-1}\right)·x_k=B_q\left(z^{-1}\right)·{(1-z^{-1})}^{-d}{ϵ}_k,$$

(15)

where $A\left(z^{-1}\right)$ and $B\left(z^{-1}\right)$ are polynomials in the discrete time delay operator $z^{-1}$, ${ϵ}_k$ is random noise with finite or infinite variance. We use Gaussian noise in this research. Fractional order $-0.5<d<0.5$ refers to process memory.

For $d\in (0,0.5)$ the process exhibits long memory or long-range positive dependence (persistence). The process has intermediate memory (anti-persistence) or long-range negative dependence, when $d\in (-0.5,0)$. The process has short memory for $d=0$; it is stationary and invertible ARMA. ARFIMA$(p,d,q)$ time series is calculated by d-fractional integrating of a classical ARMA$(p,q)$ process. The d-fractional integrating through the ${\left(1-z^{-1}\right)}^{-d}$ operator causes the dependence between observations, even as they are far apart in time.

The Geweke–Porter–Hudak (GPH) estimator proposed by [36] uses a semi-parametric procedure to estimate the memory parameter $d_{\mathrm{GPH}}$ for ARFIMA process $x_k$:

$$x_k={(1-z^{-1})}^{-d}{ϵ}_k,$$

(16)

Next, ordinary least squares (LS) are applied to estimate $\widehat{d}$ from the

$$log\left(I_x\left({\lambda }_s\right)\right)=\widehat{c}-\widehat{d}\left|1-e^{i{\lambda }_s}\right|+\mathrm{residual},$$

(17)

being evaluated for fundamental frequencies ${\lambda }_s=\frac{2\pi s}{n}$, $s=1,\dots ,m$, $m<n$, where m is the largest integer in $(n-1)/2$ and $\widehat{c}$ is a constant. Discrete Fourier transform $x_k$ is evaluated as

$${\omega }_x\left({\lambda }_s\right)=\frac{1}{\sqrt{2\pi n}}\sum _{k=1}^n{x}_k{e}^{ik{\lambda }_s}.$$

(18)

Application of the least squares algorithm to the Equation (16) yields to the final formulation

$$\widehat{d}=\frac{{\sum }_{s=1}^m{x}_{s}log{I}_x\left({\lambda }_s\right)}{2{\sum }_{s=1}^m{x}_s^2},$$

(19)

where $I_x\left({\lambda }_s\right)={\omega }_x\left({\lambda }_s\right){\omega }_x{\left({\lambda }_s\right)}^{*}$ being a periodogram and $x_s=log\left|1-e^{ik{\lambda }_s}\right|$. The GPH algorithm calculates the $d_{\mathrm{GPH}}$ without explicit assumptions about ARMA polynomial orders. We use the [37] implementation.

The use of the Geweke–Porter–Hudak fractional order estimation in the assessment task is relatively new [38] and still requires much attention. Nonetheless, the first results are quite promising and that is why they are included in the analysis. However, the argumentation could be extended to the fractal, multi-fractal and data persistence time series assessment perspective of this estimator, which finds earlier references [39,40].

4. Estimation Approaches

This section describes the machine learning approaches taken into account during the study. We decided to apply only black-box identification approaches [41]. This choice is motivated by the unknown patterns behind the data due to the dynamic geopolitical situation in recent years and due to the specificity of market practice during the determination of the price for very short shipping. Among other factors, rising inflation, Brexit and the COVID-19 pandemic are impacting truck cargo transit prices. Proposing an explicit form of the cost model that takes into account the aforementioned factors would have been a difficult task, which we decided not to undertake.

4.1. Classical Regression Models

The following regression estimation algorithms are used during the analysis.

4.1.1. Linear Models

Linear regression, like the least mean squares (LMS), is frequently the natural first choice. LMS minimizes the sum of the squares of the differences between the actual and the estimated value (model residuum). LMS is the simplest regression approach, however it is highly sensitive to outliers [42]. Robust Linear Regression [43] (R-LMS) with the intercept and linear terms compared to LMS is affected by outliers only in a minimal scale. Stepwise Linear Regression [44] (SLR) is a method that reduces the influence of less important parameters in an iterative way.

Outliers were expected to occur in the dataset, such as single long-distance transports. Regression models robust to multivariate outliers, namely Theil-Sen [45] (TS-LR) and Huber [42] regressors (H-LR), are also tested. Huber regressor differs from the Theil-Sen one, because it does not ignore the effect of the outliers, giving just a smaller weight to them.

4.1.2. Support Vector Machine

Due to the existence of many transport parameters, it was problematic to divide the data into separate sets to infer the costs of new transports. For this purpose, it was reasonable to use hyperplane methods such as Support Vector Machines [46]. Using this method, we are able to approximate the costs of new transports by reference to a test set divided into sets with a predetermined precision. In this method, we can adjust the parameters to achieve a balance between the generalization of the model and its sensitivity.

Linear Support Vector Machines [47] (LSVM) is the simplest linear kernel (20) version of the method. It is particularly effective when dealing with linear relationships between input features and the target variable. Its aim is to identify the optimal hyperplane, minimizing the error between the predicted values and the actual results. Its biggest advantages are simplicity, efficiency and robustness to outliers, but it is ineffective when dealing with complex, non-linear relationships.

$$f\left(x\right)=\sum _{i=1}^n{\alpha }_i{y}_i\langle x_i,x\rangle +b$$

(20)

Quadratic Support Vector Machine [48] (QSVM) is an extension of the LSVM. Unlike the linear version, it uses a quadratic kernel function (21) that allows the capture of more complex decision constraints. It can be particularly useful when dealing with datasets where classes are not linearly separable.

$$f\left(x\right)=\sum _{i=1}^n{\alpha }_i{y}_i{(x_i·x)}^2+b$$

(21)

Coarse Gaussian Support Vector Machine [49] (CGSVM) is suited to tasks with low-complexity data. It uses Coarse Gaussian kernel that is given by the Formula (22)

$$f\left(x\right)=exp(-4·\sqrt{p}\parallel x_i-x_j{\parallel }^2).$$

(22)

Medium Gaussian Support Vector Machines [49] (MGSVM) is mainly suited to tasks with medium-complexity data. It uses Medium Gaussian kernel that is given by the Formula (23)

$$f\left(x\right)=exp(-\sqrt{p}\parallel x_i-x_j{\parallel }^2).$$

(23)

Following, the Fine Gaussian Support Vector Machines (LGSVM) use the Formula (24)

$$f\left(x\right)=exp(-\sqrt{\frac{p}{4}}\parallel x_i-x_j{\parallel }^2).$$

(24)

Kernel Support Vector Machine [50] (KSVM) belongs to the group of kernel approximation models. Using this method, we can conduct nonlinear regression on large datasets. Training and prediction processes will generally run faster for this method than for Gaussian kernel SVM models. Metric used in model fitting is epsilon-insensitive loss.

4.1.3. Gaussian Processes

The dataset that we use is not particularly large. We search for a regression model that handles data sets of small size reasonably well. In such a case the use of Gaussian Process Regression [51] is justified, because it allows us to determine the uncertainty of the transportation cost prediction as well. Gaussian Process Regression is a non-parametric approach, which is based on kernel probabilistic models.

Exponential Gaussian Process [52] (EGPR) uses exponential kernel, which is stationary kernel and can be parameterized by a length scale. Kernel is given as a fraction of Euclidean distance and length scale parameter. By taking to the kernel squared value of Euclidean distance we will obtain Squared Exponential Gaussian Process (also named radial basis function kernel [53]—SEGPR). The advantage to this development is the small chance of generating large errors, while handling extensive data sets in higher dimensions. Matern Gaussian Process Regression (MGPR) is the next extension to the Gaussian algorithm and is generalization of radial basis function kernel. The Matern kernel uses spectral densities of the stationary kernel and create Fourier transforms of the RBF kernel. Rational Quadratic Gaussian Process Regression [54] is a probabilistic method that is effective when dealing with non-linear relationships. The model defines a distribution over possible functions of the relationship between the input features and the target variable. In this method, the kernel is used to capture the similarity between data points, allowing the model to make predictions based on relationships of points in the training data. The Rational Quadratic kernel enables the model to capture a broader range of non-linear patterns in the data.

4.1.4. Nearest Neighbors

In order to check whether better results would not be achieved by averaging the known costs of several similar transports, the k-nearest neighbors method was used. The k Nearest Neighbors method (k-NN) [55] is based on searching the training set to find transports with parameters closest to the one whose cost we are looking for. This method uses ”k” such known transports.

This algorithm is sensitive to local structures of data due to using set of transports with the most similar parameters, but is not prone to outliers from the training set [56].

4.1.5. Orthogonal Matching Pursuit

Orthogonal Matching Pursuit (OMP) stands as an algorithm in compressing sensing, adept at recovering sparse signals within noisy linear regression models [57]. This technique enriches the foundational Matching Pursuit algorithm through a least-squares minimization at each step, thereby optimizing approximations of the extant elements. Noteworthy is the inherent restraint against redundant element selection, attributed to the orthogonal relationship between the residual and the previously chosen constituents. Consequently, the residual converges to zero after k iterations. The first publications within the domain of signal processing regarding Orthogonal Matching Pursuit appeared in 1993 [58].

4.1.6. Ridge Regression

Ridge regression (RR) addresses multicollinearity by improving least squares estimates, which suffer from bias and high variance in such scenarios [59]. By introducing controlled deviation to regression estimation, ridge regression curbs standard error and enhances reliability, albeit at the expense of some accuracy. It indirectly combats multicollinearity through a constraint length factor, relinquishing unbiasedness for more practical and robust regression coefficients. This method’s flexibility blends qualitative and quantitative analysis, offering a unique solution to multi-collinearity and finding application in extensive research. However, diverse ridge parameter calculations yield divergent results, and the popular ridge trace approach relies on subjective variable selection, posing arbitrary outcomes.

Moreover, it is noteworthy that the conventional ridge regression methodology does not inherently facilitate variable reduction, leading to the retention of all variables within the model. In response to this, the utilization of the Automatic Relevance Determination (ARD-RR) method emerges as an approach capable of decisively assessing the relevance of input features [60].

Furthermore, our approach encompasses the utilization of Bayesian Ridge Regression [61] (BRR), wherein is assumed that all regression coefficients have common variance. The Bayesian instantiation of ridge regression presents a distinct advantage by obviating the necessity for explicit regularization parameter selection. Instead, the model dynamically acquires this parameter from the data, engendering a more data-driven and adaptive regularization strategy.

4.1.7. Decision Trees

Searching for a regression method that would allow us to create an accurate and robust model, we also had non-technical aspects in mind. This was one of the reasons to use decision trees [62]. One of their many advantages is that the model is intuitive and easy to explain to the decision makers for whom the model is being developed.

Regression trees [63] are a supervised learning approach that is commonly used in statistics and data mining. They are one of the most popular algorithms in machine learning. The algorithm works by dividing the data set into subsets (branches, nodes and leaves). The division is made in such a way as to obtain the greatest possible information gain or to minimize the sum of squared errors (SSE). The unquestionable benefit of this approach is its robustness to outliers and missing data. Unfortunately, the growth of the regression tree is associated with a significant increase in computational complexity (therefore, the depth of the tree can be limited).

Regression Tree [64] (DTR) divides the data into branches, nodes and leaves. Regression Tree is similar to Decision Tree that is used to predict continuous data, not only discrete data output. Regression Tree can be classified by its size, defining the depth of the regression tree. Division is created by leaf size parameter. Coarse Regression Tree has a minimum of 20 Coarse Trees, Medium Regression Tree (M-DTR) has a minimum of 12 medium trees and Fine Regression Tree has 4 fine trees. Fine Regression Tree (F-DTR) is usually highly accurate on the training data, however separate test data accuracy can be not comparable to training one. Coarse Regression Tree (C-DTR), due to the multiplicity of leaves, is inclined to overfit data. However, Coarse Tree with limited large leaves does not gain high training accuracy and its training accuracy might be close to the test one.

Boosted Regression Trees [65] (BoostRT) are an advanced machine learning approach used for regression tasks, when there are non-linear relationships in the data. Boosting in the method consists in the iterative matching of trees to test data based on the residual error. Boosted Regression Tree has found wide applications in various domains, including finance, healthcare and environmental science, owing to its ability to handle high-dimensional data and produce accurate predictions. The disadvantage of this method is that its performance is highly dependent on careful parameter tuning and sufficient training data to avoid overfitting.

Gradient boosting stands as a prevalent and effective machine learning technique, extensively applied to regression and classification tasks. The development of gradient boosting can be attributed to the work of Jerome H. Friedman [66]. This methodology culminates in the formation of an ultimate predictive model structured as an ensemble amalgamating numerous feeble predictors. At its core, gradient boosting is a process of iteratively refining a cost function within the expanse of function space. This iterative refinement is achieved by judiciously selecting functions that align with the negative gradient orientation of the cost function. Often, the ensemble constituents of gradient boosting harness decision trees as weak predictors. The amalgamation of decision trees and boosting principles results in the Gradient Boosted Decision Tree. In the Gradient Boosted Decision Tree paradigm (GBoostRT), an iterative construction unfolds, progressively assembling an ensemble of modest decision tree learners through the mechanism of boosting. The culminating prediction yielded by Gradient Boosted Decision Tree is the aggregate outcome of assimilating the prediction outputs from all constituent trees, thereby harnessing the collective predictive prowess of the ensemble.

Histogram-based Gradient Boosting Regression Tree (HGBoostRT) represents an estimator endowed with capabilities to handle missing values (NaNs) [67]. Throughout the training process, the growth of trees entails a learning mechanism that strategically determines the trajectory of samples with missing values, directing them to either the left or right child nodes, predicated upon the ensuing potential gain. During prediction, samples harboring missing values are systematically assigned to the appropriate child node. In instances where a specific feature encountered no missing values during the training phase, samples replete with missing values are routed to the child node boasting the highest sample abundance. This estimator exhibits significantly heightened efficiency compared to the Gradient Boosting Regression tree, particularly in the context of substantial datasets.

Extremely Randomized Trees Regression [68] (ERTR) is a similar but significantly faster variant of random forests method. This algorithm creates numerous decision trees and predictions are made by averaging the prediction of the decision trees. Each tree is built based on a randomly chosen subset from the feature set. Splitting value is also chosen randomly—we do not calculate entropy, information gain or SSE error as in classical regression trees. This allows us to reduce the correlation of individual trees. The main advantages of this method include improved prediction accuracy, overfitting control and reduction in bias.

Bagged Regression Trees Regression [69] (BRTR) is a supervised machine learning algorithm. The main idea of Bagged Trees is to not rely on a single decision tree, but be dependent on many decision tree models. It is used to increase predictive power and stability of regression trees. The main advantage of using Bagged Trees is its ability to minimize variation while holding bias consistent. In general, decision trees are simple and easy to explain,; however, bagged trees algorithms append complexity, so they can be difficult to interpret.

4.1.8. Random Forest Regression

A decision forest in the Random Forest Regression (RFR) can be defined as an ensemble classifier comprised of an amalgamation of tree-structured classifiers. Each individual tree within this ensemble contributes a singular vote towards the prevailing class of the input data point. Remarkable enhancements in classification accuracy have been consistently noted, irrespective of the specific algorithms employed for the construction of the constituent trees [70].

4.1.9. Regularization Techniques

Equally important as the predicted shipping cost are the features/information based on which this cost was calculated. During preprocessing, you can a priori select the features based on which you build the model, but you can also use the LASSO [71] regression (LASSO-R) model to select these features. LASSO (least absolute shrinkage and selection operator) method reduces model overfitting by extending the cost function with a penalty term i.e., the sum of the L1 norms of the model coefficients. By minimizing the value of the cost function, the method identifies features that are irrelevant from its point of view. Using this model, the transport company will know which transport parameters it can afford to deviate from the norm without changing the cost of shipping.

A commonly used alternative to the LASSO model is the LARS model [72]. It is used for multidimensional data—such as the data from the shipping company on which our experiments are being conducted. In the case of LARS (least-angle regression) method, there is no need to adjust the hyper-parameters weighing the penalty term. LARS is quite similar to forward stepwise regression method. The Least Angle Regression method aims to find an attribute that has the highest correlation with the residual. However, despite the high numerical efficiency of this method, it is very sensitive to noise, which can lead to misleading predictions and, consequently, financial losses for the company that uses such a model.

The concept of a regressor combining the advantages of the above two has also emerged. Such model is LARS LASSO (LARS-R). It has a faster convergence than the standard LASSO method. This method has also only two hyper-parameters, so tuning the model is greatly simplified.

Another example of a hybrid method that combines the advantages of other methods (LASSO and Ridge) is the Elastic Net method [73] (ENR). It handles multi-collinearity issues extremely well. The penalty function incorporates both L1 and L2 norms (see Figure 2). It reduces overfitting by eliminating redundant (mutually correlated) features.

4.2. Neural Network Approach

In order to develop transport cost prediction models, it was decided to use the PyTorch library, which allows numerical calculations, including operations on tensors and the implementation of advanced neural networks. The problem is the lack of knowledge of the ideal neural network architecture that would handle this task the best. Therefore, a study was conducted, exploring various possibilities, such as the number of hidden layers, the number of neurons in each hidden layer, the activation functions and the network training process optimizer. Details of the architectures used are as follows:

• Number of hidden neurons: 12–2048;
• Number of hidden layers: 1–8;
• Activation fun.: RELu, Tanh, identity, logistic;
• Optimizer: Adaptive Moment Estimation, Stochastic Gradient Descent.

The use of artificial neural networks (ANN) in the context of cost prediction problems has found application in a number of scientific studies on such topics as predicting the cost and duration of building construction or predicting the cost of housing engineering [74,75].

5. The Results

We start the analysis with a presentation of the calculation of common integral measures.

Table 4. Comparison of the regression models. Grey color highlights the extreme values of the models: the worst (red) and the best (green). Bold numbers indicate the worst and the best one.

No	Descriptor	Method Name	MAE	MSE	MAPE [%]
1	LMS	Least Squares	205.8	704,903	66.24
2	R-LMS	Robust Linear Regression	189.2	837,335	37.69
3	SLR	Stepwise Linear Regression	226.8	1,664,193	75.34
4	TS-LR	Theil-Sen Regressor	188.4	851,928	46.61
5	H-LR	Huber Regressor	177.6	763,898	41.4
6	LSVM	Linear Support Vector Machines	175.6	764,256	36.11
7	KSVM	Kernel Support Vector Machines	298.9	1,943,548	60.79
8	QSVM	Quadratic Support Vector Machine	187.1	877,200	45.61
9	CGSVM	Coarse Gaussian Support Vector Machines	195.6	1,088,107	50.5
10	MGSVM	Medium Gaussian Support Vector Machines	235.0	1,479,565	56.79
11	FGSVM	Fine Gaussian Support Vector Machines	304.8	1,746,572	79.59
12	EGPR	Exponential Gaussian Process Regression	155.5	748,644	42.58
13	SEGPR	Squared Exponential Gaussian Process Regression	202.4	1,193,684	42.76
14	MGPR	Matern 5/2 Gaussian Process Regression	182.1	967,428	42.37
15	RGPR	Rational Quadratic Gaussian Process Regression	159.7	718,013	38.31
16	k-NN	k-Nearest Neighbors Regressor	200.9	824,671	57.08
17	OMP	Orthogonal Matching Pursuit	198.0	858,053	40.65
18	RR	Ridge Regression	205.8	704,902	66.24
19	ARD-RR	Automatic Relevance Determination	205.4	704,991	65.99
20	B-RR	Bayesian Ridge Regression	205.7	704,758	66.31
21	DTR	Decision Tree Regressor	167.9	614,842	35.45
22	BoostRT	Boosted Regression Trees	164.7	719,251	33.58
23	GBoostRT	Gradient Boosting Regression	151.8	695,577	38.04
24	HGBoostRT	Histogram Gradient Boosting Regression	140.0	490,936	34.01
25	ERTR	Extremely Randomized Trees	131.2	712,589	27.68
26	BRTR	Bagged Regression Trees	128.5	626,157	26.83
27	F-DTR	Fine Regression Tree	161.6	688,117	29.23
28	M-DTR	Medium Regression Tree	140.6	675,587	27.12
29	C-DTR	Coarse Regression Tree	130.5	609,023	26.25
30	RFR	Random Forest Regression	136.1	625,584	30.77
31	LASSO-R	LASSO Regression	205.9	705,301	66.54
32	LARS-R	LARS Lasso	205.9	705,301	66.54
33	ENR	Elastic Net Regression	204.9	702,740	66.39
34	LAR	Least Angle Regression	205.8	704,903	66.24
35	ANN	Artificial Neural Network	134.0	651,000	27.82

compares obtained values. Even the draft analysis allows for an interesting observation. Each measure indicates different models as the best. The MSE highly penalizes large errors [76] and is sensitive to any outlying occurrences, while the MAE is less conservative. It enables closer relations to smaller variations and economic considerations [77].

Moreover, relative indexes point out or penalize other methods. Therefore, the decisions about the model that should be chosen highly depends on the selected index. Practice shows that this decision is often unaware, which might be costly in further practical applications.

In Section 3, we describe various performance indexes that can be found in the literature. Moreover, we suggest the performance of visual multi-criteria analysis using the so-called Index ratio Diagrams (IRD). This idea follows the notions of moment ratio diagrams, known in statistics.

We start the analysis from the classical moment ratio diagram that shows the relationship between the third and the fourth moment, i.e., between the skewness denoted as ${\gamma }_3$ and the kurtosis ${\gamma }_4$. Figure 3 presents the respective diagram. Each shaded circle denotes one model, which is labeled with the blue number according to the notation sketched in Table 4. The circles are shaded according to some other index, in this case it is the MAE. Generally, such a drawing brings some relative visual information—we still expect to obtain a single performance indicator. Actually, we may assume that the best tuning is reflected by the shortest distance from some optimal point. In this case we may assume the point $\left[{\gamma }_3;{\gamma }_4\right]=\left[0;3\right]$.

As we wish to obtain this value independently, we scale it and obtain the following IRD distance index d_IRD(x,y) for scaled values x and y:

$$\mathrm{ADiMe}=d_{\mathrm{IRD}(x,y)}=\frac{1}{\sqrt{2}}\sqrt{{\left(x-x_0\right)}^2+{\left(y-y_0\right)}^2}.$$

(25)

This index we name as the Aggregated Distance Measure (ADiMe). The assumed scaling factors, which are used in each case, are denoted on the plots.

The IRD(${\gamma }_3$,${\gamma }_4$) diagram points out the model no 24, i.e., the Histogram Gradient Boosting Regression (HGBoostRT) as the best modeling approach. We may observe that selected model is the same as the one selected by the MSE index. The value of the IRD distance index is equal to $d_{\mathrm{IRD}(x,y)}=$ 0.674. What is interesting is that the second-best model is the Quadratic Support Vector Machine (QSVM), which is not appreciated by any of the integral indexes.

The next Figure 4 presents the same IRD relationship IRD(${\gamma }_3$,${\gamma }_4$), but with different shading, which is conducted according to the relative MAPE index. It is decided that all the consecutive diagrams are shaded according to the MAE index.

The next two plots show the IRD diagrams showing the relationship between the standard deviation (the second moments) and the skewness. Figure 5 uses classical Gaussian standard deviation estimator ${\sigma }_{\mathrm{G}}$, while Figure 6 uses its robust counterpart ${\sigma }_{\mathrm{H}}$. In both cases the HGBoostRT model is indicated with the ADiMe measure equal to $d_{\mathrm{IRD}({\sigma }_{\mathrm{G}},{\gamma }_3)}=0.691$ and $d_{\mathrm{IRD}({\sigma }_{\mathrm{H}},{\gamma }_3)}=0.590$. In contrast, the next-best models are different, i.e., the Fine Regression Tree (F-DTR) and Coarse Regression Tree (C-DTR).

Figure 7 presents standard L-Moment Ratio Diagram, i.e., the IRD(${\tau }_3$,${\tau }_4$). As the L-skewness and L-kurtosis are normalized, there is no need for any further scaling. This approach indicates the k-Nearest Neighbors Regressor (k-NN) with $d_{\mathrm{IRD}({\tau }_3,{\tau }_4)}=0.463$. Interestingly, the two next-best models are Rational Quadratic Gaussian Process Regression (RGPR) and Exponential Gaussian Process Regression (EGPR). The favoring of these models is intriguing, because they are not indicated by other indicators. We may bring the hypothesis that their residua exhibit, in general, neutral statistical properties. This issue requires further investigation.

The following two diagrams takes into account the L-l2 scale measure together with the L-skewness in Figure 8 and L-kurtosis in Figure 9. The IRD(L-$l_2$,${\tau }_3$) approach points out the Coarse Regression Tree (C-DTR) with $d_{\mathrm{IRD}(\mathrm{L}\text{-}{l}_2,{\tau }_3)}=0.449$, which is highly favored by all integral measures. On the contrary, the IRD(L-$l_2$,${\tau }_4$) selects the Orthogonal Matching Pursuit (OMP) method with $d_{\mathrm{IRD}(\mathrm{L}\text{-}{l}_2,{\tau }_4)}=0.680$. It must be noted that the selected best models are quite close to the following ones, and thus the indications are not very decisive.

The next two plots combine the L-scale factor L-l2 with the alternative measures of the tail—the tail index $\widehat{\xi }$ in Figure 10 and the Geweke–Porter–Hudak ARFIMA filter fractional order estimator $\widehat{d}$ shown in Figure 11. Both diagrams select the same modeling approach, the Bagged Regression Trees (BRTR), which is highly favored by the MAE index. The ADiMe values for both approaches are $d_{\mathrm{IRD}(\widehat{\xi },{\tau }_3)}=0.570$ and $d_{\mathrm{IRD}(\widehat{d},{\tau }_3)}=0.503$, respectively. In both cases, the second-best model is Extremely Randomized Trees (ERTR).

Finally, the IRD diagrams are constructed using factors of the $\alpha$-stable distribution. Figure 12 presents the model selection according to the combination of the skewness $\beta$ and the stability exponent $\alpha$. The considered optimal point is the $\left[\beta ;\alpha \right]=\left[0;2\right]$, where this point indicates normal distribution. In that sense, the Stepwise Linear Regression model (SLR) is selected with the $d_{\mathrm{IRD}(\beta ,\alpha )}=0.220$. The next-best models are the RGPR—Rational Quadratic Gaussian Process Regression—and the k-NN.

The last two plots present the combination of the scale factor $\gamma$ together with stability exponent $\alpha$ in Figure 13 and with the skewness $\beta$ in Figure 14. The first one selects the Bagged Regression Trees (BRTR) approach ($d_{\mathrm{IRD}(\gamma ,\alpha )}=0.589$), while the latter the Extremely Randomized Trees (ERTR)—$d_{\mathrm{IRD}(\gamma ,\beta )}=0.226$. Both modeling approaches are seriously favored by the integral indexes.

It should be noted that the difference between these two diagrams is quite significant. The shape factor $\alpha$ is responsible for the tails, i.e., informs about the ratio of the outlying observations, while the second shape factor, the skewness $\beta$ measures the residuum asymmetry. With this difference kept in mind we may select which feature of the modeling error is considered to be the most important for us.

Finally, let us compare all the approaches and the features they favor and the models they indicate.

Table 5. Comparison of the ADiMe indications.

Diagram	Feature	Selected Model
		Number	Name
IRD(${\gamma }_3$,${\gamma }_4$)	symmetry	24	Histogram Gradient Boosting Regression
	normality
IRD(${\sigma }_{\mathrm{G}}$,${\gamma }_3$)	symmetry	24	Histogram Gradient Boosting Regression
	fluctuations
IRD(${\sigma }_{\mathrm{H}}$,${\gamma }_3$)	symmetry	24	Histogram Gradient Boosting Regression
	fluctuations
IRD(${\tau }_3$,${\tau }_4$)	symmetry	16	k-Nearest Neighbors Regressor
	normality
	unbiased by outliers
IRD(L-$l_2$,${\tau }_3$)	symmetry	29	Coarse Regression Tree
	fluctuations
	unbiased by outliers
IRD(L-$l_2$,${\tau }_4$)	normality	17	Orthogonal Matching Pursuit
	fluctuations
	unbiased by outliers
IRD(L-$l_2$,$\widehat{\xi }$)	outliers (tails)	26	Bagged Regression Trees
	fluctuations
	unbiased by outliers
IRD(L-$l_2$,$\widehat{d}$)	normality	26	Bagged Regression Trees
	fluctuations
	unbiased by outliers
IRD($\beta$,$\alpha$)	normality	3	Stepwise Linear Regression
	symmetry
	unbiased by outliers
IRD($\gamma$,$\alpha$)	normality	26	Bagged Regression Trees
	fluctuations
	unbiased by outliers
IRD($\gamma$,$\beta$)	symmetry	25	Extremely Randomized Trees
	fluctuations
	unbiased by outliers

aggregates the features favored by each of the IRD diagrams with the selected model.

Generally, the regression trees approaches are the best fitted to the considered estimation task. However, each method has different features and objective comparison is highly relative and sensitive to the selection of the index. The Histogram Gradient Boosting Regression method captures the outliers and utilizes them in the estimation, while the k-NN approach focuses on the bulk of the data and neglects the outliers.

Concluding, one should first define which feature of the estimations matters the most, and according to that one should select the assessment methodology and the indexes used. The scaling indexes and the visual inspection of the IRD diagrams deliver an additional degree of freedom, allowing for deeper insight into the properties of the considered model.

6. Conclusions and Further Research

This works focuses on two aspects. It addresses the issue of the cost estimation for the short routes of the external FTL fleet. This subject is hardly recognized in the literature, it is really difficult and has large practical importance.

The second contribution, in our opinion the most important, is connected with the model assessment. It is very subjective to assess the model, as we do not have a single universal measure. Each index favors different properties. If we neglect that fact, the resulting model can miss our expectations without any clue why. The model assessment should not be limited to the simple comparison of a single measure numbers, but deeper investigation and appropriate index selection, even using visual inspection, might help. We propose to use the novel approach using Index Ratio Diagrams (IRD) and resulting Aggregated Distance Measure (ADiMe).

The practical perspective of this research has three dimensions. It bridges the gap between statistics and machine learning, as nowadays researchers tend to forget the potential lying in statistical analysis. The review of the ML-based estimation reports and papers does not deliver positive conclusions. Almost always, authors do not try to assess why their model is so good or so bad. They simply report residuum measure index, often using one single value. They do not try to check whether their selection captures data properties. This work aims to show that the assessment task is not one-dimensional and the analysis of the nuances can improve the work and the knowledge. This fact has further and more significant consequences. Obtained models might be not so good as observed, and therefore the industrial end-user can be frustrated by the results. That might lead to a lack of satisfaction, no further use of the tool and general robustness to new ideas.

Finally, this work offers the method of multicriteria residual analysis accompanied with new, almost unknown measuring opportunities that can highlight currently unobserved properties.

The proposed method is not universal and some limitations might be observed. First of all, especially at the level of the results presentation to the end-user, it might be challenging to explain the conclusions. Also, the selection of the scaling index values might be subjective and the work on the results normalization is still required. It would be interesting to conduct more research aimed at the synthesis of the observation in the direction of the Pareto-front analysis. The connection between the IRD observations, their explanation and the way to improve the model should be investigated.

The analysis is still not over. The model assessment, though considered simple, is not as simple as perceived. One index is not equal to another index. This mistake can lead to costly consequences. A lot of subjects remain open. How can we assess models using various criteria? How can we combine our expectations about the model features with proper performance index selection? How can we make the residuum analysis simple, clear and comparable?