A Decision-Making Tool for Algorithm Selection Based on a Fuzzy TOPSIS Approach to Solve Replenishment, Production and Distribution Planning Problems

Abstract

A wide variety of methods and techniques with multiple characteristics are used in solving replenishment, production and distribution planning problems. Selecting a solution method (either a solver or an algorithm) when attempting to solve an optimization problem involves considerable difficulty. Identifying the best solution method among the many available ones is a complex activity that depends partly on human experts or a random trial-and-error procedure. This paper addresses the challenge of recommending a solution method for replenishment, production and distribution planning problems by proposing a decision-making tool for algorithm selection based on the fuzzy TOPSIS approach. This approach considers a collection of the different most commonly used solution methods in the literature, including distinct types of algorithms and solvers. To evaluate a solution method, 13 criteria were defined that all address several important dimensions when solving a planning problem, such as the computational difficulty, scheduling knowledge, mathematical knowledge, algorithm knowledge, mathematical modeling software knowledge and expected computational performance of the solution methods. An illustrative example is provided to demonstrate how planners apply the approach to select a solution method. A sensitivity analysis is also performed to examine the effect of decision maker biases on criteria ratings and how it may affect the final selection. The outcome of the approach provides planners with an effective and systematic decision support tool to follow the process of selecting a solution method.

1. Introduction

The supply chain comprises different sequential activities, such as replenishment, production and distribution, which must all be planned and optimized. The main management function of companies is planning [1]. Planning activities aim to effectively coordinate and schedule a company’s available resources [2]. Planning is accompanied by a set of decisions to be made by the planning manager; for example, a planner must make decisions about the quantity of materials needed for production by taking into account storage capacity and production batches to reduce production and inventory costs, production scheduling and sequencing on machines, and to finally make decisions about the delivery flow of finished products to customers or distribution centers [3].

Many real-world combinatorial optimization problems, such as those in transportation and logistics [4,5,6] and manufacturing [7,8,9], pose a huge challenge due to the high complexity of most companies’ operations given the type of industry to which they belong. They are also subject to not only dynamic conditions, such as customer demands, processing times, returns on investment, but also to uncertainties, such as unavailability of items, changes in market conditions and shortages due to changes in demand [10].

Thus, planning problems seek to maximize profit or gain while minimizing costs and meeting market, environmental and societal constraints. For example, in supply planning problems, there is a direct relation between inventory costs and the costs associated with distribution planning, such as transportation costs and on-time delivery to customers [11]. Therefore, the difficulty of such problems is substantial due to the amount of data they handle [12], nonlinearities and discontinuities, complex constraints, possible conflicting objectives and uncertainty [13]. Hence, different types of solvers are used to solve these problems, as are algorithms because of their computational difficulty [14].

Given the large number of algorithms for solving replenishment [15], production [16] and distribution planning problems [17], how to effectively select an algorithm for a given task or a specific problem is an important, but also difficult issue. Peres and Castelli [18] highlight that rules which standardize the formulations of existing combinatorial optimization problems (COP) in planning are lacking, which means that researchers have to start building an algorithm from scratch, which thus limits the interoperability of this field because the algorithms in the literature must be adjusted to solve a specific problem. These authors conclude that the consolidation of combinatorial optimization problems is lacking and note that this is important for the field of COPs to reach a higher degree of maturity.

The algorithm selection problem (ASP) is an active research area in many fields, such as operations research [19,20,21] and artificial intelligence (AI) [22,23]. For many decades, researchers have developed increasingly sophisticated techniques and algorithms to solve difficult optimization problems [18]. These techniques include mathematical programming approaches, heuristics, metaheuristics, nature-inspired metaheuristics, matheuristics and various hybridizations [24]. Literature reviews such as that presented by Jamalnia et al. [25], who reviewed the aggregate production planning problem under uncertainty between 1970 and 2018, detailed the use of approximately 24 different techniques to solve this type of problem out of 92 reviewed papers. Kumar et al. [26] presented a literature review covering the period from 2000 to 2019 of the quantitative approaches used to solve production and distribution planning problems. They found 13 different techniques and types of solvers, including CPLEX and LINGO, to solve this type of problem out of 74 papers. Pereira et al. [27] analyzed the tactical sales and operations planning problem. To do so, they reviewed 103 papers, where the year was not limited. They detailed about 35 different techniques to solve this type of problem. Hussain et al. [28] conducted a literature review of the applications of metaheuristic algorithms and found 140 different metaheuristic algorithms in 1222 publications over a 33-year search period (1983 to 2016).

Different research papers have conducted experimental studies to determine the performance of an algorithm [29,30,31,32] or several algorithms according to a problem type with a collection of datasets available in the literature [33,34,35]. For example, Pan et al. [36] compared three constructive heuristics and four metaheuristics (discrete artificial bee colony, scatter search, iterated local search, iterated greedy algorithm) for the distributed permutation flowshop problem, for which they made extensive comparative evaluations based on 720 instances. However, these comparisons do not provide any enlightening results because they are generally limited to a set of algorithms and to a specific problem set [24].

In practice, algorithm performance vastly varies from one problem state to another. In many cases, heuristic [37], metaheuristic [28] and matheuristic [38] techniques involve randomization, such as genetic algorithm, particle swarm optimization, bee swarm optimization, bat algorithm, artificial tribe algorithm and firefly algorithm [39,40,41,42,43], which results in performance variability, even across repeated trials in a single problem instance [44]. Risk is an important additional feature of algorithms because the planner or the person in charge of selecting an algorithm for planning must be willing to settle for average or lower performance in exchange for a reasonable answer or may also find a better solution than that expected in the same resolution time. This situation is often encountered in companies that attempt to maximize their profits because these problems are solved by constructing mixed strategies, i.e., strategies that meet the desired risk and return.

Nowadays, if a study demonstrates the superiority of one algorithm over other algorithms, that algorithm can be expected to be useful for other problem types for which it has not yet been tested. No-free-lunch (NFL) theorems [45] describe that there is no single algorithm that outperforms all algorithms in all the instances of a problem [24].

Therefore, the selection of the most suitable algorithm to solve an optimization problem for replenishment, production and distribution planning is a very difficult task. Algorithm selection requires advanced knowledge of the efficiency of algorithms, the characteristics of the problem, as well as mathematical and statistical knowledge. However, having the necessary knowledge to find a solution with algorithms does not guarantee success [46].

Algorithm selection depends mainly on the expected results and the data that the company has at the time. Therefore, the properties or characteristics of the business problem must be examined. For this purpose, the linearity of the problem, the number of parameters and the characteristics that the solution supports must be analyzed.

Evaluating algorithms to solve a problem usually involves more than one criterion, such as problem type, problem knowledge, performance, computation time, the quality of the expected solution and programming knowledge. Therefore, algorithm selection can be modeled as a multicriteria decision making problem [22].

The objective of multicriteria decision making (MCDM) is to identify the most eligible alternatives from a set of alternatives based on qualitative and/or quantitative criteria with different units of measurement to select or rank them [47]. Different techniques such as AHP, ELECTRE, PROMETHEE, SAW, TOPSIS and VIKOR are used to solve MCDM problems [3]. Several studies have been conducted to compare the performance of these techniques; for example, that presented by Zanakis et al. [48], which compared eight MCDM techniques (four variations of AHP, ELECTRE, TOPSIS and SAW). It concluded that different techniques are affected mainly by the number of alternatives because as alternatives increase, methods tend to generate similar final rankings. Opricovic and Tzeng [49] performed a comparative analysis of the VIKOR and TOPSIS methods. Both these methods are based on an aggregation function that represents the closeness to the ideal. The study revealed that the main differences between the two methods were the employed normalization method types.

Opricovic and Tzeng [50] compared the extended VIKOR method to ELECTRE II, PROMETHEE and TOPSIS. The obtained results showed that ELECTRE II, PROMETHEE and VIKOR gave similar results, while TOPSIS presented different results in some alternatives. Chu et al. [51] made a comparison of the VIKOR, TOPSIS and SAW methods. The study revealed that SAW and TOPSIS presented similar classifications, while VIKOR presented different results. These authors concluded that VIKOR and TOPSIS provided results that were close to reality. Ozcan et al. [52] presented a comparative analysis of the TOPSIS, ELECTRE and Grey Theory techniques for the warehouse location selection problem, where the Grey Theory provided different results to TOPSIS and ELECTRE. Instead, the last two obtained similar results.

In situations in which information is not quantifiable or incomplete, as in real-world problems where data may be incomplete or imprecise, i.e., nondeterministic, data can be represented in a fuzzy way using linguistic variables to represent decision makers’ preferences in complex or not well-defined situations. Imprecision in MCDM problems can be modeled using the fuzzy Set Theory, which is used to extend different MCDM techniques. In this background, Ertuğrul and Karakaşoğlu [53] conducted a comparative study of the fuzzy AHP and fuzzy TOPSIS methods for the facility location selection problem. Both methods obtained the same results, i.e., the same rank order of alternatives.

Other studies have used an extension of the classical fuzzy set called the intuitionistic fuzzy set, as proposed by Atanassov [54]. Intuitionistic fuzzy sets have been applied in many fields, such as facility location selection [55], supplier selection [56], evaluation of project and portfolio management information systems [57,58], and personnel selection [59]. Büyüközkan and Güleryüz [60] compared the performance of ranked fuzzy TOPSIS and intuitionistic fuzzy TOPSIS by detailing how the alternatives ranking barely differed between the two approaches.

From the above comparisons, it is clear that many techniques are available for multi-criteria decision making [61]. These techniques have their advantages and limitations over others depending on the type of problem [62].

Different MCDM techniques have been used for the classification algorithm selection problem, such as the study by Lamba et al. [63] in which TOPSIS and VIKOR were used to evaluate 20 classification algorithms. Both methods obtained similar results. Peng et al. [22] used four different MCDM techniques (TOPSIS, VIKOR, PROMETHEE II and WSM) to select multiclass classification algorithms. The TOPSIS, VIKOR and PROMETHEE II methods achieved similar classifications, while WSM obtained slightly different ones. Peng et al. [64] evaluated ranking algorithms for financial risk prediction purposes. Using TOPSIS, PROMETHEE and VIKOR, they obtained similar results for the three main ranking algorithms. They concluded that the followed techniques were advantageous for choosing a classification algorithm.

Along these lines, TOPSIS stands out as a widely used technique that is efficient for selecting classification algorithms. It has been successful in different areas such as supply chain and logistics management, environment and energy management, health and safety management, business and marketing management, engineering and manufacturing, human resource management and transportation management [47,65,66,67] and, according to Chu et al. [51], is able to represent reality. It is also useful for companies because it can be run with a spreadsheet [68]. For all these reasons and given the fact that the choice of a solution method is subject to vagueness and uncertainty, we use the fuzzy TOPSIS method.

In this context, the present paper aims to answer this question: which solution method is suitable for a replenishment, production and distribution planning problem given a portfolio of algorithms or solvers?

To answer this question, and by taking into account that no research to date has analyzed the selection of algorithms for planning with a multicriteria decision method, a decision-making tool to select algorithms for a planning problem based on fuzzy TOPSIS is presented. To validate the use of the tool herein proposed, an illustrative example is presented, which has been validated by four different manufacturing companies. This paper is organized as follows. Section 2 deals with the literature review. The adopted methodology is shown in Section 3 and the numerical application of the methodology is presented in Section 4. The sensitivity analysis of the results is provided in Section 5. Finally, Section 6 includes the conclusions and future research lines.

2. Algorithm Selection Problem Literature Review

Algorithm selection has been widely addressed by the scientific community in both the mathematics [69,70] and Artificial Intelligence (AI) [71,72] areas. In the mathematical area, Stützle and Fernandes [73] report how the characteristics of problem instances make the performance of metaheuristics relative to the properties of instances. Therefore, it is necessary to explore the relation between algorithms and instances. In the AI area, different models have been developed to predict which algorithm is the best one for a problem instance, which is conducted by analyzing the relation between the characteristics of an instance and a set of training data used by an algorithm. In this way, with an algorithms portfolio it is possible to predict which algorithm in a new problem instance is most likely to work [74].

Growing interest has been shown in the ASP to put previously developed algorithms to best use to solve a specific problem instead of developing new ones [75]. According to Leyton-Brown et al. [76], some algorithms are better than others on average, and there is rarely a best algorithm for a given problem. Instead “it is often the case that different algorithms perform well on different problem instances. This phenomenon is most pronounced among algorithms for solving NP-Hard problems, because runtimes for these algorithms are often highly variable from instance to instance”. In this context, Rice [77] proposes the first description of methodologies to select algorithms. Kotthoff [75] defines this as the “task of algorithm selection involves choosing an algorithm from a set of algorithms on a per-instance basis in order to exploit the varying performance of algorithms over a set of instances”.

In this regard, algorithm selection approaches have been successfully applied in different problem domains [78]. The following table summarizes a literature review of the various papers that have approached ASP from different perspectives (

Table 1. Research studies addressing the algorithm selection problem.

Author	Proposal
Lagoudakis and Littman [79]	Algorithm selection using reinforcement learning.
Xu et al. [80]	A scalable and completely automated portfolio construction. The authors improve the ASP methodology by integrating local search solvers as candidate solvers by predicting performance scores instead of runtime, and by using hierarchical hardness models that take into account different types of instances.
Smith-Miles [81]	A unified framework to take the algorithm selection problem as a learning problem and to use this framework to tie together cross-disciplinary developments in tackling the algorithm selection problem. The authors generalize metalearning concepts to algorithms that focus on tasks, including sorting, forecasting, constraint satisfaction and optimization.
Bischl et al. [35]	An algorithm selection problem as a cost-sensitive classification task that is based on an Exploratory Landscape Analysis.
Hoos et al. [82]	A modular open-solver architecture that integrates several different portfolio-based algorithm selection approaches and techniques.
Kotthoff [75]	An algorithm selection for combinatorial search problems.
Tierney and Malitsky [83]	An algorithm selection benchmark based on optimal search algorithms to solve the container premarshalling problem (CPMP), an NP-hard problem from the container terminal optimization field.
Cunha et al. [84]	A metalearning method is used to select the best recommendation algorithms within different scopes to allow to understand the relations between data characteristics and the relative performance of recommendation algorithms, which can be used to select the best algorithm(s) for a new problem. This work analyzes the algorithm selection problem for Recommender Systems by focusing on Collaborative Filtering.
Bożejko et al. [85]	A local and optima network analysis and machine learning is used to select appropriate algorithms on an instance-to-instance basis.
Drozdov et al. [86]	Graph convolutional network-based generative adversarial networks for the algorithm selection problem in classification terms.
Vilas Boas et al. [87]	Integer programming-based approaches to build decision trees for the algorithm selection problem. These techniques allow the automation of three crucial decisions by discerning the most important problem features to determine problem classes by grouping problems into classes, and then selecting the best algorithm configuration for each class.
Marrero et al. [88]	An efficient parallel genetic algorithm (GA) is proposed as a first step to solve the algorithm selection problem. GA is able to attain competitive results in optimal objective value terms and in a short time. The computational results show that the approach is able to efficiently scale and considerably reduce the average elapsed time to solve Knapsack Problem (KNP) instances.
De Carvalho et al. [21]	A cross-domain evaluation for multi-objective optimization. The authors investigate how four state-of-the-art online hyperheuristics with different characteristics perform to find solutions for 18 real-world multi-objective optimization problems. These hyperheuristics were designed in previous studies and tackle the algorithm selection problem from different perspectives: election-based, based on Reinforcement Learning and based on a mathematical function.

).

In manufacturing environments, formulations are usually very complex [78] because they present a variety of specific constraints related to the company’s scope. Generally, these formulations can serve as blocks or subproblems for other formulations of other specific manufacturing environments. In this way, many formulations or algorithms can obtain similar results to the formulations proposed above. When selecting a formulation or algorithm, tuning the parameters of the different techniques is a very demanding task because each algorithm has different characteristics and the number of times that a parameter tuning has to be performed against different instances of a problem when performing a comparison can exponentially grow [33]. Furthermore, to compare algorithms and select one, the feature set of the instances must be taken into account because the characterization of instances determines a solution approach’s performance. In practice, the information needed to establish the characteristics is not always available [89], and experimental results may lead to the fact that there is no single best or worst algorithm for all problem instances [46]. In this context, and as shown in Table 1, several approaches have been proposed to address the algorithm selection challenge, including heuristic algorithms, metaheuristics, hybrid metaheuristics, hyperheuristics, and machine-learning techniques. Many of these approaches integrate similarities, such as using a set of instances to learn, measuring or predicting the performance of the best algorithm. The success of algorithm selection approaches for some problem domains has motivated us to develop a decision making tool to support planners of companies to select a solution method (algorithm or a solver) for replenishment, production and distribution planning problems.

3. Solution Methodology

For combinatorial optimization problems with realistic discrete decision variables, such as scheduling, sequencing, distribution and transportation planning problems, performing an exhaustive search space for this problem type is not a realistic option despite having a finite search space. The literature includes several heuristic, metaheuristic and matheuristic algorithms, as well as tests with commercial and non-commercial high-performance solvers to solve such problems. So, this question arises: which algorithm is to be chosen for a combinatorial optimization problem?

Generally one way of finding an algorithm to solve a combinatorial optimization problem is to exhaustively run all the available algorithms and choose the best solution [90]. However, this method requires unlimited computational resources and companies have limited computational, programming and mathematical resources, which makes it impossible to test all the algorithms or to use several solvers to test one instance or several for a specific problem. Weise et al. [12] emphasize that there is a variety of methods to solve different types of problems with acceptable performance, but they can be outperformed by very specialized methods.

Weise et al. [12] consider that there is no optimization method that is better or can outperform others, and the NFL Theorem [45] corroborates this theory. This theorem states that no optimization algorithm is likely to outperform several existing types of methods in different types of problems.

In turn, the same authors mention that the efficiency of an optimization algorithm is based on knowledge of a problem. Radcliffe [91] emphasizes that the algorithm’s performance will improve with adequate knowledge of the problem. However, knowledge of one type of problem can be misleading for another type of problem [89] because there is no algorithm that outperforms others in all instances of a problem. Therefore, an algorithm’s performance will be based on experience and empirical results.

Algorithm selection schemes are based mainly on approaches that either run a sequence of algorithms in a limited execution time [80,82] or predict the performance of an algorithm for a given instance and select the algorithm with the best predicted performance [92].

Real-world planning problems are subject to inaccuracies and uncertainties, conflicts between constraints and objectives, discontinuities and nonlinearities [13]. Therefore, determining which algorithm is appropriate poses a challenge that can be analyzed using a multicriteria decision technique for ranking and prioritizing algorithms because algorithm selection involves multiple decisions that require the simultaneous assessment of the various advantages and disadvantages.

In most companies, the complexity of operations has several components that must be addressed at the same time. Evaluating an algorithm to solve a problem often involves more than one criterion, such as problem type, problem knowledge, performance, computation time, the quality of the expected solution and programming knowledge.

MCDM techniques integrate different criteria and an order of preference to evaluate and select the optimal option among multiple alternatives based on the expected outcome. The objective of these techniques is to obtain an ideal solution to a problem in which a decision makers’ experience does not allow them to decide among the various considered parameters. As a result, a ranking is obtained according to the selected criteria, their respective values and the assigned weights [93].

There are many criteria in real-life problems that can directly or indirectly affect the outcome of different decisions. Decision making often involves inaccuracies and vagueness that can be effectively dealt with using fuzzy sets. This method is especially important for clarifying decisions that are difficult to quantify or compare, especially if decision makers have different perspectives, as in this study. Therefore, we herein adopt the fuzzy TOPSIS methodology to model an algorithm or solver selection given a solution methods portfolio to solve replenishment, production and distribution planning problems.

In decision making problems, the Fuzzy Set Theory was introduced by Zadeh [94] to overcome the ambiguity and uncertainty of human thought and reasoning by using linguistic terms to represent decision makers’ choices.

The TOPSIS method was originally proposed by Hwang and Yoon [95]. It is based on choosing an alternative that should have the shortest distance between the positive ideal solution (PIS) and the negative ideal solution (NIS), i.e., the selected alternative is obtained with the closest solution to the PIS and is farthest away from the NIS. The main limitation of this technique is that it cannot capture ambiguity in the decision making process [96]. To overcome this limitation, Chen [97] developed the Fuzzy TOPSIS Method to quantitatively evaluate the score of different alternatives by conferring weight to the different criteria described with linguistic variables. This section briefly describes the employed Fuzzy Set Theory and Fuzzy TOPSIS Method.

3.1. Fuzzy Set Theory and Fuzzy Numbers

The Fuzzy Set Theory [94,98,99] is associated with the TOPSIS method, and are related to another by the degree of membership of the elements in fuzzy sets. A fuzzy set is characterized by the membership function, which can come in different formats, e.g., triangular, sigmoid or trapezoidal. The membership function assigns a degree of membership to each object according to its relevance ${\mu }_A\left(x\right):x\to \left[0.0, 1.0\right]$. To represent a fuzzy set, a tilde ‘∼’ is placed [68].

For our study, we consider a triangular fuzzy number, $\tilde{A},$ which is denoted by its vertices (l, m, u), as shown in Figure 1. Triangular fuzzy numbers are used to adapt decision makers’ preference to capture the vagueness of linguistic evaluations, where l, u and m, respectively, denote the lower bound, the upper bound and the crisp central value.

Membership function d of triangular fuzzy number $\tilde{A}$ is defined as:

$${\mu }_{\tilde{A}}\left(x\right)=\{\begin{array}{c}\frac{x-l}{m-l}, l \le x \le m,\\ \frac{u-x}{u-m}, m \le x \le n,\\ 0, otherwise\end{array}$$

(1)

where $\tilde{A}=\left(l_A, m_A,u_A\right)$ and $\tilde{B}=\left(l_B, m_B,u_B\right)$ are two triangular fuzzy numbers with bases l, m, u. Then, the basic operational laws for triangular numbers are defined as:

$$\widetilde{A }(+) \tilde{B}=\left(l_A, m_A,u_A\right) (+) \left(l_B, m_B,u_B\right)=\left(l_A+l_B, m_A+m_B , u_A+u_B\right)$$

(2)

$$\widetilde{A }(-) \tilde{B}=\left(l_A, m_A,u_A\right) (-) \left(l_B, m_B,u_B\right)=(l_A-l_B, m_A-m_B , u_A-u_B$$

(3)

$$\widetilde{A }(\times ) \tilde{B}=\left(l_A, m_A,u_A\right) (\times ) \left(l_B, m_B,u_B\right)=\left(l_A\times l_B, m_A\times m_B , u_A\times u_B\right) \mathrm{for} l_{A }, l_B>0 ; m_{A }, m_B>0; u_{A }, u_B>0$$

(4)

$$\widetilde{A }(÷) \tilde{B}=\left(l_A, m_A,u_A\right) (÷) \left(l_B, m_B,u_B\right)=\left(\frac{l_A}{u_B}, \frac{m_A}{m_B}, \frac{u_A}{l_B}\right) \mathrm{for} l_{A }, l_B>0 ; m_{A }, m_B>0; u_{A }, u_B>0$$

(5)

$$k\widetilde{A }=k{l}_A,k{m}_A,k{u}_A$$

(6)

$${\widetilde{A }}^{-1}={\left(l_A, m_A,u_A\right)}^{-1}=\left(\frac{1}{u_A}, \frac{1}{m_A}, \frac{1}{l_A}\right) \mathrm{for} l_{A }, l_B>0 ; m_{A }, m_B>0; u_{A }, u_B>0$$

(7)

By assuming that fuzzy numbers $\tilde{A}$ and $\tilde{B}$ are real numbers, then the distance measure is identical to the Euclidean distance. Therefore, the vertex method is defined to calculate the distance between two fuzzy numbers (see Equation (8)). Although there are several ways of measuring distances between fuzzy numbers [100], the vertex method is a simple and efficient method [97,101].

$$d\left(\tilde{A}, \tilde{B}\right)=\sqrt{\frac{1}{3}\left[{\left(l_A-l_B\right)}^2+{\left(m_A-m_B\right)}^2+{\left(u_A-u_B\right)}^2\right] }$$

(8)

3.2. The Fuzzy TOPSIS Method

The main fuzzy TOPSIS idea is based on defining the fuzzy positive ideal solution (FPIS) and the fuzzy negative ideal solution (FNIS). The chosen alternative should have the shortest distance to the FPIS and the farthest distance to the FNIS. TOPSIS follows a systematic process and logic that seek to express the logic of human choice [102]. The basic fuzzy TOPSIS method steps are described in the following way (see [97,103,104]):

Step 1. Consider a set of k decision makers (D₁, D₂,…, D_k) with m alternatives (A₁, A₂,…, A_n) and n criteria (C₁, C₂,…, C_n) for which the decision matrix is established:

$$\begin{array}{c}{C}_1 C_2 \dots C_n\tilde{D}=\begin{array}{c}\begin{array}{c}{A}_1\\ A_2\end{array}\\ ⋮\\ A_m\end{array} \left[\begin{array}{cc}\begin{array}{cc}{\tilde{X}}_{11}& {\tilde{X}}_{12}\\ {\tilde{X}}_{21}& {\tilde{X}}_{22}\end{array}& \begin{array}{cc}\dots & {\tilde{X}}_{1n}\\ \dots & {\tilde{X}}_{2n}\end{array}\\ \begin{array}{cc}⋮& ⋮\\ {\tilde{X}}_{n1}& {\tilde{X}}_{n2}\end{array}& \begin{array}{cc}⋮& ⋮\\ \dots & {\tilde{X}}_{nm}\end{array}\end{array}\right] i=1, 2, \dots , m; j=1, 2, \dots , n\\ \tilde{W}=\left[{\tilde{w}}_1, {\tilde{w}}_{2 }, \dots ,{\tilde{w}}_{n }\right] \end{array}$$

(9)

Considering that the perception of algorithms and solvers varies according to knowledge and experience with algorithms for planning, the average value method is applied; where ${\tilde{x}}_{ij}^k$ is the rating or score of the alternative A_i in relation to criterion C_j evaluated by the K-th decision maker (Equation (10)). The weights of criteria are aggregated using Equation (11), where ${\tilde{w}}_j^k$ describes the weight of each criterion C_j according to decision makers D_k.

$${\tilde{x}}_{ij}=\frac{1}{k}\left({\tilde{x}}_{ij}^1+{\tilde{x}}_{ij}^2+\dots +{\tilde{x}}_{ij}^k\right)$$

(10)

$${\tilde{w}}_j=\frac{1}{k}\left({\tilde{w}}_j^1+{\tilde{w}}_j^2+\dots +{\tilde{w}}_j^k\right)$$

(11)

Step 2. Normalize the fuzzy decision matrix. Decision matrix $\tilde{D}$ with m alternatives and n criteria is normalized to eliminate inconsistencies with the different units of measurement or scales to preserve the ranges of the normalized triangular fuzzy numbers. $\tilde{R}$ represents the normalized decision matrix (Equation (12)):

$${\tilde{R}}_j={\left[{\tilde{r}}_{ij}\right]}_{m\times n}, i=1, 2, \dots , m;j=1, 2, \dots , n$$

(12)

The normalization process is performed by Equations (13) and (14), where B and C represent the set of benefit and cost criteria, respectively.

$${\tilde{r}}_{ij}=\left(\frac{l_{ij}}{u_j^{+}}, \frac{ij}{u_j^{+}}, \frac{u_{ij}}{u_j^{+}}\right), \mathrm{and} u_j^{+}=ma{x}_i{u}_{ij }if j \in B$$

(13)

$${\tilde{r}}_{ij}=\left(\frac{l_j^{-}}{u_{ij}}, \frac{l_j^{-}}{m_{ij}}, \frac{l_j^{-}}{l_{ij}}\right), \mathrm{and} l_j^{-}=mi{n}_i{l}_{ij }if j \in C$$

(14)

Step 3. Construct the weighted normalized fuzzy decision matrix $\widetilde{V }$(Equation (15)). ${\tilde{v}}_{ij}$ is obtained by multiplying the weights of criteria ${\tilde{w}}_j$ and the normalized fuzzy decision matrix ${\tilde{r}}_{ij}$ values:

$$\tilde{V}={\left[{\tilde{v}}_{ij}\right]}_{m\times n} , i=1, 2, \dots , m;j=1, 2, \dots , n$$

(15)

$${\tilde{v}}_{ij}={\tilde{r}}_{ij }\times {\tilde{w}}_j$$

(16)

Step 4. Obtain the FPIS (FPIS, A⁺) and the FNIS (FNIS, A⁻), as shown in Equations (17) and (18), respectively. The ideal solutions can be defined according to Chen [97] as: ${\tilde{v}}_j^{+}=\left(1, 1, 1\right)$ and ${\tilde{v}}_j^{-}=\left(0, 0, 0\right)$

$$A^{+}=\left\{{\tilde{v}}_1^{+}, {\tilde{v}}_j^{+}, \dots , {\tilde{v}}_m^{+}\right\}$$

(17)

$$A^{-}=\left\{{\tilde{v}}_1^{-}, {\tilde{v}}_j^{-}, \dots , {\tilde{v}}_m^{-}\right\}$$

(18)

Step 5. Calculate the distances for each alternative, where $D_i^{+}$ indicates the distance between the scores of alternative A_i to the FPIS (Equation (19)), and $D_i^{-}$ denotes the distances between the values of alternative A_i to the FNIS (Equation (20)), where $d\left({\tilde{v}}_a, {\tilde{v}}_b\right)$ represents the distance between two fuzzy numbers.

$$D_i^{+}={{\displaystyle \sum }}_{j=1}^{n}d\left({\tilde{v}}_{ij}, {\tilde{v}}_j^{+}\right) , i=1, 2, \dots , m;j=1, 2, \dots , n$$

(19)

$$D_i^{-}={{\displaystyle \sum }}_{j=1}^{n}d\left({\tilde{v}}_{ij}, {\tilde{v}}_j^{-}\right) , i=1, 2, \dots , m;j=1, 2, \dots , n$$

(20)

Step 6. Determine proximity coefficient $C{C}_i$, which evaluates the rank order of all the alternatives A_i according to their overall performance. The proximity coefficient is calculated as shown in Equation (21).

$$C{C}_i=\frac{D_i^{-}}{\left(D_i^{+}+D_i^{-}\right)}$$

(21)

Step 7. Rank alternatives A_i, using a decreasing order of CC_i values, the shortest distances from the FPIS, i.e., close to 1, to indicate that the overall performance of alternative A_i is better because it is farther away from the FNIS. Having obtained the ranking order, decision makers select the most feasible alternative A_i.

4. The Methodological Approach for the Algorithm Selection Problem

This paper employs a three-stage methodology to select an algorithm or solver to solve a replenishment, production and distribution planning problem (see Figure 2). The objective of this section is to present a numerical analysis to demonstrate the performance of the proposed methodology.

The three stages of the proposed methodology are described in the following subsections.

4.1. Stage 1—Define Criteria and Alternatives

We first identify the different criteria that are taken into account when selecting a solution method; these criteria can be identified in the literature and are based on the opinion of experts in the field [105]. According to each identified criterion, the decision maker evaluates the suitability of a solution method for the type of problem; that is, how algorithms or solvers can be suitable and formulated for a given problem.

In this research, 13 criteria are identified based on an exhaustive review of the literature (see [8,18,25,27,106,107]) and the assessments of experts in the field of operations research. These criteria are presented in Table 2.

Table 2. Criteria for algorithm selection.

Id	Criteria	Definition
C1	Problem type	The replenishment (source), production (make) and distribution (deliver) planning problem type is determined by the SCOR (Supply Chain Operation Reference) methodology [106,108] (see Figure 3). Each problem type has its own characteristics and computational difficulty. According to Weise et al. [12], it is very difficult to make accurate estimates of a problems’ computational performance because a solution method’s performance will almost always depend on experience, the empirical results based on related research areas and the rules of thumb established for these problems. So, a problem’s computational performance depends on different factors. Some of the main factors of a problem’s complexity are: problem size, linearity, variables and presence of constraints [109]. Based on these considerations, criteria C2–C7 are proposed.
C2	Equation type	It expresses the equations present in the problem. These equations can be linear or nonlinear.
C3	Variable type	It represents the elements to be modeled. Variables can be integer, binary and continuous. Planning problems generally contain a combination of variables: Continuous + Integer, Integer + Binary, Continuous + Binary, Continuous + Integer + Binary. These combinations normally generate greater computational difficulty. Each combination can generate a different behavior for the solution method because algorithm or solver performance is linked with the amount of resources used. These resources can be: amount of memory, processing time to deal with each type of variable [12].
C4	Number of instantiated variables	It determines the number of variables present in a problem, which is a determining factor when establishing the expected response time to obtain an answer.
C5	Type of constraints and solutions	The constraint type determines the computational difficulty that the problem will have because constraints express limitations of resources. Some constraints can be expressed as follows: ▪ decision >= data (e.g., production >= demand) ▪ decision <= data (e.g., load <= capacity) ▪ decision == data (e.g., production >= demand) ▪ decision >= decision (e.g., production of A >= production of B) ▪ decision <= decision (e.g., load of M <= load of N) ▪ decision == decision (e.g., inventory A == inventory B) ▪ continuity equations of some variables (e.g., Inventory == Inventory prior period + Production - Demand) One factor that affects a problem’s difficulty is when the expected solutions to the problem contain a route or sequence. These routing planning or sequencing problems are generally NP-hard [110,111].
C6	Number of constraints	The number of constraints contained in a problem can be a limiting factor for establishing the problem’s difficulty. Therefore, the evaluator analyzes whether the set of constraints can be adapted to an algorithm or to a solver.
C7	Dataset size	When representing the problem input data size, a problem’s computation is directly related to the amount of data.
C8	Programming knowledge	Programming knowledge is a determining factor when selecting an algorithm because it determines decision makers’ ability to program one algorithm or several algorithms when having to test different algorithms in the hope to obtain a solution that meets the company’s needs.
C9	Mathematical knowledge	Mathematical knowledge is important when choosing whether to express the problem as a mathematical model or to directly choose an algorithm. Algorithms generally require certain mathematical knowledge.
C10	Knowledge of algorithms	One aspect to take into account in companies is knowledge of the different algorithms.
C11	Software	This criterion is considered if the company has mathematical modeling software, but is not considered if the company does not. If the company has specific optimization software, the decision maker defines the scope of its performance against each alternative to solve a planning problem.
C12	Quality of solutions	This criterion establishes the quality of the expected solutions to the problem. These solutions can be optimal, near-optimal or good.
C13	Calculation time	The computation time sets the amount of expected time to obtain a solution for the problem.

Table 2. Criteria for algorithm selection.

Id	Criteria	Definition
C1	Problem type	The replenishment (source), production (make) and distribution (deliver) planning problem type is determined by the SCOR (Supply Chain Operation Reference) methodology [106,108] (see Figure 3). Each problem type has its own characteristics and computational difficulty. According to Weise et al. [12], it is very difficult to make accurate estimates of a problems’ computational performance because a solution method’s performance will almost always depend on experience, the empirical results based on related research areas and the rules of thumb established for these problems. So, a problem’s computational performance depends on different factors. Some of the main factors of a problem’s complexity are: problem size, linearity, variables and presence of constraints [109]. Based on these considerations, criteria C2–C7 are proposed.
C2	Equation type	It expresses the equations present in the problem. These equations can be linear or nonlinear.
C3	Variable type	It represents the elements to be modeled. Variables can be integer, binary and continuous. Planning problems generally contain a combination of variables: Continuous + Integer, Integer + Binary, Continuous + Binary, Continuous + Integer + Binary. These combinations normally generate greater computational difficulty. Each combination can generate a different behavior for the solution method because algorithm or solver performance is linked with the amount of resources used. These resources can be: amount of memory, processing time to deal with each type of variable [12].
C4	Number of instantiated variables	It determines the number of variables present in a problem, which is a determining factor when establishing the expected response time to obtain an answer.
C5	Type of constraints and solutions	The constraint type determines the computational difficulty that the problem will have because constraints express limitations of resources. Some constraints can be expressed as follows: ▪ decision >= data (e.g., production >= demand) ▪ decision <= data (e.g., load <= capacity) ▪ decision == data (e.g., production >= demand) ▪ decision >= decision (e.g., production of A >= production of B) ▪ decision <= decision (e.g., load of M <= load of N) ▪ decision == decision (e.g., inventory A == inventory B) ▪ continuity equations of some variables (e.g., Inventory == Inventory prior period + Production - Demand) One factor that affects a problem’s difficulty is when the expected solutions to the problem contain a route or sequence. These routing planning or sequencing problems are generally NP-hard [110,111].
C6	Number of constraints	The number of constraints contained in a problem can be a limiting factor for establishing the problem’s difficulty. Therefore, the evaluator analyzes whether the set of constraints can be adapted to an algorithm or to a solver.
C7	Dataset size	When representing the problem input data size, a problem’s computation is directly related to the amount of data.
C8	Programming knowledge	Programming knowledge is a determining factor when selecting an algorithm because it determines decision makers’ ability to program one algorithm or several algorithms when having to test different algorithms in the hope to obtain a solution that meets the company’s needs.
C9	Mathematical knowledge	Mathematical knowledge is important when choosing whether to express the problem as a mathematical model or to directly choose an algorithm. Algorithms generally require certain mathematical knowledge.
C10	Knowledge of algorithms	One aspect to take into account in companies is knowledge of the different algorithms.
C11	Software	This criterion is considered if the company has mathematical modeling software, but is not considered if the company does not. If the company has specific optimization software, the decision maker defines the scope of its performance against each alternative to solve a planning problem.
C12	Quality of solutions	This criterion establishes the quality of the expected solutions to the problem. These solutions can be optimal, near-optimal or good.
C13	Calculation time	The computation time sets the amount of expected time to obtain a solution for the problem.

Second, we identify the portfolio of solution methods (alternatives). This portfolio is composed of a set of nine algorithms and four solvers, identified as the most commonly used ones in the planning problems reported in [107,112]. Alternatives are divided between different algorithm types, which are:

• ▪ Heuristic algorithms (HA). They are used when solvers or exact techniques cannot reach solutions in acceptable computation times. These techniques do not provide optimal solutions, but can offer solutions that come very close to the optimum in acceptable computation times [113];
• ▪ Metaheuristic algorithms (MA). According to Swan et al. [114], these techniques are: “an iterative master process that guides and modifies the operations of subordinate heuristics to efficiently produce high-quality solutions. At each iteration, it manipulates either a complete (or partial) single solution or a collection of such solutions”;
• ▪ Matheuristic algorithms (MTA). They combine mathematical programming techniques and heuristic or metaheuristic algorithms [115].

The alternatives in this classification are A1—HA/Benders’ decomposition, A2—HA/LP and Fix, A3—HA/LP Relaxation, A4—MA/Tabu Search, A5—MA/Genetic Algorithm, A6—MA/Simulated annealing, A7—MA/Variable Neighborhood Search, A8—MTA/ Genetic Algorithm + Mathematical Model, A9—MTA/Simulated annealing + Mathematical Model. Different solver types used to solve planning problems are also considered. For this purpose, commercial and non-commercial solvers are identified to deal with mathematical models with linear and nonlinear equations. These are: A10—CPLEX (Commercial), A11—CBC (Non-Commercial), A12—BONMIN (Non-Commercial—Nonlinear), A13—LINDO (Commercial—Linear/Nonlinear).

Figure 3. Replenishment, production and distribution planning problem types.

Figure 3. Replenishment, production and distribution planning problem types.

The hierarchical structure that has been defined and constructed to assist in the process of selecting an algorithm or solver is shown in Figure 4. This structure is composed of four layers: the first one corresponds to the objective of this study; the second structure corresponds to the characterization of the different dimensions that have been proposed, which is composed of four dimensions (the problem type and its characteristics, programming knowledge, the software and the expected performance of algorithms or solvers); in the third layer comes the categorization of the 13 identified criteria; in the last one, methods or solution alternatives appear. The correlation between layer 3 and 4 is related to the performance of an algorithm or a solution method.

4.2. Stage 2—Problem Statement

In this stage, the type of planning problem to be addressed was defined, for which four expert decision makers working in different manufacturing companies in the planning area were invited to propose a planning problem. The decision makers proposed that the problem to be studied should be a production planning problem falling within the make classification, as shown in Figure 3.

Once the problem type has been defined, a questionnaire is developed to obtain the weight of preference of criteria and to thus evaluate alternatives according to the criteria. To devise the questionnaire, it is necessary to construct a fuzzy linguistic scale.

Linguistic scales are used to transform linguistic terms into fuzzy numbers [96]. Linguistic terms are subjective categories of the linguistic variable [116]. Zadeh [117] introduced the linguistic variable concept. A linguistic variable is a variable whose values allow computation with words instead of numbers [118]. Linguistic variables are used to represent decision makers’ assessments, estimates and subjectivity [119].

To evaluate the criteria, we use a scale between 0 and 1. To rate the alternatives, we employ a scale from 0 to 10 [97]. The linguistic scales that evaluate the weights of the criteria and alternatives are shown in

Table 3. Linguistic scales to assess the criteria and alternatives (Chen [97]).

Linguistic Expression for Rating Alternatives (Algorithms)				Linguistic Variable for the Relative Importance Weight of Criteria
Linguistic Expression	l	m	u	Linguistic Expression	l	m	u
Very Low (VL)	0.1	0.1	2.5	Very Low Importance (VLI)	0.01	0.03	0.25
Low (L)	0.1	2.5	5.0	Low Importance (LI)	0.01	0.25	0.50
Moderate (M)	2.5	5.0	7.5	Medium Importance (MI)	0.25	0.50	0.75
High (H)	5.0	7.5	10.0	High Importance (HI)	0.50	0.75	1.00
Very High (VH)	7.5	10.0	10.0	Very High Importance (VHI)	0.75	1.00	1.00

.

Finally, decision makers were invited to review the questionnaire and to check its content. Based on this review, we were able to adjust the questionnaire.

In this same stage, we invited the four decision makers who worked in the planning area to evaluate the alternatives and to determine the weights of the criteria. For this purpose, we asked the decision makers to use the linguistic scale described in Table 3. An extract of the questionnaires used by the decision makers is shown in

Table A1. Questionnaire used to know decision makers’ preferences for the identified criteria.

	Very Low Importance (VLI)	Low Importance (LI)	Medium Importance (MI)	High Importance (HI)	Very High Importance (VHI)
C1
C2
C3
C4
C5
C6
C7
C8
C9
C10
C11
C12
C13

and

Table A2. Questionnaire used to know the decision makers’ preferences for the 13 alternatives according to the criteria.

	C1
	Very Low (VL)	Low (L)	Moderate (M)	High (H)	Very High (VH)
A1
A2
A3
A4
A5
A6
A7
A8
A9
A10
A11
A12
A13

.

Table 4. Decision makers’ linguistic assessment of the criteria.

	D1	D2	D3	D4
C1	MI	MI	MI	MI
C2	LI	LI	LI	LI
C3	VHI	VHI	VHI	VHI
C4	HI	HI	HI	HI
C5	LI	LI	LI	LI
C6	HI	MI	MI	MI
C7	HI	VHI	VHI	MI
C8	LI	MI	HI	HI
C9	LI	MI	HI	HI
C10	MI	MI	LI	HI
C11	LI	LI	LI	LI
C12	VHI	VHI	VHI	VHI
C13	VHI	VHI	VHI	VHI

details the fuzzy weights of each criterion based on the linguistic scales selected by the decision makers. The decision makers’ ratings of the alternatives against all criteria are shown in

Table A3. Decision maker 1′s linguistic assessment.

	C1	C2	C3	C4	C5	C6	C7	C8	C9	C10	C11	C12	C13
A1	L	H	M	M	M	M	M	L	L	L	VL	M	H
A2	M	H	H	M	M	M	M	L	VL	L	L	M	H
A3	VH	H	H	M	M	M	M	L	M	L	L	M	H
A4	VH	H	VL	M	M	M	M	VH	VL	H	VL	VH	H
A5	VH	VH	H	M	M	M	M	H	L	H	VH	H	VH
A6	M	M	L	M	M	M	M	L	M	L	L	M	L
A7	M	M	L	M	M	M	M	L	M	VL	L	M	M
A8	L	L	L	M	M	M	M	L	L	L	L	M	M
A9	L	L	L	M	M	M	M	L	M	L	L	M	M
A10	H	VH	H	M	H	M	M	M	H	H	H	VH	VH
A11	M	M	M	M	M	M	M	L	M	M	M	M	VH
A12	L	L	L	M	L	M	M	L	L	L	L	L	L

,

Table A4. Decision maker 2′s linguistic assessment.

	C1	C2	C3	C4	C5	C6	C7	C8	C9	C10	C11	C12	C13
A1	L	L	L	L	VL	L	H	L	L	VL	VL	VL	M
A2	H	H	H	L	M	L	H	L	VL	M	L	VL	M
A3	VH	H	H	H	M	M	H	L	M	M	L	M	M
A4	VH	H	VL	H	M	M	H	VH	VL	H	VL	VH	H
A5	VH	VH	H	H	M	M	H	H	L	H	VH	H	VH
A6	M	M	L	M	M	L	H	L	M	L	L	M	L
A7	M	M	L	M	VL	L	H	L	M	VL	L	M	M
A8	L	L	L	M	VL	L	H	L	L	L	L	M	M
A9	L	L	L	M	VL	L	H	L	M	L	L	M	M
A10	H	VH	H	M	H	M	H	M	H	VH	H	VH	VH
A11	M	M	M	M	M	L	H	L	M	M	M	M	VH
A12	L	VL	L	VL	VL	L	L	L	L	L	L	L	L

,

Table A5. Decision maker 3′s linguistic assessment.

	C1	C2	C3	C4	C5	C6	C7	C8	C9	C10	C11	C12	C13
A1	L	VL	L	VL	VL	M	H	L	L	L	VL	VL	L
A2	L	M	L	L	VL	M	H	H	H	H	L	VL	M
A3	H	M	M	L	M	M	H	VH	H	H	L	M	M
A4	H	M	M	VL	VH	H	H	VH	H	VL	VL	VH	H
A5	H	M	M	VH	H	VH	H	VH	VH	H	VH	H	VH
A6	M	M	L	L	M	L	H	M	M	H	H	M	H
A7	M	VL	L	L	M	M	H	M	M	L	L	M	M
A8	M	VL	L	L	M	M	H	L	L	L	L	M	M
A9	M	VL	L	L	M	M	H	L	L	L	L	M	M
A10	M	H	M	H	M	M	H	H	M	M	M	M	M
A11	M	M	L	M	M	M	H	M	M	M	M	M	M
A12	VL	VL	L	L	L	L	L	L	VL	L	L	L	L
A13	VL	VL	L	L	L	L	L	L	L	L	L	L	L

and

Table A6. Decision maker 4′s linguistic assessment.

	C1	C2	C3	C4	C5	C6	C7	C8	C9	C10	C11	C12	C13
A1	L	L	L	VL	M	H	H	L	L	L	VL	M	L
A2	H	H	H	VL	M	H	H	L	VL	L	L	M	L
A3	VH	H	H	M	M	H	H	L	M	L	L	M	M
A4	VH	H	VL	VH	H	H	H	VH	VL	H	VL	VH	M
A5	VH	VH	H	H	VH	M	H	H	L	H	VH	H	H
A6	M	M	H	M	L	M	H	L	M	L	L	M	H
A7	M	M	L	M	M	M	H	L	M	VL	L	M	L
A8	L	L	L	M	M	M	H	L	L	L	L	M	L
A9	L	L	L	M	M	M	H	L	M	L	L	M	L
A10	H	M	M	M	M	M	H	M	H	H	H	H	H
A11	M	M	M	M	M	H	H	L	M	M	M	VH	VH
A12	L	VL	L	L	L	L	L	L	L	L	L	L	L
A13	L	L	L	L	L	L	L	L	L	L	L	L	L

.

4.3. Stage 3—Application of the Fuzzy TOPSIS Method

In this stage, the Fuzzy TOPSIS Method is used to analyze the different alternatives in relation to the identified criteria. The process used to apply the Fuzzy TOPSIS Method consists of five steps, which are detailed below.

Step 1. Based on the linguistic assessments of the alternatives (see Table A3, Table A4, Table A5 and Table A6), the linguistic terms are converted into fuzzy numbers according to Table 3 and the fuzzy decision matrix is constructed. The aggregation of the ratings is performed using the fuzzy arithmetic mean, and the aggregate ratings for each alternative are obtained using Equation (10) (see

Table 5. Decision matrix with the aggregated scores.

	C1			C2			C3			C4			C5			C6			C7			C8			C9			C10			C11			C12			C13
	l	m	u	l	m	u	l	m	u	l	m	u	l	m	u	l	m	u	l	m	u	l	m	u	l	m	u	l	m	u	l	m	u	l	m	u	l	m	u
A1	0.10	2.50	5.00	1.33	3.15	5.63	0.70	3.13	5.63	0.70	1.93	4.38	1.30	2.55	5.00	2.53	5.00	7.50	4.38	6.88	9.38	0.10	2.50	5.00	0.10	2.50	5.00	0.10	1.90	4.38	0.10	0.10	2.50	1.30	2.55	5.00	1.93	4.38	6.88
A2	3.15	5.63	8.13	4.38	6.88	9.38	3.78	6.25	8.75	0.70	2.53	5.00	1.90	3.78	6.25	2.53	5.00	7.50	4.38	6.88	9.38	1.33	3.75	6.25	1.33	1.95	4.38	1.93	4.38	10.00	0.10	2.50	5.00	1.30	2.55	5.00	2.53	5.00	7.50
A3	6.88	9.38	10.00	4.38	6.88	9.38	4.38	6.88	9.38	2.53	5.00	7.50	2.50	5.00	7.50	3.13	5.63	8.13	4.38	6.88	9.38	1.95	4.38	6.25	3.13	5.63	8.13	1.93	4.38	7.50	0.10	2.50	5.00	2.50	5.00	7.50	3.13	5.63	8.13
A4	6.88	9.38	10.00	4.38	6.88	9.38	0.70	1.33	3.75	3.78	5.65	7.50	4.38	6.88	8.75	3.75	6.25	8.75	4.38	6.88	9.38	7.50	10.00	10.00	1.33	1.95	4.38	3.78	5.65	10.00	0.10	0.10	2.50	7.50	10.00	10.00	4.38	6.88	9.38
A5	6.88	9.38	10.00	6.25	8.75	9.38	4.38	6.88	9.38	5.00	7.50	9.38	4.38	6.88	8.75	3.75	6.25	8.13	4.38	6.88	9.38	5.63	8.13	10.00	1.95	4.38	6.25	5.00	7.50	10.00	7.50	10.00	10.00	5.00	7.50	10.00	6.88	9.38	10.00
A6	2.50	5.00	7.50	2.50	5.00	7.50	1.33	3.75	6.25	1.90	4.38	6.88	1.90	4.38	6.88	1.30	3.75	6.25	4.38	6.88	9.38	0.70	3.13	5.63	2.50	5.00	7.50	1.33	3.75	5.00	1.33	3.75	6.25	2.50	5.00	7.50	2.55	5.00	7.50
A7	2.50	5.00	7.50	1.90	3.78	6.25	0.10	2.50	5.00	1.90	4.38	6.88	1.90	3.78	6.25	1.90	4.38	6.88	4.38	6.88	9.38	0.70	3.13	5.63	2.50	5.00	7.50	0.10	0.70	10.00	0.10	2.50	5.00	2.50	5.00	7.50	1.90	4.38	6.88
A8	0.70	3.13	5.63	0.10	1.90	4.38	0.10	2.50	5.00	1.90	4.38	6.88	1.90	3.78	6.25	1.90	4.38	6.88	4.38	6.88	9.38	0.10	2.50	5.00	0.10	2.50	5.00	0.10	2.50	5.00	0.10	2.50	5.00	2.50	5.00	7.50	1.90	4.38	6.88
A9	0.70	3.13	5.63	0.10	1.90	4.38	0.10	2.50	5.00	1.90	4.38	6.88	1.90	3.78	6.25	1.90	4.38	6.88	4.38	6.88	9.38	0.10	2.50	5.00	1.90	4.38	6.88	0.10	2.50	10.00	0.10	2.50	5.00	2.50	5.00	7.50	1.90	4.38	6.88
A10	4.38	6.88	9.38	5.63	8.13	9.38	3.75	6.25	8.75	3.13	5.63	8.13	3.75	6.25	8.75	2.50	5.00	7.50	4.38	6.88	9.38	3.13	5.63	8.13	4.38	6.88	9.38	5.00	7.50	10.00	4.38	6.88	9.38	5.63	8.13	9.38	5.63	8.13	9.38
A11	2.50	5.00	7.50	2.50	5.00	7.50	1.90	4.38	6.88	2.50	5.00	7.50	2.50	5.00	7.50	2.53	5.00	7.50	4.38	6.88	9.38	0.70	3.13	5.63	2.50	5.00	7.50	2.50	5.00	7.50	2.50	5.00	7.50	3.75	6.25	8.13	6.25	8.75	9.38
A12	0.10	1.90	4.38	0.10	0.70	3.13	0.10	2.50	5.00	0.70	2.53	5.00	0.10	1.90	4.38	0.70	3.13	5.63	0.70	3.13	5.63	0.10	2.50	5.00	0.10	1.90	4.38	0.10	2.50	10.00	0.10	2.50	5.00	0.10	2.50	5.00	0.10	2.50	5.00
A13	0.10	1.90	4.38	0.10	1.90	4.38	0.10	2.50	5.00	0.70	2.53	5.00	0.10	1.90	4.38	0.70	3.13	5.63	0.70	3.13	5.63	0.10	2.50	5.00	0.10	2.50	5.00	0.10	2.50	10.00	0.10	2.50	5.00	0.10	2.50	5.00	0.10	2.50	5.00

).

In order to obtain the aggregate weights of each criterion, the fuzzy weights of each criterion are used, which are extracted by converting the linguistic terms of the four decision makers (see Table 4) into fuzzy numbers according to Table 3; for example, the fuzzy weights of criterion C7 of the four decision makers are D₁ = (0.50, 0.75, 1.00), D₂ = (0.75, 1.00, 1.00), D₃ = (0.75, 1.00, 1.00), D₄ = (0.25, 0.50, 0.75), and applying Equation (11), the aggregate fuzzy weight of C7 = (0.56, 0.81, 0.93) is obtained. The results of the aggregate fuzzy weights of all the criteria are tabulated in

Table 6. Aggregate fuzzy weights for each criterion.

Criteria	Aggregate Fuzzy Weights
C1	(0.25, 0.50, 0.75)
C2	(0.01, 0.25, 0.50)
C3	(0.75, 1.00, 1.00)
C4	(0.50, 0.75, 1.00)
C5	(0.01, 0.25, 0.50)
C6	(0.31, 0.56, 0.81)
C7	(0.56, 0.81, 0.93)
C8	(0.32, 0.56, 0.81)
C9	(0.32, 0.56, 0.81)
C10	(0.25, 0.50, 0.75)
C11	(0.01, 0.25, 0.50)
C12	(0.75, 1.00, 1.00)
C13	(0.75, 1.00, 1.00)

.

Step 2 and Step 3. Using Equations (13) and (14), the normalized fuzzy decision matrix is obtained. For criteria C1–12, Equation (13) is used because the objective of these criteria is to maximize. For criterion C13, Equation (14) is applied because the aim is to minimize the computation time criterion.

Table A7. Normalized fuzzy decision matrix.

	C1			C2			C3			C4			C5			C6			C7			C8			C9			C10			C11			C12			C13
	l	m	u	l	m	u	l	m	u	l	m	u	l	m	u	l	m	u	l	m	u	l	m	u	l	m	u	l	m	u	l	m	u	l	m	u	l	m	u
A1	0.01	0.25	0.50	0.14	0.34	0.60	0.07	0.33	0.60	0.07	0.21	0.47	0.15	0.29	0.57	0.29	0.57	0.86	0.47	0.73	1.00	0.01	0.25	0.50	0.01	0.27	0.53	0.01	0.19	0.44	0.01	0.01	0.25	0.13	0.26	0.50	0.01	0.02	0.05
A2	0.32	0.56	0.81	0.47	0.73	1.00	0.40	0.67	0.93	0.07	0.27	0.53	0.22	0.43	0.71	0.29	0.57	0.86	0.47	0.73	1.00	0.13	0.38	0.63	0.14	0.21	0.47	0.19	0.44	1.00	0.01	0.25	0.50	0.13	0.26	0.50	0.01	0.02	0.04
A3	0.69	0.94	1.00	0.47	0.73	1.00	0.47	0.73	1.00	0.27	0.53	0.80	0.29	0.57	0.86	0.36	0.64	0.93	0.47	0.73	1.00	0.20	0.44	0.63	0.33	0.60	0.87	0.19	0.44	0.75	0.01	0.25	0.50	0.25	0.50	0.75	0.01	0.02	0.03
A4	0.69	0.94	1.00	0.47	0.73	1.00	0.07	0.14	0.40	0.40	0.60	0.80	0.50	0.79	1.00	0.43	0.71	1.00	0.47	0.73	1.00	0.75	1.00	1.00	0.14	0.21	0.47	0.38	0.57	1.00	0.01	0.01	0.25	0.75	1.00	1.00	0.01	0.01	0.02
A5	0.69	0.94	1.00	0.67	0.93	1.00	0.47	0.73	1.00	0.53	0.80	1.00	0.50	0.79	1.00	0.43	0.71	0.93	0.47	0.73	1.00	0.56	0.81	1.00	0.21	0.47	0.67	0.50	0.75	1.00	0.75	1.00	1.00	0.50	0.75	1.00	0.01	0.01	0.01
A6	0.25	0.50	0.75	0.27	0.53	0.80	0.14	0.40	0.67	0.20	0.47	0.73	0.22	0.50	0.79	0.15	0.43	0.71	0.47	0.73	1.00	0.07	0.31	0.56	0.27	0.53	0.80	0.13	0.38	0.50	0.13	0.38	0.63	0.25	0.50	0.75	0.01	0.02	0.04
A7	0.25	0.50	0.75	0.20	0.40	0.67	0.01	0.27	0.53	0.20	0.47	0.73	0.22	0.43	0.71	0.22	0.50	0.79	0.47	0.73	1.00	0.07	0.31	0.56	0.27	0.53	0.80	0.01	0.07	1.00	0.01	0.25	0.50	0.25	0.50	0.75	0.01	0.02	0.05
A8	0.07	0.31	0.56	0.01	0.20	0.47	0.01	0.27	0.53	0.20	0.47	0.73	0.22	0.43	0.71	0.22	0.50	0.79	0.47	0.73	1.00	0.01	0.25	0.50	0.01	0.27	0.53	0.01	0.25	0.50	0.01	0.25	0.50	0.25	0.50	0.75	0.01	0.02	0.05
A9	0.07	0.31	0.56	0.01	0.20	0.47	0.01	0.27	0.53	0.20	0.47	0.73	0.22	0.43	0.71	0.22	0.50	0.79	0.47	0.73	1.00	0.01	0.25	0.50	0.20	0.47	0.73	0.01	0.25	1.00	0.01	0.25	0.50	0.25	0.50	0.75	0.01	0.02	0.05
A10	0.44	0.69	0.94	0.60	0.87	1.00	0.40	0.67	0.93	0.33	0.60	0.87	0.43	0.71	1.00	0.29	0.57	0.86	0.47	0.73	1.00	0.31	0.56	0.81	0.47	0.73	1.00	0.50	0.75	1.00	0.44	0.69	0.94	0.56	0.81	0.94	0.01	0.01	0.02
A11	0.25	0.50	0.75	0.27	0.53	0.80	0.20	0.47	0.73	0.27	0.53	0.80	0.29	0.57	0.86	0.29	0.57	0.86	0.47	0.73	1.00	0.07	0.31	0.56	0.27	0.53	0.80	0.25	0.50	0.75	0.25	0.50	0.75	0.38	0.63	0.81	0.01	0.01	0.02
A12	0.01	0.19	0.44	0.01	0.07	0.33	0.01	0.27	0.53	0.07	0.27	0.53	0.01	0.22	0.50	0.08	0.36	0.64	0.07	0.33	0.60	0.01	0.25	0.50	0.01	0.20	0.47	0.01	0.25	1.00	0.01	0.25	0.50	0.01	0.25	0.50	0.02	0.04	1.00
A13	0.01	0.19	0.44	0.01	0.20	0.47	0.01	0.27	0.53	0.07	0.27	0.53	0.01	0.22	0.50	0.08	0.36	0.64	0.07	0.33	0.60	0.01	0.25	0.50	0.01	0.27	0.53	0.01	0.25	1.00	0.01	0.25	0.50	0.01	0.25	0.50	0.02	0.04	1.00

shows the results of the normalized matrix.

After normalization, the weighted normalized decision matrix is calculated using Equation (16). The results are shown in

Table A8. Weighted normalized fuzzy decision matrix.

	C1			C2			C3			C4			C5			C6			C7			C8			C9			C10			C11			C12			C13
	l	m	u	l	m	u	l	m	u	l	m	u	l	m	u	l	m	u	l	m	u	l	m	u	l	m	u	l	m	u	l	m	u	l	m	u	l	m	u
A1	0.00	0.13	0.38	0.00	0.08	0.30	0.06	0.33	0.60	0.04	0.15	0.47	0.00	0.07	0.29	0.09	0.32	0.70	0.26	0.60	0.94	0.00	0.14	0.41	0.00	0.15	0.43	0.00	0.10	0.33	0.00	0.00	0.13	0.10	0.26	0.50	0.01	0.02	0.05
A2	0.08	0.28	0.61	0.00	0.18	0.50	0.30	0.67	0.93	0.04	0.20	0.53	0.00	0.11	0.36	0.09	0.32	0.70	0.26	0.60	0.94	0.04	0.21	0.51	0.04	0.12	0.38	0.05	0.22	0.75	0.00	0.06	0.25	0.10	0.26	0.50	0.01	0.02	0.04
A3	0.17	0.47	0.75	0.00	0.18	0.50	0.35	0.73	1.00	0.13	0.40	0.80	0.00	0.14	0.43	0.11	0.36	0.75	0.26	0.60	0.94	0.06	0.25	0.51	0.11	0.34	0.70	0.05	0.22	0.56	0.00	0.06	0.25	0.19	0.50	0.75	0.01	0.02	0.03
A4	0.17	0.47	0.75	0.00	0.18	0.50	0.06	0.14	0.40	0.20	0.45	0.80	0.01	0.20	0.50	0.13	0.40	0.81	0.26	0.60	0.94	0.24	0.56	0.81	0.04	0.12	0.38	0.10	0.28	0.75	0.00	0.00	0.13	0.56	1.00	1.00	0.01	0.01	0.02
A5	0.17	0.47	0.75	0.01	0.23	0.50	0.35	0.73	1.00	0.27	0.60	1.00	0.01	0.20	0.50	0.13	0.40	0.75	0.26	0.60	0.94	0.18	0.46	0.81	0.07	0.26	0.54	0.13	0.38	0.75	0.01	0.25	0.50	0.38	0.75	1.00	0.01	0.01	0.01
A6	0.06	0.25	0.56	0.00	0.13	0.40	0.11	0.40	0.67	0.10	0.35	0.73	0.00	0.13	0.39	0.05	0.24	0.58	0.26	0.60	0.94	0.02	0.18	0.46	0.08	0.30	0.65	0.03	0.19	0.38	0.00	0.09	0.31	0.19	0.50	0.75	0.01	0.02	0.04
A7	0.06	0.25	0.56	0.00	0.10	0.33	0.01	0.27	0.53	0.10	0.35	0.73	0.00	0.11	0.36	0.07	0.28	0.64	0.26	0.60	0.94	0.02	0.18	0.46	0.08	0.30	0.65	0.00	0.04	0.75	0.00	0.06	0.25	0.19	0.50	0.75	0.01	0.02	0.05
A8	0.02	0.16	0.42	0.00	0.05	0.23	0.01	0.27	0.53	0.10	0.35	0.73	0.00	0.11	0.36	0.07	0.28	0.64	0.26	0.60	0.94	0.00	0.14	0.41	0.00	0.15	0.43	0.00	0.13	0.38	0.00	0.06	0.25	0.19	0.50	0.75	0.01	0.02	0.05
A9	0.02	0.16	0.42	0.00	0.05	0.23	0.01	0.27	0.53	0.10	0.35	0.73	0.00	0.11	0.36	0.07	0.28	0.64	0.26	0.60	0.94	0.00	0.14	0.41	0.06	0.26	0.60	0.00	0.13	0.75	0.00	0.06	0.25	0.19	0.50	0.75	0.01	0.02	0.05
A10	0.11	0.34	0.70	0.01	0.22	0.50	0.30	0.67	0.93	0.17	0.45	0.87	0.00	0.18	0.50	0.09	0.32	0.70	0.26	0.60	0.94	0.10	0.32	0.66	0.15	0.41	0.81	0.13	0.38	0.75	0.00	0.17	0.47	0.42	0.81	0.94	0.01	0.01	0.02
A11	0.06	0.25	0.56	0.00	0.13	0.40	0.15	0.47	0.73	0.13	0.40	0.80	0.00	0.14	0.43	0.09	0.32	0.70	0.26	0.60	0.94	0.02	0.18	0.46	0.08	0.30	0.65	0.06	0.25	0.56	0.00	0.13	0.38	0.28	0.63	0.81	0.01	0.01	0.02
A12	0.00	0.10	0.33	0.00	0.02	0.17	0.01	0.27	0.53	0.04	0.20	0.53	0.00	0.05	0.25	0.03	0.20	0.52	0.04	0.27	0.56	0.00	0.14	0.41	0.00	0.11	0.38	0.00	0.13	0.75	0.00	0.06	0.25	0.01	0.25	0.50	0.02	0.04	1.00
A13	0.00	0.10	0.33	0.00	0.05	0.23	0.01	0.27	0.53	0.04	0.20	0.53	0.00	0.05	0.25	0.03	0.20	0.52	0.04	0.27	0.56	0.00	0.14	0.41	0.00	0.15	0.43	0.00	0.13	0.75	0.00	0.06	0.25	0.01	0.25	0.50	0.02	0.04	1.00

.

Step 4. It is followed to calculate the FPIS and the FNIS because the positive triangular fuzzy numbers fall within the range [0, 1], and the FPIS and the FNIS are obtained by Equations (17) and (18). Then, the relative distance is calculated between the algorithms (alternatives) and is computed with Equations (19) and (20) (see

Table 7. Distances between alternatives.

	${\mathit{D}}^{+}$	${\mathit{D}}^{-}$
A1	6.051	2.149
A2	5.596	2.724
A3	5.192	3.296
A4	5.136	3.370
A5	4.770	3.802
A6	5.595	2.748
A7	5.677	2.786
A8	5.861	2.467
A9	5.766	2.654
A10	4.918	3.597
A11	5.391	3.004
A12	6.176	2.279
A13	6.144	2.323

).

Step 5. It is followed to determine the closeness coefficient using Equation (21). $C{C}_i$ The obtained values represent the total score of each algorithm for a production planning problem.

Table 8. Closeness quotient and algorithms ranking.

Alternative	Algorithm	CCi	Rank
A1	HA/Benders decomposition	0.262	13
A2	HA/LP and Fix	0.327	8
A3	HA/LP Relaxation	0.388	4
A4	MA/Tabu Search	0.396	3
A5	MA/Genetic Algorithm	0.444	1
A6	MA/Simulated annealing	0.329	6
A7	MA/Variable Neighborhood Search	0.329	7
A8	MTA Genetic Algorithm + Mathematical Model	0.296	10
A9	MTA Simulated annealing + Mathematical Model	0.315	9
A10	CPLEX (Commercial)	0.422	2
A11	CBC (Non-Commercial)	0.358	5
A12	BONMIN (Non-Commercial—Nonlinear)	0.270	12
A13	LINDO (Commercial—Linear/Nonlinear)	0.274	11

shows the obtained results.

When applying the proposed methodological approach based on the Fuzzy TOPSIS Method for a production planning problem, the GA is the most suitable solution method. This finding is not new because the literature review by Guzman et al. [107] concludes that GAs are the most widely used for this problem. Second in the ranking is CPLEX, which is the most widespread solver [107].

5. Sensitivity Analysis

This section evaluates the effects of the different weightings of the criteria, i.e., we aim to evaluate the answers given by decision makers and how they influence algorithm selection. The aim of the sensitivity analysis is to make minor variations in the weights and to observe the influence of this variation on algorithm choice. The weights for the rating of algorithms range from very low importance (VLI) to very high importance (VHI). Using this rating, we performed an analysis of 10 combinations, where each combination was expressed as an experiment.

The criteria that were evaluated with the highest weight were the type of variables (C3), the quality of solutions (C12) and computation time (C13) (see Table 4). These parameters in a planning problem are dominant when choosing an algorithm. Therefore, we made a minimal variation and looked for criteria with lower scores, such as: problem type (C1), knowledge of algorithms (C10), dataset size (C7), programming knowledge (C8). The details of the experiments are shown in

Table 9. Quantitative results of the sensitivity analysis.

Experiment No.	Changes in Weights of Criteria	A1	A2	A3	A4	A5	A6	A7	A8	A9	A10	A11	A12	A13	Alternatives Ranking
E1	C1 = (0.50, 0.75, 1.00)	0.267	0.337	0.402	0.409	0.457	0.338	0.337	0.302	0.321	0.433	0.366	0.274	0.279	A5 > A10 > A4 > A3 > A11 > A6 > A7 > A2 > A9 > A8 > A13 > A12 > A1
E2	C1 = (0.75, 1.00, 1.00)	0.268	0.341	0.410	0.418	0.465	0.341	0.341	0.304	0.323	0.439	0.370	0.275	0.279	A5 > A10 > A4 > A3 > A11 > A6 > A2 > A7 > A9 > A8 > A13 > A12 > A1
E3	C3 = (0.50, 0.75, 1.00)	0.260	0.322	0.383	0.395	0.438	0.327	0.328	0.295	0.314	0.417	0.355	0.268	0.273	A5 > A10 > A4 > A3 > A11 > A7 > A6 > A2 > A9 > A8 > A13 > A12 > A1
E4	C5 = (0.25, 0.50, 0.75)	0.268	0.335	0.398	0.408	0.455	0.338	0.337	0.304	0.323	0.434	0.367	0.274	0.279	A5 > A10 > A4 > A3 > A11 > A6 > A7 > A2 > A9 > A8 > A13 > A12 > A1
E5	C10 = (0.50, 0.75, 1.00)	0.266	0.336	0.396	0.406	0.455	0.335	0.336	0.301	0.323	0.434	0.366	0.278	0.282	A5 > A10 > A4 > A3 > A11 > A2 > A7 > A6 > A9 > A8 > A13 > A12 > A1
E6	C10 = (0.75, 1.00, 1.00)	0.267	0.339	0.399	0.410	0.461	0.338	0.336	0.302	0.324	0.440	0.370	0.278	0.283	A5 > A10 > A4 > A3 > A11 > A2 > A6 > A7 > A9 > A8 > A13 > A12 > A1
E7	C8 = (0.01, 0.25, 0.50), C12= (0.50, 0.75, 1.00)	0.254	0.317	0.376	0.370	0.422	0.319	0.319	0.287	0.306	0.404	0.346	0.262	0.267	A5 > A10 > A3 > A4 > A11 > A6 > A7 > A2 > A9 > A8 > A13 > A12 > A1
E8	C10 = (0.75, 1.00, 1.00), C11 = (0.50, 0.75, 1.00)	0.268	0.342	0.402	0.411	0.473	0.342	0.339	0.305	0.326	0.448	0.375	0.280	0.285	A5 > A10 > A4 > A3 > A11 > A6 > A2 > A7 > A9 > A8 > A13 > A12 > A1
E9	C7 = (0.50, 0.75, 1.00)	0.262	0.327	0.388	0.396	0.443	0.329	0.329	0.296	0.315	0.422	0.358	0.270	0.275	A5 > A10 > A4 > A3 > A11 > A6 > A7 > A2 > A9 > A8 > A13 > A12 > A1
E10	C7 = (0.75, 1.00, 1.00)	0.268	0.333	0.394	0.402	0.449	0.335	0.335	0.302	0.321	0.428	0.364	0.272	0.277	A5 > A10 > A4 > A3 > A11 > A6 > A7 > A2 > A9 > A8 > A13 > A12 > A1

, where the second column details the changes in the weights of the criteria, and the third column shows the results of the proximity coefficient, while the last column expresses the alternatives ranking.

The sensitivity analysis shows that alternatives A5 (GA), A10 (CPLEX), A4 (Tabu Search) have the best scores and occupy the first three positions. Hence, the variation in the weights in the chosen criteria minimally affects these alternatives; for example, A5 reaches the first position in all the experiments. The main variations occur in the sixth, seventh and eighth positions with alternatives A2, A6 and A7. However, the last ranking positions remain unchanged in the classification. In this context, decision makers can use these variations or make other modifications to weightings to prioritize a criterion and to thus facilitate the evaluation process in decision making.

6. Conclusions

The complexity of real-world problems should be seen not only as an obstacle, but also as a research challenge for effective solutions for large-scale planning problems. Relatively small companies often face very complex problems.

It is usually very difficult for production planners in companies to determine or choose an algorithm. The algorithm selection process normally involves the experimental evaluation of several algorithms with different dataset sizes. However, these sets of experiments require considerable computational resources and long processing times. This adds to the disadvantage of having fewer resources to invest in commercial solvers. In addition, efforts often have to be duplicated when attempting to replicate the algorithms or models available in the literature.

To overcome these drawbacks, the methodological approach based on the fuzzy TOPSIS proposed herein intends to be a support tool to select a solution method for replenishment, production and distribution planning problems. To this end, 13 different criteria were defined and used to select nine different algorithm types (heuristic, metaheuristic, and matheuristic) and four solvers (commercial and non-commercial) that are often employed in planning problems. All these criteria address several important dimensions when solving a planning problem. These dimensions are related to the computational difficulty of the planning problem, programming skills, mathematical skills, algorithmic skills, mathematical modeling software skills, and also to the expected computational performance of the solution methods. These criteria were analyzed based on the linguistic values given by four planning experts from different manufacturing companies. The problem selected to apply the proposed approach was that of production planning. For this problem, the results of the methodology showed that the GA was the best alternative, while Benders’ decomposition was the worst. Given our study results, it can be concluded that it is possible to select a set of suitable candidate algorithms for solving optimization problems with the proposed approach. In this way, not only can one algorithm be selected, but so can other algorithms that provide similar solutions at the same time. The results of this methodology can guide companies to choose whether to use a commercial or non-commercial algorithm or solver. This can help companies to determine whether they should invest in a solver or use mathematical modeling or algorithm programming software and, at the same time, to understand planning staff’s training needs.

There are different approaches for algorithm selection [44,70,75,79]. These approaches are heuristic, metaheuristic and AI, and they offer benefits and disadvantages. However, these techniques can be restrictive for companies because they involve a large number of computational resources and experiments that can be affected by accuracy, the number of tested instances, instance generation, consistency, AI techniques, and training time. The proposed approach requires very few resources, is very useful thanks to its simplicity and is easily replicable. The main limitation of this technique is the appropriate selection of criteria and the balance between them, which is a subjective issue that requires experts in the planning problems field, not to mention the personal bias of experts’ opinions.

Future research could be conducted to experiment the proposed approach with the portfolio of algorithms and solvers defined in [107], where some 50 algorithms are identified, including optimizing, heuristic, metaheuristic and matheuristic algorithms, as well as different types of commercial solvers. Alternatives and criteria could be evaluated with more decision makers. Other MCDM techniques such as ELECTRE, PROMETHEE, intuitionistic fuzzy TOPSIS, or novel methods such as the performance calculation technique of the integrated multiple multi-attribute decision making (PCIM-MADM) [120], which incorporates four techniques (COPRAS, GRA, SAW and VIKOR) into a single final classification index, could be used.