Intelligent Decision Support Systems—An Analysis of Machine Learning and Multicriteria Decision-Making Methods

Abstract

Context: The selection and use of appropriate multi-criteria decision making (MCDM) methods for solving complex problems is one of the challenging issues faced by decision makers in the search for appropriate decisions. To address these challenges, MCDM methods have effectively been used in the areas of ICT, farming, business, and trade, for example. This study explores the integration of machine learning and MCDM methods, which has been used effectively in diverse application areas. Objective: The objective of the research is to critically analyze state-of-the-art research methods used in intelligent decision support systems and to further identify their application areas, the significance of decision support systems, and the methods, approaches, frameworks, or algorithms exploited to solve complex problems. The study provides insights for early-stage researchers to design more intelligent and cost-effective solutions for solving problems in various application domains. Method: To achieve the objective, literature from the years 2015 to early 2020 was searched and considered in the study based on quality assessment criteria. The selected relevant literature was studied to respond to the research questions proposed in this study. To find answers to the research questions, pertinent literature was analyzed to identify the application domains where decision support systems are exploited, the impact and significance of the contributions, and the algorithms, methods, and techniques which are exploited in various domains to solve decision-making problems. Results: Results of the study show that decision support systems are widely used as useful decision-making tools in various application domains. The research has collectively studied machine learning, artificial intelligence, and multi-criteria decision-making models used to provide efficient solutions to complex decision-making problems. In addition, the study delivers detailed insights into the use of AI, ML and MCDM methods to the early-stage researchers to start their research in the right direction and provide them with a clear roadmap of research. Hence, the development of Intelligent Decision Support Systems (IDSS) using machine learning (ML) and multicriteria decision-making (MCDM) can assist researchers to design and develop better decision support systems. These findings can help researchers in designing more robust, efficient, and effective multicriteria-based decision models, frameworks, techniques, and integrated solutions.

1. Introduction

An intelligent system is any system or tool which can support informed decision-making [1,2]. The task of decision making is one of the challenging tasks, which eventually results in the realization or failure of a specific scenario or case. For instance, in a business scenario, a decision support system helps managers to take the right decisions regarding certain complicated problems that arise due to the rapidly changing environment.

Decision-making becomes more challenging in circumstances where it is grounded on multiple criteria. To deal with problems involving multiple criteria, multiple-criteria decision-making (MCDM) or multiple-criteria decision analysis (MCDA) analyze these multiple conflicting criteria to get a more appropriate decision [3,4]. There are a number of MCDM problems and methods classifications but the major difference among them is the way a solution is obtained that might be either implicit or explicit.

Modern decision support systems (DSS) are equipped with the command of multicriteria decision analysis (MCDA) methods to help decision-makers in taking appropriate decisions in complex scenarios. Because of the usefulness of multi-criteria decision-making systems, investigators are struggling to integrate them with machine learning algorithms for providing better alternatives for accurate decision-making [5,6,7]. DSS has widely been used in diverse fields of application, such as agriculture [8], the energy sector [9,10], business [11], and so on. Different fields use different methods of decision-making for problems ranging from simple to complex scenarios [12,13]. Due to the effectiveness of DSS in the decision-making process, researchers are attracted to providing more reliable and robust solutions to solve current as well as future problems. Intelligent decision support systems are an active area of study for researchers where they try to explore new algorithms and propose methods for improving the capabilities of decision support systems.

In light of the importance and effectiveness of DSS in various domains, a systematic literature study was conducted covering the period 2015 to early 2020. The objective of the presented review is to identify the scope, trends, and methods exploited in various domains for the design and development of a decision support system using multiple criteria. Based on the objective, the following research questions are defined and researched throughout the study.

• RQ1: What are the major application areas of IDSS for intelligent decision-making, based on multi-criteria?
• RQ2: What is the significance and impact of multicriteria-based decision support systems (DSS) for solving problems in diverse application areas?
• RQ3: What are the major problems solving approaches, algorithms, and methods used for multicriteria-based decision-making?

The paper is structured as follows. A detailed discussion regarding the search method used for the subject study is shown in Section 2. Section 3 clarifies the procedure for evaluating the papers chosen for the review. The analysis of the papers and their results are discussed in Section 4. Section 5 presents the threats to the validity of the conducted research while Section 6 provides the conclusion.

2. Research Methodology

For the assessment of the uses and significance of the decision support system, a systematic literature review (SLR) was performed. SLR follows a systematic approach to comprehensively identify as well as analyze the relevant studies regarding the selected area of interest [14]. Several guidelines have been proposed by experts to conduct a comprehensive and effective literature study. The strategy followed in the present study is tailored based on the approaches proposed by the researchers [15,16].

First of all, a needs assessment was conducted for the literature review. In this regard, many papers including journal articles, and conference proceedings were studied. Based on this analysis, it was observed that the decision support system (DSS) is used as a problem-solving means in many diverse areas for appropriate decision-making. Based on the needs assessment, research questions were formulated to show the outcome and effectiveness of the proposed study.

Furthermore, for searching and selecting relevant studies to effectively and comprehensively cover the area of interest, research libraries were selected so that the selected literature is reliable and from an authentic source. Six (6) popular libraries were selected such as ACM, Science Direct, IEEE Xplore, Springer, Hindawi, and Taylor and Francis. After selecting the research libraries, queries were formulated to search and filter the relevant studies. For this purpose, different keywords were selected based on the proposed research questions. These keywords reflect and represent the research questions to objectively search the relevant studies.

The idea of looking into the libraries for required relevant content, simply by using keywords, is not an appealing method. So instead of searching the library by individual keywords, the keywords are joined by using the logical operators, such as “AND” and “OR” in between the keywords to formulate correct queries. The support for query format in these libraries was assessed with pilot query attempts and the query was formatted accordingly. The final framed queries were then used on all the selected libraries for the selection of relevant literature, published in the period 2015–2020. For each search query on each library, papers were selected and downloaded based on the relevance of the subject title and contents of the abstract. The retrieved papers were then critically reviewed based on their contents. Finally, a set of the most relevant papers was selected. The refined list of the selected papers was reviewed for finding answers to the proposed set of questions (as discussed above). An overview of the whole research process and methodology is depicted in Figure 1.

Figure 1. Summary overview of the overall research study.

2.1. Formulating the Research Questions

In the beginning, many papers including the literature regarding the decision support system (DSS) based on multi-criteria domains published in recent years were studied to get familiar with its importance, working, concepts, terminologies, and applications. Based on the potential of the decision support system (DSS) in solving problems in different areas, research questions were defined to perform a systematic review of the work done during 2015–2020. The presented study aims to analyze the applications of decision support systems, their impacts on solving problems potentially in various domains, and the major approach followed to know the scope, current status, and future needs to propose a more reliable and enhanced decision support system for solving complex problems. The research questions for the proposed study are discussed in detail in Table 1.

Table 1. The research question for the presented study.

S. No	Research Question	Objective
RQ1	What are the major application areas for IDSS for intelligent decision-making based on multi-criteria?	To identify the scope of the decision support system and multi-criteria used for decision-making in different application areas
RQ2	What is the significance and impact of multi-criteria-based decision support systems (DSS) for solving problems in the diverse area of application	To know the impact or significance of the multi-criteria decision support system on the different application areas
RQ3	What are the major problems solving approaches, algorithms, and methods used for multi-criteria-based decision-making?	To assess the main approaches, algorithms and methods used by decision support systems to solve the problems of the application domain.

Details of each phase of the proposed research study are discussed in the subsequent sections.

2.2. Keywords Definitions in the Light of Proposed Research Questions

After defining the aim of the study and the research questions, keywords were identified based on the preliminary studies and research questions. Only a single keyword term is not sufficient to extract the relevant research studies as different keywords may refer to the same logical terms such as abbreviations, and synonyms. Keeping in view this problem, alike keywords are enlisted for each of the keyword terms. The detail of similar keywords and the purposes for which these are considered are given in Table 2.

Table 2. Detail of the proposed keywords.

S. No	Similar Keyword	Purpose
1	intelligent decision, IDSS, IDSSs, intelligent decision support system, DSS	Keywords to intelligent decision support system in the literature
2	machine learning, algorithm	To search the papers with the machine learning approach, algorithm, or related procedure
3	multi-criteria, multiple criteria, MCDM	To include the decision support systems that are based on multiple criteria

2.3. Formulating the Query

Searching the libraries with one single keyword is not a suitable process. For an effective search process, it is important to formulate a query that should include all the important keyword terms. For this purpose, similar keywords such as alternative names, abbreviations, and synonyms are first connected with the help of the logical “OR” operator. For example, “Decision support system” and “DSS” combined by the “OR” operator will formulate the query as (“Decision support system” OR “DSS”). Similarly, queries as a result of the logical OR operator are then associated with each other with the help of the logical “AND” operator to get more refined search results such as (intelligent decision OR IDSS OR IDSSs OR intelligent decision support system OR DSS) AND (machine learning OR algorithm), for example. By using the OR and AND logical operators, the final query is formulated to get the relevant search result. For confirmation of logical “AND” and “OR” operators working, some experimental search terms are tested to check the support of the search library as well as the format of the query the library can support. As a result, a query may be formulated in two forms due to the relevancy of search results and library format support, as shown in Table 3. Moreover, due to the incomplete provision of certain libraries, the searching operation was performed with individual keyword search terms such as Hindawi.

Table 3. Final queries and their format.

S. No	Query Format	Query
1	Without quotations	(Intelligent decision OR IDSS OR IDSSs OR intelligent decision support system OR DSS) AND (machine learning OR algorithm) AND (multicriteria OR multi-criteria OR multiple criteria OR MCDM)
2	With quotations	(“intelligent decision” OR “IDSS” OR “IDSSs” OR “decision support system” OR “DSS”) AND (“machine learning” OR “algorithm”) AND (“multicriteria” OR “multi-criteria” OR “multiple criteria” OR “MCDM”)

2.4. Searched Libraries

For searching the literature of the domain, popular libraries were selected, such as IEEE Xplore, Science Direct, ACM, Springer, Hindawi, and Taylor and Francis. The following are the libraries used and the associated web links for reference:

• Science Direct library: https://www.sciencedirect.com/ accessed date: 29 December 2019
• ACM library: https://dl.acm.org/ accessed date: 30 December 2019
• Springer library: https://link.springer.com/ accessed date: 30 December 2019
• Taylor and Francis library: https://www.tandfonline.com/ accessed date: 31 December 2019
• Hindawi library: https://www.hindawi.com accessed date: 31 December 2019
• IEEE Xplore library: https://ieeexplore.ieee.org/Xplore/home.jsp accessed date: 29 December 2019

Each of the libraries was searched for the formulated query and search results were filtered to include papers ranging from 2015 to early 2020. Some pilot searches were also performed to get the most relevant search results. Among the two alternative formats depicted in Table 3, most of the libraries confined the search to more specific queries with quotations as compared to an alternative one. The search result of each library in response to the provided query is shown in Table 4.

Table 4. Summary statistics of query execution (2015–2020).

S. No	Libraries	Keywords Query	Total	Included	Period
1	ACM	(“intelligent decision” OR “IDSS” OR “IDSSs” OR “decision support system” OR “DSS”) AND (“machine learning” OR “algorithm”) AND (“multicriteria” OR “multi-criteria” OR “multiple criteria” OR “MCDM”)	45	17	2015–2020
2	IEEE Xplore	(intelligent decision OR IDSS OR IDSSs OR intelligent decision support system OR DSS) AND (machine learning OR algorithm) AND (multicriteria OR multi-criteria OR multiple criteria OR MCDM)	62	20	2015–2020
3	Science Direct	(“intelligent decision” OR “IDSS” OR “IDSSs” OR “intelligent decision support system” OR “DSS”) AND (“machine learning” OR “algorithm”) AND (“multicriteria” OR “multi-criteria” OR “multiple criteria” OR “MCDM”)	458	42	2015–2020
4	Springer	(“intelligent decision” OR “IDSS” OR “IDSSs” OR “intelligent decision support system” OR “DSS”) AND (“machine learning” OR “algorithm”) AND (“multicriteria” OR “multi-criteria” OR “multiple criteria” OR “MCDM”)	555	77	2015–2020
5	Taylor and Francis	(“intelligent decision” OR “IDSS” OR “IDSSs” OR “intelligent decision support system” OR “DSS”) AND (“machine learning” OR “algorithm”) AND (“multicriteria” OR “multi-criteria” OR “multiple criteria” OR “MCDM”)	89	17	2015–2020

2.5. Selection of the Research Articles

Each selected library is searched for the formulated query and after obtaining the search results, these results are then filtered from the year 2015 to 2020. Some of the libraries did not include the year 2020; however, for a general protocol to be followed for each library, results from 2020 are included where they were available. The same procedure and filter by years were applied for all other selected libraries also. For reliability of search results, some experimental attempts were made by altering the format of the query such as with quotation and without quotation, and results were considered when relevant. Query format checking also helped in the identification of library support for the query. Using the above procedure, the queries were applied and search results obtained as shown in detail in Table 4. Furthermore, one of the libraries showed less support for executing complex queries with multiple “AND” and “OR” logical operators, so in this case, a manual procedure was applied by executing each keyword search term individually.

Search results in response to the query by a library were analyzed based on the title and abstract. After the short analysis of each search research, papers that were found to be relevant to the research questions and formulated query were then selected and downloaded in separate folders for future analysis. In the meantime, references of the same papers were downloaded. For reference management, endnote X8 is used in our case. There are several other software reference management tools available; however, due to its compatibility with Microsoft office packages and its richer functionality such as creating groups, labels, pdf attachments, and other customization options, this was found to be better for reference management. The same above procedure regarding papers selection and reference management was followed for all the libraries. As a result, based on the title and abstract of the research material, a total of 202 papers were selected. The details of the selected paper from each library are shown in Table 5.

Table 5. Summary of selected and the final relevant papers.

S. No	Library	Selected Papers	Final Selected Papers
1	ACM	7	4
2	Science Direct	42	18
3	Springer	77	52
4	Hindawi	39	13
5	IEEE Xplore	20	7
6	Taylor and Francis	17	4

2.6. Selection of Finalized Relevant Papers

After selecting the papers based on their title and abstract, these downloaded papers were merged into a single folder to avoid any repetition regarding the papers. In the same way, all the references associated with these papers were analyzed and merged to eliminate duplication for effective future referencing All the papers were then analyzed one by one in detail in the context of the formulated research questions to select the most relevant papers. Those papers that failed to be relevant to any of the research questions were discarded. At the end of the final analysis, 98 papers were selected as the most relevant papers in the context of this study. Details of the papers selected as the most relevant papers from each library are shown in Table 5.

2.7. Effectiveness of the Search Process

At the end of the search process, a total of 98 papers were selected as the most relevant papers. It is important to analyze the effectiveness of the search procedure outcome as the same papers are further analyzed for the main aim of the study. For this purpose, the set of final papers is assessed to know the overall effectiveness of the literature collection procedure. The assessment is based on the following aspects:

Distribution of the final selection research material based on the type of content such as a book, conference proceeding, journal publication, or book section;

Distribution of the papers in association with each library selected;

Yearly distribution of the finalized papers to know the shift of trends of research focus on the area of the research domain.

2.7.1. Papers’ Distribution Based on the Type of Research Material

The quality of the finalized research papers’ is analyzed based on the type of content. In the presented study, research papers include the journal article, book section, book, and conference proceedings. Among a total of 98 selected papers, most papers are journal articles (51), 2nd are the conference proceedings (27), 3rd is the book section (17), and the least number of materials are from books (03), as shown in Figure 2. Most of the portion of the final selected papers are journal articles that show the research focus and effectiveness of the research domain. Furthermore, the percentage distribution for research articles including conference proceedings, journal articles, books, and book sections are 28%, 52%, 03%, and 17%, respectively, as shown in Figure 3.

Figure 2. Frequency distribution of papers type-wise.

Figure 3. Distribution of selected paper by type.

The overall distribution of research papers based on their content type is not enough to represent the trend of the research domain. To find this, the papers are categorized year-wise along with the categorization by their type to know the frequency of research studies along with their type. Figure 4 shows the year-wise distribution of the final selected research papers by type.

Figure 4. Year-wise distribution of selected papers based on their type.

From Figure 4, it is evident that the frequency of research articles has increased in recent years such as 2017, 2018, and 2019 and the major contribution of these papers is through journal publication. These statistics show that the decision support system (DSS) based on multi-criteria is an active area of research on solving decision-making problems in various domains.

2.7.2. Papers’ Distribution Based on the Library

All the selected final relevant papers from the selected libraries were categorized based on the library from which they were derived such as IEEE Xplore, ACM, Science Direct, Hindawi, and Taylor and Francis. Each library and selected papers’ references are shown in Table 6. Based on the statistics, a total of 98 relevant final papers were selected among which 4 papers were obtained from ACM, 7 papers from IEEE Xplore, 52 papers from Springer, 4 papers from Hindawi, 18 from Science Direct and the remaining 4 papers from Taylor and Francis. The least number of papers are from ACM and Taylor and Francis regarding the research domain, while the greatest number of papers are from Springer and Science Direct.

Table 6. Distribution of papers library-wise.

Library	Papers
ACM	[17,18,19,20]
Science Direct	[10,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37]
Hindawi	[9,38,39,40,41,42,43,44,45,46,47,48,49]
IEEE Explore	[4,7,8,50,51,52,53]
Taylor & Francis	[54,55,56,57]
Springer	[1,2,3,5,6,11,12,13,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101]

The frequency distribution of the final selected papers in association with libraries is depicted in Figure 5. It is evident from the statistics that most papers belong to Springer library as compared to the other libraries.

Figure 5. Papers selected from each libraries.

Furthermore, a more detailed representation of the year-wise distribution of papers from the libraries is presented in Figure 6. More sources of papers in the three years (2017, 2018, and 2019) belong to Springer, IEEE Xplore, Hindawi, and Science Direct. All these libraries are popular and it shows that the collected literature belongs to reliable sources.

Figure 6. Year-wise papers for each library.

2.7.3. Overall Effectiveness of the Search Process

Based on the statistics of the final selected papers, it can be seen that most of the papers are journal articles and conference proceedings; this shows that the contents are from reliable sources and their contribution is significant. Papers belong to popular libraries, so the literature is more reliable.

The finalized pool of papers shows an association with the research questions defined for the study. These papers are further analyzed in the context of each research question to derive useful insights. A more generalized representation of the selected papers is provided in Figure 6, which shows the distribution of papers selected from each library, the year of publication, and the reference number. In addition, the overall representation of decision support systems in different application domains is depicted in Figure 7. In addition, Figure 8 shows the overall trend of the year-wise frequency of research, indicating active trends in recent years.

Figure 7. Application domain that is supported by DSS.

Figure 8. Year-wise distribution of papers.

3. Quality Assessment of Papers

After selecting the final set of papers, these papers were further analyzed to provide the quality assessment to derive useful insight which may help the researchers to design or develop effective decision support systems. In the context of the presented study, several quality measures were taken into consideration such as the scope of the study and its effectiveness. In addition, an overview of the method and approaches followed by the decision support system in an application domain is presented.

After the assessment of each paper’s scope and impact, the overall quality assessment for each paper was done by considering the scope as well as the effectiveness. Based on the overall score, the quality of the paper was measured and finalized as “High quality”, “Good quality” or “Acceptable quality”. The subsequent section explains the overall procedure for the quality assessment of papers.

Criteria for Assigning a Score to the Individual Paper

For quality assessment, each paper is assessed here by identifying what type of contribution the selected papers provide. In the current setup, the paper is separated into three categories. These categories include the papers that provide a generalized framework or model or approach, the papers that provide a review or survey and, as the last category, those papers that provide implementation or experimental decision support systems for solving decision-making problems. Based on the type of contribution, papers are assigned some weights to show the importance of their contribution. A weight of 0.5 is assigned for the papers that were a review or survey, 0.7 is assigned for those papers that proposed a model or framework, and the papers that proposed a decision support system that is implemented or has an experimental prototype are assigned weight 1. Furthermore, it is important to mention that the criteria for assigning weight to the type of contribution are based on the nature of the conducted study and the objective of the research to be achieved in the current study. The criterion is not absolute and may vary if the goal of a study is different.

After assigning the weight to the papers’ type of contribution, the active application domain is also considered for the assessment of quality. It has been observed that decision support systems are exploited more frequently in certain application domains such as the medical field or computer domain. Therefore, we have given preference to the papers in association with the support of the application domain so that the researcher will get to know the major areas for applying decision support systems and the studies which have a significant contribution as well. Therefore, the frequency of studies that belong to an area of the domain is also taken into account for the impact of each paper. Details of the quality assessment measures, their necessity, weights, and the mathematical equations are shown in Table 7.

Table 7. Criteria for assessment of the selected papers.

Quality Assessment	Description	Formula
QA1 (Impact)	The impact of a paper is considered as high if a working system is developed for solving decision-making problems as compared to the framework/model proposed or the review or survey conducted.	Review or survey = 0.5 Framework or model = 0.7 Decision support system = 1
QA2 (Scope)	This measures the support for use of a decision support system in the application domain in solving decision-making problems. The application domain is considered to have higher priority if more applications from that domain are supported by decision support systems.	$\frac{Freq(Domai{n}_i)}{Total\_papers}$ Where $Freq(Domai{n}_i)$ is the no of papers belonging to the same area of application $i∊Domai{n}_i where i=1, 2, \dots , 22$
QA3 (Accumulative)	To score the papers by the inclusion of paper impact as well as the importance of the application domain	$Impac{t}_j\ast Scope{ }_j$ $where j=1, 2, \dots , 98$
Quality criteria	Based on the scoring criteria, papers are measured in terms of overall quality	If value (QA3) >= 0.7 (High Quality) If value (QA3) >= 0.5 (Good Quality) If value (QA3) < 0.5 (Acceptable Quality)

Using the procedure as discussed, each paper is assessed based on the quality assessment criteria. First of all, the score is assigned based on the nature of its contribution. In the next step, each application domain is identified for each paper. In the third step, many studies belongings to the same application domain are identified and the portion of their contribution in the context of all selected papers is measured. Furthermore, papers are scored including the impact of the paper as well as the significance of the application domain the paper belongs to. Finally, the papers are assigned different quality levels based on the overall score the paper gets.

4. Results and Discussion

After a detailed assessment of the papers, papers are assigned a quality level based on the nature of the contribution and the scope of the application area. In this section, the results of the quality assessment are presented. Decision support systems assist the decision makers of various domains in solving problems associated with decision-making.

4.1. Overall Assessment

Based on the assessment, a total of 22 application domains are identified where the decision support system actively assists in decision-making problems. These domains are diverse and they show the potential of a decision support system using a multi-criteria approach for solving problems. These problems vary from the computing field to agriculture, the medical domain, personal safety such as police departments, decision-making in business and management activities, employee satisfaction, waste management, and so on. It has been observed that the role of decision support systems in solving the problems associated with a certain domain such as the medical domain, computer field, transportation, energy sector, and agriculture is major. These are the application areas where decision support systems provide an intelligent, efficient, robust solution regarding decision-making in simple to complex problems. The application of the decision support system in various application areas is shown in Figure 9. Based on its effectiveness in the different application domains, a decision support system based on the multi-criteria approach is an active area of research that aims to solve more complex problems by incorporating machine learning techniques and the power of artificial intelligence.

Figure 9. Applications of DSS in various application domains.

Table 8 shows the details of the application domain, the number of studies addressing decision support based on multi-criteria support for solving decision-making problems, and the references of the papers. Based on the statistics, one can see that most of the active application areas for decision support systems are the computing field, business sector, transportation, and medical domain.

Table 8. Details of references associated with each application domain.

S. No	Application Domain	No of Papers	Reference
1	Agriculture	5	[8,29,33,69,85]
2	Business sector	8	[6,11,20,28,31,76,83,95]
3	Computer field	15	[2,3,4,12,25,27,36,56,72,77,78,81,97,98,101]
4	General problem solving	7	[19,21,24,39,59,61,94]
5	Law	1	[66]
6	Industrial domain	6	[17,46,71,74,82,86]
7	Media evaluation	1	[73]
8	Medical field	15	[5,13,38,41,43,44,45,47,48,50,52,92,100,102,103]
9	People safety	2	[22,35]
10	Supply chain	4	[7,55,62,89]
11	Sustainable computing	1	[80]
12	Transportation	10	[23,49,60,63,64,90,93,96,99,104]
13	University or Company	1	[88]
14	Waste management	3	[9,57,87]
15	Water resource management	2	[34,84]
16	Geography	2	[58,65]
17	Energy sector	6	[10,26,37,51,53,75]
18	Disaster management	4	[54,67,79,91]
19	Information about DSS	2	[1,68]
20	Environmental side	1	[32]
21	Hydrogeological sciences	1	[70]
22	Refugees settlement	1	[30]

To know the trends of papers based on the quality of contributions. Figure 10 represents the year-wise trends of the papers based on their quality. From Figure 10, it is evident that most of the contributions in recent years have high quality; there is also a trend towards a higher frequency of contributions. A somewhat similar trend can be seen for the good quality contributions and their frequency is higher as compared to the contribution having acceptable quality. In addition, it is important to note that paper belongs to 2020 are published in the earlier months of 2020 when the research was conducted.

Figure 10. Year-wise distribution of papers based on their quality.

4.2. High-Quality Papers Details

After quality assessment and overall distribution of the contribution of papers based on their quality, contributions of high quality are further analyzed. A detailed representation of high-quality contributions is shown in Figure 11, where a higher number of papers belong to the “Medical domain” and “Computing discipline” i.e., 15 contributions. Furthermore, the statistics show the number of contributions as 7, 10, and 8 for the application domains “General problem solving”, “Transportation”, and “Business sector”, respectively. All these high-quality contributions are depicted in Table 9 along with their application domain, the year of their publication and reference no.; a short description is provided in Table 8.

Figure 11. Distribution of high-quality papers.

Table 9. Details of high-quality contributions along with their description.

Citation	Short Description	Year	Application Domain	Quality
[6]	The study proposed a DSS to measure job satisfaction with the help of the MUSA method and the continuous genetic algorithm.	2017	Business sector	High Quality
[28]	The study proposed systematic learning and an approach regarding the problem of price negotiation.	2017	Business sector	High Quality
[83]	The authors presented a decision support system in general along with some details regarding its use for business purposes that can be used by top management.	2017	Business sector	High Quality
[11]	The study proposed DSS help to reduce the cost of standardization as well as the implementation of a process that minimizes the degree of organizational changes required in standardization and ERP implementation project.	2018	Business sector	High Quality
[31]	The proposed approach utilizes the multiple criteria approach for identifying and assigning a rating to the different companies based on their social data. The study concludes that social media data provide better insights relating to the credibility of companies.	2018	Business sector	High Quality
[20]	The paper discusses how intelligent decision support systems can be equipped with the ability to explain proposed solutions. The proposed system is based on case-based reasoning (CBR) as a resolution methodology.	2018	Business sector	High Quality
[95]	The proposed system helps the decision maker regarding the decision for inventory classification. The presented Improved Decision Support System (IDSS) software considerably decreased the time for inventory classification.	2019	Business sector	High Quality
[2]	The latest intelligent support systems and their types are discussed, such as text analytics, mining, ambient intelligence, and IoT-based decision support systems; biometrics-based systems; recommender, advisory, and expert systems; data mining, data analytics, neural networks, remote sensing and their integration with decision support systems and other IDSSs; GA-based DSS; fuzzy sets DSS; rough sets-based DSS; intelligent agent-assisted DSS; process mining integration to decision support, adaptive DSS; computer vision based DSS; sensory DSS and robotic DSS.	2015	Computer Field	High Quality
[25]	Definition, characteristics, components, etc. and analyses several specific examples and applications of the IoT in the built environment.	2016	Computer Field	High Quality
[72]	Application of multiple criteria decision analysis for designing and deployment of embedded systems.	2016	Computer Field	High Quality
[27]	A systematic literature review regarding the multi-criteria decision-making approaches in e-learning.	2016	Computer Field	High Quality
[77]	The paper presents DSS based on the architectural design of ontology for the optimization of choice of the problem-solving process of commutation circuit partitioning.	2017	Computer Field	High Quality
[3]	The paper proposes a web-based multicriteria GDSS for solving multi-criteria ranking problems. The system helps in generating advice regarding how decision-makers should change their preference to achieve a high degree of consensus.	2017	Computer Field	High Quality
[78]	This study develops a recommender system to enable decision-makers to perform their activities based on users’ profiles.	2017	Computer Field	High Quality
[81]	This paper presents an all-inclusive decision support system, called ClouDSS, which uses different Multi-Criteria Decision-making (MCDM) approaches to optimize decisions in cloud service selection.	2017	Computer Field	High Quality
[56]	The paper presents a decision support system that helps the decision maker in selecting the most suitable database technology. The exploitation of DSS for software selection reduces the cost and time involved in the decision-making.	2018	Computer Field	High Quality
[4]	In this paper, the authors discuss the importance of creating a Mobile DSS based on the multi-criteria approach to support problems of a variable nature. A mobile application is developed which works on smartphones and tablets.	2018	Computer Field	High Quality
[12]	Knowledge-based DSS review is presented which is a part of the decision-making tool. Details are discussed regarding its development, implementation, etc.	2018	Computer Field	High Quality
[97]	The presented approach can provide decision-makers with a suggested candidate method that delivers a robust outcome.	2019	Computer Field	High Quality
[98]	This study presents a decision support system based on automatic rule checking to identify compliance with rules and adopt a case-based reasoning approach to make recommendations using ontology and web semantics. This system increases the design efficiency in both design checking and modification.	2019	Computer Field	High Quality
[36]	Web-based implementation of DSS is presented to solve decision-making problems based on multiple criteria and exploits the TOPSIS and VIkor in the fuzzy and non-fuzzy environment. The system can solve single as well as group decision-making problems.	2019	Computer Field	High Quality
[101]	The authors presented data mining-based IDSS to give an understanding of the theoretical concepts that can be used for designing intelligent decision support systems.	2020	Computer Field	High Quality
[21]	An improved decision support system is provided by integrating the neural network and genetic algorithm for efficient calculation and appropriate decision-making.	2015	General problem solving	High Quality
[61]	The author has discussed how the DSS based on multiple criteria can help the decision makers to identify the effect of preference on the various alternatives.	2015	General problem solving	High Quality
[39]	A hybrid decision-making model (HDMSM) for ranking and selecting appropriate alternatives.	2015	General problem solving	High Quality
[24]	The focus of this research is on multi-criteria spatial decision support systems (MC-SDSS). This study classifies the important issues used in the process of designing and applying MC-SDSS to real systems. To solve each of the issues, the related meta-choices are explained in the paper. Future research directions and insights for future applications are also explained.	2016	General problem solving	High Quality
[19]	The study presents computer-based IDSS to optimize the lapping of plane surfaces via Taguchi’s method of arrays of experiments.	2017	General problem solving	High Quality
[94]	IDSS is proposed for incremental periodic prediction of the decision class to which the action is likely to belong.	2019	General problem solving	High Quality
[13]	The proposed work shows the potential of a Decision Support System for solving the decision-making in a medical facility. Calls from patients are received and, based on the call information, appointment times are allocated to the patients.	2015	Medical Field	High Quality
[38]	This is a special issue for the decision support systems.	2015	Medical Field	High Quality
[50]	A multi-criteria-decision-system (MCDS) is proposed to detect the occurrence of seizures via receiving information from wearable sensors. The objective is achieved with the help of smartphone applications and usage of long-short-term-memory (LSTM) based anomaly detection algorithms and logistic classifiers.	2017	Medical Field	High Quality
[41]	Clinical Decision Support System (CDSS) for recommendations to assist physicians regarding drug prescriptions.	2017	Medical Field	High Quality
[102]	Decision support system based on a three-layer model for improving accuracy prediction in comparison to a two-layer model.	2017	Medical Field	High Quality
[103]	In this study, the association between fractures and inhaled corticosteroids is investigated through the DSS.	2018	Medical Field	High Quality
[5]	A review of DSS was conducted for heart disease diagnosis in the clinical setting. The review considered the types of diseases, data set formation methods, usage of machine learning algorithms, types of comparators as well as the clinical implications for the reported decision support systems.	2018	Medical Field	High Quality
[43]	DSS for the prediction of a number of liver diseases such as alcoholic liver damage, liver cirrhosis, primary hepatoma, and cholelithiasis.	2018	Medical Field	High Quality
[44]	A review is provided regarding the applications and limitations of radiomics to develop better DSS for breast case diagnosis and prognosis.	2018	Medical Field	High Quality
[92]	The study presents a novel idea to implement the decision support system using the MEAN architecture and R statistical language platform. The proposed framework is verified by using the implementation of a user side, multi-participant decision support system supporting the group decision-making process, on the selection of the best as well as commonly agreed senior center for co-living after retirement.	2018	Medical Field	High Quality
[52]	The proposed study provides a medical DSS base on IoT and Soft computing to observe diabetes patients.	2019	Medical Field	High Quality
[45]	The study investigates the concordance of DSS and the recommendation of surgeons regarding back pain.	2019	Medical Field	High Quality
[47]	This study uses data from National Health Insurance Scheme claims, acquired from the hospitals in Ghana, for identifying health insurance fraud and other irregularities. Genetic support vector machines (GSVMs), which are new hybridized data mining and statistical machine learning tools are used. It offers a set of classy methods for the automatic discovery of fraudulent claims in health insurance databases.	2019	Medical Field	High Quality
[100]	The IDSS is embedded in system multi-agents to model the emergency process in operating rooms. A decision support system assists the agents to plan reflective surgery and in allocating human and necessary medical resources.	2019	Medical Field	High Quality
[48]	An overview of literature focused on investigating the currently available mDSS. The mDSS discussed is focused on prostate cancer (PCa).	2019	Medical Field	High Quality
[23]	The authors developed a procedure to identify the interventions for driving safety by taking information history of events happening on the road. A real case study is performed using the infrastructure of the road.	2015	Transportation	High Quality
[60]	In this paper, the authors describe the application of MCDA regarding the identification of routes for the vehicles which collect waste for recycling.	2015	Transportation	High Quality
[63]	MC-based DSS for devising the plan for shipments of hazardous material based on the geographical information.	2015	Transportation	High Quality
[64]	DSS for territory management based on a multi-criteria decision-making scientific and methodological approach for solving automation problems.	2015	Transportation	High Quality
[49]	The decision-making algorithm to reduce traffic jams considers two main challenges, i.e., uncertainties in received data from sensors and variability of the context. AHP algorithm adopted to ITS.	2016	Transportation	High Quality
[104]	In the proposed work the authors have provided a decision support system for municipal waste collection by exploiting the GIS, agent-based models, and smart devices.	2016	Transportation	High Quality
[90]	In this study, a multi-criteria decision support system (MDSS) is developed for carpooling problems. The presented system is a web-based multi-criteria decision support system.	2018	Transportation	High Quality
[93]	The proposed DSS can generate effective, efficient fleet and freight plans under different types of uncertainties and risk-levels in intermodal transportation networks.	2019	Transportation	High Quality
[99]	In this research, a DSS is proposed and discussed which improves the logistics services for the delivery of large and heavy goods using road transportation.	2019	Transportation	High Quality

4.3. Good Quality Papers

Some papers were filtered based on the “Good quality” of the contribution of the paper. The overall distribution of these papers based on the frequency of the application domain to which these papers belong is shown in Figure 12. Based on the statistics, good quality papers show the support of decision support systems in five application domains such as “General problem solving”, “Industrial domain”, “Energy sector”, “Agriculture”, and “Transportation” having a frequency of 7, 6, 6, 5, and 10, respectively. All these “Good quality” contributions are represented in more detail in Table 10 along with their short description, year of publication, application domain, and quality level.

Figure 12. Distribution of good quality papers.

Table 10. Details of good quality papers with their description.

Citation	Short Description	Year	Application Domain	Quality
[69]	In this paper, the authors exploit two multiple-criteria decision-making algorithms in a web-based system. The proposed work classifies and ranks the rural areas of Central Macedonia in Northern Greece.	2016	Agriculture	Good Quality
[29]	The study proposed GDSS for the agriculture supply chain in France. GDSS uses quality function deployment (QFD) and technique for order preference by similarity to the ideal situation (TOPSIS) to select a logistics provider. The proposed system aids the agricultural partners and investors in the selection of third-party logistic providers.	2017	Agriculture	Good Quality
[85]	The proposed DSS supports the preservation of farmland by facilitating zoning decisions. Parcel rating is done to allow the end users to find agricultural lands where defensive zoning should be applied by selecting gradually lower-scoring packages till the desired total area is satisfied.	2018	Agriculture	Good Quality
[33]	Due to the effectiveness and support of decision support systems in solving the problem in various domains, this special issue considers the efforts made for improving decision-making in agriculture using decision-support systems.	2019	Agriculture	Good Quality
[26]	This study discussed the wind farm site selection through multi-criteria-based DSS while using GIS. The proposed system determines the areas which can install wind farms for green energy resources.	2016	Energy sector	Good Quality
[75]	The authors proposed a hybrid integrated decision support system that helps in the prediction of the power supply and demand. The approach focuses on the prediction methodologies for the consumption of natural gas at the country level.	2017	Energy sector	Good Quality
[51]	In this research, the techniques of evaluation of renewable energy sources’ potential are used. These techniques further apply multi-criteria decision-making system (MCDM) methods, which use a probabilistic model, i.e., the Bayesian approach.	2018	Energy sector	Good Quality
[53]	This study explains the established multi-criteria decision-making system (MCDM) and other software which are used for processing spatially heterogeneous data.	2019	Energy sector	Good Quality
[37]	This research explains energy retrofit interventions for the available building stocks and the integration of participative processes in decision-making. In this research, a new multicriteria spatial decision support system (MC-SDSS) is developed. The MC-SDSS is an interactive energy-related plug-in for a geographic information system (GIS).	2019	Energy sector	Good Quality
[59]	A novel framework is presented based on multi-criteria to tackle uncertain situations. Evidence-based theory of Dempster–Shafer (DST) for solving problems in decision-making for uncertain situations. The proposed model can be exploited in real-life situations.	2015	General problem solving	Good Quality
[17]	Presented papers describe decision support systems based on data mining to extract knowledge as well the simulation-based optimization. The proposed systems have the potential to fulfill the industrial requirement regarding the problem or to support important decision-making.	2015	Industrial domain	Good Quality
[71]	In the proposed work, the authors present a decision support system to help in the selection of machine tools in a manufacturing process.	2016	Industrial domain	Good Quality
[74]	Ambient intelligent DSS development to assist in the creation of work procedures to ensure that the production quality is efficient.	2017	Industrial domain	Good Quality
[82]	The paper presents how to develop the decision support system to assist the manufacturer regarding the formulation of optimization problems by taking into account the number of manufacturing levels, the representation of data, and the creation of suitable models for optimal solutions. The presented work will reduce the cost of effort as well as time to achieve sustainable performance.	2017	Industrial domain	Good Quality
[46]	In the proposed study, DSS is used which is based on the ontology and case-based reasoning to assist in manufacturing process selection.	2019	Industrial domain	Good Quality
[96]	In this study, the author develops MCDM methodology to make the appropriate decisions in the selection of the most desired type of vehicle.	2019	Transportation	Good Quality

4.4. Acceptable Quality

Papers that were considered to have the least acceptable quality based on the assessment criteria in the context of this study are shown in Figure 13. Acceptable quality papers belong to the greatest number of application domains; however, these include fewer overall contributions in these domains. A total of 18 application domains are identified for contributions with acceptable quality.

Figure 13. Distribution of acceptable papers.

It should be noted that the number of papers in the application domain is higher in comparison to the “High quality” and “Good quality” context. Many contributions to particular domains are few, ranging from 1 to 8 papers. The quality is also assessed for the significance of their contribution. A more detailed summary of the acceptable quality contribution is presented along with their year of publication, application domain, and short description in Table 11.

Table 11. Details of acceptable quality contributions with their short description.

Citation	Short Description	Year	Application Domain	Quality
[8]	Proposed system helps in selecting the appropriate drone for the specific needs of farmers using drones.	2017	Agriculture	Acceptable
[76]	The research is about the design and development of smart decision support systems, utilizing financial ontology and a model of manager knowledge. This model is made by the use of eye-tracking analysis by experts. To create the model of manager knowledge, many financial experts and economists were consulted to analyze the pre-defined financial reports.	2017	Business sector	Acceptable
[54]	A multi-criteria-based decision support system is proposed for tackling problems regarding emergency service locations. The system is based on GIS and PROMETHEE IV. The applicability of the system is evaluated by the use of a case study regarding earthquake emergency service station planning.	2015	Disaster management	Acceptable
[67]	A framework is proposed for designing a group-based DSS to broadcast events of earthquake occurrences. The framework is based on a multi-criteria decision-making technique.	2016	Disaster management	Acceptable
[79]	Proposes an approach for selecting the important factors which affect risk or crisis. The system calculates the weight of each criterion based on the weights measures for different indicators. The importance of the criterion is directly proportional to the weight it gets.	2017	Disaster management	Acceptable
[91]	In the presented study, a skill taxonomy is established for the S-HELP DSS toolkit to interlink main important emergency tasks with main emergency responders.	2018	Disaster management	Acceptable
[10]	A systematic literature review is conducted to review the application of approaches used for decision-making in the energy sector. The overview provides detail of these approaches regarding 13 different fields belonging to the energy sector such as environmental impact assessment, sustainability assessment, waste management, renewable energy, land management, energy sustainability, green management topics, climate change, water resources management, strategic environmental assessment, energy management areas and construction and environmental management.	2017	Energy sector	Acceptable
[32]	The paper proposes DSS to select the design and the operating parameters for long-distance heat transportation. Optimized selection is based on the multi-criteria approach by considering various measures.	2018	Environmental side	Acceptable
[58]	Potential support is investigated that can be provided by decision support system (DSS). The study proposes that a collaborative Multi-criteria Spatial Decision Support System can assist in the selection of the appropriate policies regarding landscape enhancement.	2015	Geography	Acceptable
[65]	This paper provides methodology regarding the gathering of data by using spatial analysis techniques with integration to the multi-criteria procedure using weighted linear combination.	2016	Geography	Acceptable
[70]	DSS suggests that groundwater managers, policymakers, and stakeholders take the decisions.	2016	Hydrogeological sciences	Acceptable
[86]	The proposed MC DSS framework tries to select or choose technology for a given part to be produced, keeping in view the contractors who will order the parts.	2018	Industrial domain	Acceptable
[1]	Decision-making areas are addressed, and concepts, models, and techniques to support or develop DSS.	2015	Information about DSS	Acceptable
[68]	The authors discussed knowledge-based systems and genetic algorithm application scenarios, and their suitability is explained.	2016	Information about DSS	Acceptable
[66]	Proposed decision-making system that can assist municipal officials to resolve conflicts.	2016	Law	Acceptable
[73]	In this paper, the authors illustrate the potential of a group decision support system to solve advertising media evaluation problems by integrating well-known multi-criteria decision-making techniques such as MCDM ANP VIKOR DEMATEL.	2016	Media evaluation	Acceptable
[22]	A decision support system is exploited for the identification and prediction of patrolling areas to help the police department in making effective patrolling strategies. With the collaboration of the Spanish National Police Corps (SNPC), a case study is tested where the proposed system outperforms when compared to the patrolling area definitions under the usage of SNPC.	2015	People’s safety	Acceptable
[35]	The proposed system is presented for people’s safety regarding ranking the different buildings. The system enables the designers to evaluate the number of design alternatives for the selection of the optimum design.	2019	People’s safety	Acceptable
[30]	The proposed system is used for refugee settlement planning by taking into account the actors involved who participate in providing aid to them. The proposed DSS ranks the alternative sites for refugee settlements.	2018	Refugee settlement	Acceptable
[62]	Proposed multi-criteria based decision support system to select best and active suppliers.	2015	Supply chain	Acceptable
[55]	The proposed system is helpful for a better supply chain of packed products. The system works by combining unique tools for sales forecasting as well as order planning.	2018	Supply chain	Acceptable
[7]	An IDSS model is proposed to assist the pre-procurement process by evaluating the pre-selected preference of given suppliers who make offers. The most suitable offers to consider are selected based on the supplier’s preferences.	2018	Supply chain	Acceptable
[89]	Considers DSS in the field of ports and maritime industry. A survey of the literature is made regarding the design and development of DSS in the domain. The findings may help the researchers to develop effective models and problem-solving procedures. Implications to DSS developers for port and maritime supply chains are discussed.	2018	Supply chain	Acceptable
[80]	The authors briefly summarize computational intelligence (CI) models and their notable features used in intelligent decision support. Analytics of sustainable computing are also computed.	2017	Sustainable computing	Acceptable
[88]	DSS is used to analyze and select the most suitable model of university or IT company cooperation (UIC). Another observer procedure is proposed for checking the consequences of the fuzzy rule before the finalized correction.	2018	University or Company	Acceptable
[9]	An MCDA framework is proposed for the evaluation of nuclear waste management strategies. The framework compares and ranks options using three waste management alternatives.	2017	Waste management	Acceptable
[87]	In the presented study, the site selection problem is supported using the decision support system based on the multi-criteria decision analysis (MCDA) and geographic information system (GIS).	2018	Waste management	Acceptable
[57]	The paper highlights and hopes for supporting decisions for route planning and optimization regarding garbage collection. The objective of the study was to provide practical solutions to the impact of CO2 emissions and to reduce the negative impact on the environment.	2018	Waste management	Acceptable
[84]	Attempt to exploit the GIS system by representing water reservation areas in a map and integrate this information using DSS to recommend water reservation structures and their measures. Water reservation areas are mapped in GIS and, based on the results, water conservation structures are recommended.	2017	Water resource management	Acceptable
[34]	A review of the state of the art is conducted regarding DSS’s application in WWTP. It will help the researchers, designers, and operations to select the most suitable methods for solving problems.	2019	Water resource management	Acceptable

4.5. Frameworks, Methods, and Algorithms Used in Different Domains

After a detailed analysis of the papers’ assessments, in this section, a detailed analysis is made of the approaches, methods, or algorithms used in a specific application domain. It is necessary to analyze what kind of algorithms or methods are used to solve decision-making problems using a decision support system in an application domain. This analysis will help the researchers in knowing the approaches that are implemented based on the nature of the problems, enabling integration of more intelligent algorithms for better support. The overall detailed summary of the framework, algorithms, methods, or techniques used in a specific application domain is provided in Table 12.

Table 12. A detailed summary of approaches, techniques, or algorithms used in the application domains.

S. No	Application Domain	Framework/Algorithms/Methods/Techniques
1	Agriculture	TOPSIS, VIKOR, zoning of land, supervised decision tree, ancillary textual information, biophysical criteria of land.
2	Business sector	MUSA method, continuous genetic algorithm, different fuzzy-based systems, machine learning algorithm, bottom-up process mining models, top-down standardization of information, sentiment analysis, web mining, text classification method, pairwise comparisons, cumulative belief degree approach, case-based reasoning (CBR), ABC classification models among Annual Dollar Usage (ADU), Analytic Hierarchy Process (AHP), Scoring (SCR), Fuzzy C-means Algorithm (FCM), and Analytic Network Process (ANP).
3	Computer field	MCDA process, local cost functions, fuzzy outranking relations, information from users’ profiles, CludDss, MCDM, software selection technology, AHP, MAHP, TOPSIS, Knowledge-based DSS, PROMETHEE II, Building Information Modeling (BIM) models, AHP-TOPSIS (Analytic hierarchy process-Technique for Order Preference by Similarity to an Ideal Solution), TOPSIS, VIKOR, data mining techniques.
4	Disaster management	Geographical information systems (GIS), Preference Ranking Organization Method for Enrichment Evaluation IV (PROMETHEE IV), S-HELP DSS.
5	Energy sector	Geographical Information Systems (GIS), climatological conditions, Adaptive Neuro-Fuzzy Systems (ANFISANFIS), Neural Networks (NN), Fuzzy Cognitive Maps (FCM), econometric models, hybrid MCDM, fuzzy MCDM, Bayesian methods, AHP, MCSDSS (multicriteria spatial decision SS).
6	Environmental side	Static HTS model, terrain elevation profile.
7	University or Company	Fuzzy DSS’s.
8	Waste management	Multicriteria decision analysis (MCDA) framework, GIS, MCDA.
9	Water resource management	GIS system, WWTP.
10	General problem solving	Hybridized MCDM tool, evidence-based theory of Dempster–Shafer (DST), neural network, genetic algorithm, robustness analysis, integration of DSS and Group DSS for considering all the decision criteria simultaneously, Delphi, DEMATEL, ANP, MDS, Multi-criteria spatial decision support system (MC-SDSS), Taguchi’s method, decision table, DRSA-incremental algorithm, decision rules, Supervised DRSA based method (Dominance-based Rough Set Approach).
11	Geography	Web-based data, common knowledge, Landscape Spatial Decision Support System (SDSS), Volunteered Geographic Information (VGIs), Weighted Linear Combination (WLC), Spatial multi-criteria analysis.
12	Industrial domain	Knowledge extraction, data mining, simulation-based optimization, artificial neural network, fuzzy analytic network process, ambient intelligence, optimization heuristics, machine learning, analytical hierarchy process (AHP), P hard problem, metaheuristic (NSGAII algorithm), case-based reasoning (CBR), ontology.
13	Information about DSS	The knowledge-based system, genetic programming.
14	Media evaluation	ANP, VIKOR, DEMATEL.
15	Medical field	Electro-cardiograph (ECG), electro-dermal activity (EDA), long-short-term-memory (LSTM) based anomaly detection algorithms, logistic classifiers, TOPSIS, three layer knowledge base model (disease-symptom-property), multisymptom naïve Bayes algorithm, genetic algorithm (GA), support vector machine (SVM), machine learning, artificial neural network (ANN), myocardia perfusion scintigraphy, CART, EHC-ERT based intelligent-integrated model, ERF for disease prediction, intuitionistic fuzzy number (IFN), Model of Original Multi-objective Decision-making (MODM), Analytical Hierarchy Process (AHP), genetic support vector machines (GSVM), ELECTRE II method, CNP (contract net protocol), Multifactorial Decision Support System (mDSS).
16	People safety	Multi-criteria ranking, 3d modeling, agent-based simulation (ABS).
17	Refugee settlements	Multi-agent system (MAS), intelligent agents, multi-criteria decision-making (MCDM), fuzzy analytical hierarchy process (FAHP), fuzzy axiomatic design approach with risk factor (RFAD).
18	Supply chain	Analytical Hierarchy Process (AHP), the fuzzy technique for order performance by similarity to ideal solutions (TOPSIS), forecasting model parameters, ARIMA, ARIMAX.
19	Sustainable computing	Computational intelligence (CI).
20	Transportation	Concordance analysis, intelligent decision-making models, multiple criteria decision-making, group decision-making, artificial neural networks, metaheuristic, Fuzzy Logic, case-based reasoning, Analytical Hierarchy Process (AHP), GIS, smart devices, agent-based models, genetic algorithm, Fuzzy stochastic mathematical programming-based model, Oracle for the object-rational database.

5. Threats to Validity

The proposed study provides a comprehensive literature survey of efforts made in solving decision-making problems via a decision support system based on multi-criteria approaches. The study mainly focuses on the inclusion of those studies which provide a working decision support system or have the potential to work as a decision-making problem-solving tool. Other studies related to mathematical problem-solving and theoretical assumptions were not part of this study. Furthermore, the outcome and results are based on the final selected papers that are included as the most relevant papers in the context of the research questions of this paper. In the context of the conducted study, the following are the main threats to validity.

5.1. Query Search Support of a Library

For research studies, popular libraries are searched in the context of the formulated query. Search results are based on the formulation of the query and the support of searched libraries. For searching reliability, pilot searches have also been performed in this study. To get the most relevant and better search results, variants of search queries were applied to some of the libraries. However, the relevance of the search results is still based on the manual assessment of the literature as well as the internal searching algorithm of the library that performs the search query processing.

In addition, for proper and systematic analysis of research libraries, random sources are avoided to analyze the association of the research domain with the selected library. In our case, other sources such as random web searching were avoided. Due to the above reasons, there might be some papers that may not be included in the presented study.

5.2. Scope of the Conducted Study

Due to the effectiveness and potential of decision-making based on multi-criteria, the research has a wide scope that provides a problem-solving foundation and frameworks for solving complex problems. The decision support approach based on multiple criteria is a very wide domain to cover; however, the scope of this study was confined to achieving the aims in the context of the research questions. Due to the scope of the presented study, those studies are included in the final selected pool of papers where multi-criteria-based decision support systems are adopted or designed in a way such that they can be adopted.

5.3. Existing Theoretical Approaches/Frameworks

Decision-making based on multiple criteria has a wide potential scope ranging from mathematical theories to the implementation of working systems deployed for solving decision-making problems. In the current study, we have mainly focused on the studies that proposed or implemented decision support systems using the multi-criteria approach in various domains for decision-making. However, studies where mathematical pure theories, assumptions, or suggestions that lack the perspective of how the actual system will work or the methodology of how the system will operate on the multiple attributes criteria were not included.

6. Conclusions

In this paper, a systematic literature review is conducted for the period 2015–2019 and early 2020. For this purpose, such literature is analyzed from six popular research libraries to identify the scope and application of decision support systems in various domains as well as the methods used for solving decision-making problems. Based on the analyses, the study identified that a decision support system based on multi-criteria approaches has major applications in various application domains. The research exploits the power of machine learning and artificial intelligence as an efficient alternative for solving complex problems where problem solution is based on multiple criteria. Furthermore, papers are analyzed using quality assessment criteria taking into account the scope of the domain and the significant of the contribution. The presented study will provide useful insight to the readers and experts in the various domains regarding the current status of research to provide more intelligent and effective models, frameworks, and practical solutions to cope with more complex decision-making problems.