Framework for Selecting Manufacturing Simulation Software in Industry 4.0 Environment

Abstract

Even though the use of simulation software packages is widespread in industrial and manufacturing companies, the criteria and methods proposed in the scientific literature to evaluate them do not adequately help companies in identifying a package able to enhance the efficiency of their production system. Hence, the main objective of this paper is to develop a framework to guide companies in choosing the most suitable manufacturing simulation software package. The evaluation framework developed in this study is based on two different multi-criteria methods: analytic hierarchy process (AHP) integrated with benefits, opportunities, costs, risks (BOCR) analysis and the best-worst method (BWM). The framework was developed on the basis of the suggestions from the literature and from a panel of experts, both from academia and industry, trying to capture all the facets of the software selection problem. For testing purposes, the proposed approach was applied to a mid-sized enterprise located in the south of Italy, which was facing the problem of buying an effective simulation software for Participatory Design. From a practical perspective, the application showed that the framework is effective in identifying the most suitable simulation software package according to the needs of the company. From a theoretical point of view, the multi-criteria methods suggested in the framework have never been applied to the problem of selecting simulation software; their usage in this context could bring some advantages compared to other decision-making tools.

1. Introduction

The industrial sector has always been the scene of fundamental innovations and discoveries over the years, which have contributed to revolutionize and transform not only the sector itself, but also the entire society [1]. This has meant that the world scenario has become increasingly globalized and competitive. Nowadays, especially in the industrial field, the strongest companies are those that continually aim at renewal, because the only way to get ahead, in a dynamic and volatile market, is the search for a winning idea and the use of increasingly innovative technologies [2]. These transformations have been given the name of industrial revolutions, which deeply involved the manufacturing sector. Mankind has already faced three of these revolutions throughout history, and currently we are in the middle of the Fourth Industrial Revolution, also known as Industry 4.0 [3]. Moeuf et al. [4] defined Industry 4.0—which was introduced in 2011 at the Hannover Fair—as a general concept enabling manufacturing with the elements of tactical intelligence using techniques and technologies such as the Internet of Things (IoTs), cloud computing and big data. According to [5], the main pillars of Industry 4.0 are nine technologies, which form the basis of the transition to this new industrial revolution: big data and analytics, autonomous robots, simulation, horizontal and vertical system integration, The Industrial Internet of Things, cybersecurity, cloud, additive manufacturing and augmented reality. Moreover, the main characteristics of Industry 4.0 are [6]: interconnection, interoperability, virtualization, decentralization, real-time operating system, modularity, cyber-physical systems, Internet of Things, Internet of services and smart factory.

Simulation is one of the fundamental pillars of Industry 4.0, thanks above all to its flexibility, being implementable in any field, from manufacturing, to services, to design and even to the healthcare sector. Simulation, according to [7], is “the process of designing a model of a real system and conducting experiments with this model for the purpose either of understanding the behaviour of the system or of evaluating various strategies (within the limits imposed by a criterion or set of criteria) for the operation of the system” [8]. Industry 4.0 and its software simulations can largely benefit from the usage of analytics and simulation models [9].

Technological development, globalization and the growing demand coming from the market in terms of product customization and speed of response have forced the manufacturing world to rethink its processes and its organization, laying the foundations for Industry 4.0 [10]. Before investing in the new technologies, enterprise managers need to know their companies’ readiness for making the right decisions to preserve or enhance their competitiveness. Industry 4.0 readiness assessment and maturity models can support the management in measuring and setting up a plan for the digital transformation of their company, providing guidance and support to align business strategies and operations. The notion of “maturity” defines “the state of being complete, perfect, or ready” [11] and, in a business context, it describes the ability of an organization for continuous improvement in a particular discipline [12]. Maturity models enable a structured approach for analysing the initial state in which weaknesses can be identified, the potential for improvement can be shown and specific steps for improvement can be initiated [13]. The degree of maturity defines a certain development state within a scale interval, determined by a starting point, which is the lowest level, and an ending point, the highest level [14]. A shift to a higher level indicates an improvement. A specific level of maturity includes the respective characteristics of previously defined objects and their required characteristics [15]. Hence, maturity models enable users to identify the need for change and to derive the necessary measures to guide the change process. These models represent a path towards an organized and systematic way of doing business. Two approaches for implementing maturity models exist. With a top-down approach [15] a fixed number of maturity stages (levels) is specified first and further corroborated with characteristics that support the initial assumptions about how maturity evolves. When using a bottom-up approach [16], distinct characteristic or assessment items are determined first and clustered in a second step into maturity levels to induce a more general view of the different steps of maturity evolution. These maturity models are generally operationalized by the submission of a standard questionnaire to the organizations. Answers are mapped in the defined maturity model and standard recommendations based on the assessed maturity level are provided to the organizations [17].

In recent years, the selection of the most suitable simulation software to correctly face industrial problems has become vital for companies. The choice should be made among several alternatives and some companies, especially small and medium enterprises (SMEs), may not have the appropriate tools to support that choice [18]. The simulation software selection is an articulated decision-making process and is often costly and time consuming; a careful selection can take as long as one year [19]. In short, the task of software package selection has become more complex due to: (1) difficulties in assessing the applicability of software packages to the business needs of the organization due to availability of a large number of software packages in the market; (2) possible incompatibilities between the different hardware and software systems; (3) lack of technical knowledge and experience of decision-makers; (4) ongoing improvements in information technology [20]. The simulation software evaluation and selection is a multi-criteria problem, which can be solved by decision analysis techniques [21]. Multi-criteria decision-making (MCDM) methods provide a possibility to evaluate and decide which alternative is the most suitable according to different criteria. This is the main advantage when comparing with other possibilities regarding making a decision [22].

Moving from the considerations above, this study tries to contribute to the literature by helping the management of the enterprises in selecting the most suitable manufacturing simulation software package for their needs, proposing a new methodology. The evaluation framework developed here makes use of two MCDM methods, namely:

• Analytic hierarchy process (AHP) integrated with benefits, opportunities, costs, risks (BOCR) analysis [23];
• Best-worst method (BWM), an approach proposed in recent studies as an alternative to AHP for finding the criteria weights in MCDM problems [24].

From a technical point of view, none of these MCDM approaches has been applied to the problem of selecting simulation software; therefore, their application could lead to some interesting findings.

The paper proceeds as follows. Section 2 reviews the literature relevant to this study, covering in particular Industry 4.0 assessment tools, readiness, maturity models and MCDM methods. Section 3 describes the research methodology followed to derive the proposed framework, while Section 4 details it step by step. In Section 5, the proposed framework is applied on a real case for testing purpose. Finally, the conclusions about the results and contributions made by this study are formalized, highlighting the main limitations and suggesting future research directions.

2. State of the Art

2.1. Industry 4.0 Maturity Models

Industry 4.0 has a strong Information Technology (IT) orientation and is often implemented as part of projects, so it suits well as an appropriate application domain for maturity models that are commonly used as an instrument to conceptualize and measure the maturity of an organization or a process regarding some specific target states.

In line with the aim of this study, this subsection reviews the literature focused on Industry 4.0 maturity models. Schumacher et al. [25] developed a maturity model and a related tool for assessing the Industry 4.0 maturity of manufacturing enterprise. The model has been developed using a multi-methodological approach including a systematic literature review, conceptual modelling and qualitative and quantitative methods for empirical validation. First validations of the model’s structure and content have showed that the model was transparent and easy to use and have proved its applicability in real production environments. Ganzarain & Errasti [26] proposed a process model as a guiding framework for Industry 4.0 collaborative diversification vision, strategy and action building. To be more precise, the authors suggested a stage process model to guide and train companies to identify new opportunities for diversification within Industry 4.0. Results have showed a real need for guided support in developing a company-specific Industry 4.0 vision and specific project planning. Weber et al. [27] have presented the Maturity Model for Data-Driven Manufacturing (M2DDM), consisting of six maturity levels, logically building upon each other. It was meant to serve as a guidance for architects, project managers and others who work and collaborate in smart factory projects. With the maturity model SIMMI 4.0 (System Integration Maturity Model Industry 4.0), Leyh et al. [14] provided a toolset for enterprises to assess their current IT system landscape. The authors presented the application of SIMMI 4.0 in selected enterprises as proof of concept; they used a questionnaire-based approach to apply their model. After observing the strengths and weaknesses of existing models and frameworks, Gökalp et al. [28] proposed a new SPICE-based Maturity Model for Industry 4.0, named Industry 4.0-MM. The aim of the model is to provide a means for assessing a manufacturer’s current Industry 4.0 maturity stage and for identifying concrete measures to help them reach a higher maturity stage in order to maximize the economic benefits of Industry 4.0. Gracel & Łebkowski [29] developed and presented the manufacturing technology ManuTech Maturity Model (MTMM) concept related to Industry 4.0. The role of this model was to help top management answer critical questions such as, “What is the current level of technological advancement of the factory?” or “How should the manufacturing technologies be deployed to ensure the effective execution of a new Industry 4.0 strategy or new business models?”. Guimarães et al. [30] have developed a new method to help an industry select the right Discrete-Event Simulation Software (DESS), which helps improve the productivity of a given process. First, it proposed a methodology that allows companies to self-assess their current internal processes based on a maturity model; finally, it applied the AHP to support simulation software selection by detailing and weighting the components that are important for the specific company to meet its business objectives. Finally, De Carolis et al. [31] proposed a model to set the ground for the investigation of company digital maturity. The authors, subsequently, defined a scoring method for maturity assessment, in order to identify the criticalities in implementing the digital transformation and to drive the improvement of the whole system.

Table 1. Overview of scientific Industry 4.0 maturity models.

Year	Reference	Model	Structure	Evaluation
2016	[25]	Industry 4.0 maturity model	Hierarchical with five levels	Nine dimensions: strategy, leadership, customers, products, operations, culture, people, governance, technology
2016	[26]	Three stages maturity model	Hierarchical with five levels	Three dimensions: envision, enable, enact
2017	[27]	M2DDM	Hierarchical with five levels	Focused on IT systems, which is the only dimension
2017	[14]	SIMMI 4.0	Hierarchical with five levels	Four dimensions: vertical integration, horizontal integration, digital product development, cross-sectional technology
2017	[28]	Industry 4.0 maturity model	Hierarchical with six levels	Five dimensions: asset management, data governance, application management, process transformation, organizational alignment
2018	[29]	MTMM	Hierarchical with five levels	Eight dimensions: core technologies, people and culture, knowledge management, real-time integration, infrastructure, strategic awareness and alignment, process excellence, cybersecurity
2018	[30]	DESS maturity model	Hierarchical with five levels	Six dimensions: knowledge of simulation, process standardization, specialist knowledge, process organization, measurement and evaluation, management programs
2018	[31]	Digital Readiness Assessment Maturity—DREAMY	Hierarchical with five levels	Four dimensions: process, monitoring and controlling, technology, organization

summarizes the main studies about Industry 4.0 maturity models, along with their structure and evaluation.

2.2. Simulation Software Selection

In recent decades, the issue of selecting simulation software packages has attracted the interest of many researchers. Indeed, there are numerous publications on this topic. For this reason, a systematic literature review was set up in order to span in the field of simulation software selection and collect the main contributions about such a topic. The databases considered for the analysis are Scopus (https://www.scopus.com/) and Google Scholar (https://scholar.google.com/). The two databases were analysed and screened using different combinations of keywords, namely “simulation software”, “criteria”, “evaluation”, “selection”, and “framework”. For a paper to be considered as pertinent for the analysis, the combinations of keywords and operators must be found in the title, abstract or keywords of the article. A huge number of documents was found to respond to these requirements, as shown in Figure 1.

These documents were further screened according to different criteria. In particular, papers were retained if:

-

they were written in English;

-

they were in the form of peer-reviewed document. “Grey” literature, i.e., any material, usually not peer-reviewed, produced by institutions and organizations outside classical scientific distribution channels (industrial reports, position papers, or government normative) was excluded from the analysis;

-

all relevant data were available. Lack of relevant information (e.g., authors’ names, title, journal or conference name) involved the exclusion of the paper;

-

they were in line with the goal of the analysis. Studies that only dealt with the application of simulation software to a case study or performance evaluation of production lines through simulation were not considered in the analysis;

-

the full text was available;

-

at least the software selection criteria were discussed in the document.

At the end, as a result of this systematic literature review process, 45 articles emerged as pertinent. Descriptive statistics about the publication year of the articles reviewed are illustrated in Figure 2.

As can be seen from Figure 2, the interest in simulation software selection has intensified in the last ten years with a peak of five and four papers published in 2014 and 2011 respectively.

A summary of the contribution of the literature in the field of evaluation and selection of software packages is given in

Table 2. Overview of contributions to literature of selected documents.

Paper	Evaluation Framework	Maturity Model	Selection Criteria	Evaluation Technique	Practical Application	Comparison Software Characteristics
[32]			x
[33]			x
[34]			x			x
[35]			x
[36]			x
[37]			x	x	x
[38]			x	x	x
[39]			x			x
[40]	x				x
[41]	x		x	x	x
[42]			x
[43]	x		x
[44]	x		x
[45]	x		x
[46]	x		x	x
[47]			x
[48]			x	x	x
[49]				x
[50]			x	x	x
[51]	x		x	x	x
[52]			x
[53]	x		x	x
[54]			x	x	x
[55]			x	x	x
[56]			x	x	x
[57]			x	x
[58]				x
[59]	x		x	x
[19]			x	x
[60]			x	x
[61]			x
[62]			x	x	x
[21]			x	x	x
[63]	x		x	x	x
[64]			x			x
[65]			x	x
[66]			x			x
[24]			x
[67]			x	x
[68]				x	x
[69]			x	x	x
[70]			x	x
[71]			x			x
[30]		x	x	x	x
[18]	x		x	x	x

, in which we have classified it into six categories. To be more precise, columns 2, 4 and 5 highlight the presence of structured frameworks for simulation software selection, evaluation criteria and evaluation technique, respectively. Column 6 indicates whether the proposed selection methodology, evaluation technique, selection criteria have been applied practically. Column 7 indicates whether the authors made a comparison of the technical features of the software packages only. Finally, column 3 indicates if the authors proposed a maturity model.

Only four papers [18,41,51,63] have proposed a structured framework, evaluation criteria, evaluation technique and a practical application. Almost all the selected works provide a set of evaluation criteria, but not all of them also suggest an evaluation technique. Moreover, only few papers make a practical application of this suggested technique or suggest a structured framework with the steps to be followed for selecting simulation software. Finally, six works only made a comparison of technical characteristics of some software packages available on the market. The work of [30] is the first study that tries to relate maturity models for assessing Industry 4.0 readiness to the simulation software selection problem; however, it does not provide a structured framework for simulation software selection.

2.3. MCDM Methods: Evaluation Techniques

This subsection reviews the evaluation techniques (mainly MCDM methods) that could be applied to the simulation software selection problem and summarises their usage in scientific literature (see Figure 3).

The first studies proposing MCDM evaluation techniques date back to the 1990s. AHP, in particular, was the first structured method used and applied in problems of selecting the best option among software alternatives [37]. A dedicated software called SimSelect was then developed by [38], with the precise aim of solving that problem. Looking at Figure 3, it is clear that AHP still remains the most used and studied MCDM technique [59]. However, despite its wide usage, it is plainly acknowledged in the literature that AHP also has some limitations. For example, this technique does not allow costs to be taken into account in the evaluation [23]. Moreover, possible errors and inconsistencies can arise during the association of the weights to a criterion; if applied to the software selection problem, this implies that a company could be doubtful about the final score of a software and thus about the selection of the most appropriate one [50,54]. In addition, the computational procedure of AHP become quite complex if several criteria and alternatives are to be taken into account.

Moving on from these considerations, researchers have started developing different Fuzzy MCDM approaches to refine or complement AHP or have studied how to integrated it with other techniques. More precisely, AHP can be integrated with BOCR analysis for a more comprehensive evaluation which also allows for cost to be taken into account in the assessment [23]. Recently, the BWM is gaining over AHP in popularity among researchers for simplifying the process of carrying out pairwise comparison and reducing the occurrence of inconsistencies during comparisons [72]. Instead, analytic network process (ANP), fuzzy analytic hierarchy process (FAHP), Technique for Order Preference by Similarity to Ideal Solution (TOPSIS), preference selection index (PSI), elimination and choice translating reality (ELECTRE) [67,73,74,75] and other approaches are all examples of methods that have been created to this end. Most of them have been applied to the problem under examination or to similar problems; for example, TOPSIS has been used for evaluating the performance of IoT in organizations, while AHP, FAHP, TOPSIS and PSI have all been applied to the problem of evaluating and selecting simulation software packages [70,75].

In works expressively dedicated to the software selection problem, specific indexes have sometimes been introduced with the purpose of analysing and classifying the software behaviour. Examples of these studies have been by Rincon et al. [51] and Sahay & Gupta [48]. The former authors defined the weight global quality rate (WGQR) strategy for analysing the behaviour of the software within the hierarchy, because it quantifies the influence of the weight assigned to each category. The WGQR is estimated as a function of the quality rate and of the weights of the categories. Sahay & Gupta [48] have instead defined the software Solution Merit Index (SMI) which is indicative of software utility and its merit to the organization.

To sum up, on the basis of the considerations above, in the proposed framework the MCDM approaches suggested for implementation are AHP coupled with BOCR and BWM. AHP is indeed a well-structured and known MCDM methodology and the most applied in literature [18]; coupling it with a BOCR analysis allows the decision-maker to take into account the positive and negative aspects of the problem separately, which is expected to improve the effectiveness of the procedure [23]. Next to AHP, BWM has been selected as it represents an evolution of the first method, is faster from a computational point of view and reduces the risk of inconsistencies during comparisons [72].

3. Research Methodology

The framework presented in this paper was developed to support the evaluation and selection of simulation software packages. Given this aim, the framework design was supported by an appropriate multi-disciplinary panel of experts, composed of 14 members, namely:

• two academics from the University of Parma and three academics from the University of Naples, chosen among researchers whose interests are in the field of simulation and process modelling;
• three people from as many software houses manufacturing different kinds of software packages, including simulation software packages;
• six people from two small-size companies. For each company, representatives were the company’s owner, the plant manager and the purchasing manager.

The panel work was led by the academics as follows. On the basis of their skills in simulation and process modelling, as well as following the suggestions retrieved from the literature (as illustrated in Section 2), academics set up a preliminary framework for the selection of simulation software packages. This version of the framework consisted of eight steps, with a related description, and was explained to the remaining panel members during a roundtable discussion. Each member was then asked to express their level of agreement against each step, on a 5-point scale (1 = totally disagree; 5 = totally agree); each expert could also suggest to add/remove a step, as well as to combine steps together. Hence, academics collected and summarized the outcomes of the roundtable discussion, and amended the framework accordingly. The updated version of the framework was submitted to the panel members for a second roundtable discussion, in which experts were asked to refine their preliminary considerations, as well as to express again their agreement with the steps included in the second version of the framework. At the end of the second roundtable discussion, the panel members expressed a general agreement on the framework proposed, which was, therefore, considered as validated. The framework in its final configuration is presented in Section 4.

4. Evaluation Framework for Simulation Software Selection

This section describes the evaluation framework for selecting simulation software. The need for a structured and standardized approach to simulation software selection is evident: evaluating simulation software packages and selecting the best one for an organization is a time consuming task unless an efficient methodology is used. This study provides a structured approach to simulation software selection. The proposed selection framework is a nine-step process and its structure is shown in Figure 4.

As can be seen from Figure 4, the proposed framework has been designed to provide an agile but structured approach to the selection of simulation software packages. Indeed, the approach consists of a sequence of stages that could be customised if needed, depending on the specific application and/or industrial problem the simulation software should face.

Stage 1 is the self-assessment of the digital readiness of the company using a maturity model. The maturity model proposed in this paper aims to contribute to the generation of knowledge about how to situate an industry in relation to the degree of maturity of its production processes, technology, governance and the culture in order to be able to extract the best outcomes from the digitalization and the adoption of a simulation software. Stage 2 is the identification of the core problem and the need of a company for purchasing a simulation software. Stage 3 represents the definition of the general objective and requirements of the simulation software. Stage 4 is an initial survey of the software packages available on the market to make a preliminary screening of the candidates and to get a short list of alternatives. Stage 5 is the effective evaluation and selection of the software through:

-

the new integrated BOCR-AHP methodology. It consists of six ordered sub-steps and should help select the best software and to provide a ranking of alternatives;

-

the BWM approach. It includes eight ordered steps to select the best simulation software from a list of candidates.

Stage 6 is the output of the selection methodology: a ranking of the candidate software packages. Stages 7 to 9 are practical steps. To be more precise, stage 7 involves purchasing the top-ranked software package. Then, in stage 8, a solution of the core problem is implemented using the selected simulation software. Finally, stage 9 involves the maintenance and evolution of the implemented solution.

Each stage will be described in detail in the following subsections.

4.1. Self-Assessment of the Digital Readiness of the Company Using Maturity Model

Before selecting a simulation software, through a maturity model, it is appropriate for companies to make a self-assessment of their manufacturing processes, for evaluating their readiness to embrace new Industry 4.0 key enabling technologies. This is a crucial point because it allows to understand if the company is ready to really take advantage of the implementation of a simulation software and of its usage in the Industry 4.0 perspective. If the company is not ready, the self-assessment could allow to identify the needs for improvement.

The starting point in the development of the model was to identify the possible structure to be used when representing a maturity level. Two type of structures are available in the literature, hierarchical and process-oriented [76]. In this study, a hierarchical structure was adopted. According to this structure, the degree of readiness of a process for the usage/implementation of computer simulation can be evaluated by looking at some specific dimensions. Weckenmann & Akkasoglu [77] note that hierarchical models usually have four to six levels. According to [34,78], the maturity model proposed in this work has five levels—basic, repetitive, defined, integrated and optimal—and is grounded in four dimensions—process, technology, organization and people. For the present study, the levels and dimensions of the maturity model were defined as per

Table 3. Levels and dimensions of maturity model.

		LEVEL
		Basic	Repetitive	Defined	Integrated	Optimal
DIMENSION	Process	No standardization Documents only maintained locally Minimal information sharing Limited product feedback	Static reports of operational activity Isolated attempts of standardization Information is starting to be shared	Processes are formally described Implementation of standards	Processes controlled and managed statistically and quantitatively Real-time analytical data processing Predictive analytics	Integration into corporate processes Digital oriented processes Processes improved by incremental innovations Cognitive analytics Quantitative goals
	Technology	No historical data Sensors are not connected	Connected devices Data localized	Software tunable assets Data in real-time	Self-optimization Interaction with ecosystem	Converged technology Real-time infrastructure Machine learning
	Organization	Ad hoc Decision-making No prediction capabilities Use trial and error or experience for troubleshooting Minimal strategic planning	Near-term focused Measurements are made in the form of manual notes Some evaluation procedures have started to be defined	Process-driven Longer-term focused Quality inspections Quality maintenance procedures	Policy-driven Long-term focused Smart decision-making Limited enterprise-wide integration	Value-oriented Strategy iterates rapidly in response to competitive opportunities and threats Implementation of programs such as QPM, OPP, OPM
	People	Ad hoc people management	Policies developed for capability improvement	Standardized people management across organization	Quantitative goals for people management in place	Continuous focus on improving individual competence and workforce motivation

.

In our proposed approach, the evaluation of maturity level is suggested to be performed by applying a questionnaire survey. A total of 33 questions were formulated for this purpose and grouped according to the four dimensions; for each dimension, questions were classified on the basis of the different aspect investigated, according to the scheme proposed in Appendix A. Most of the questions are to be answered on a scale of perception with five options, from “1 = low” to “5 = high”; the remaining questions are Yes/No questions.

The maturity level for each dimension is then assessed by assigning numerical values to each response, according to the following rules:

[1 → 0.00; 2 → 2.50; 3 → 5.00; 4 → 7.50; 5 → 10.00] for questions with the response scale from 1 to 5;

[1 → 0.00; 2 → 10.00] for yes/no questions.

The score of each dimension, henceforth referred to as the “individual dimension maturity level” (IDML), is obtained as follows:

$$IDML =\left({\sum }_{i=1}^n\raisebox{1ex}{$Q_i$}\!\left/ \!\raisebox{-1ex}{$n$}\right.\right)*1$$

(1)

where:

$Q_i$ is the numerical value associated with the response to question $i$;

$n$ is the number of questions in each dimension.

According to the numerical values set above, IDLM can vary from 0 to 100. The range of possible scores was divided into five equal bands corresponding to the five levels of maturity, as shown in

Table 4. Score band for maturity levels.

Maturity Level	Score Band
Basic	0 ≤ IDML < 20
Repetitive	20 ≤ IDML < 40
Defined	40 ≤ IDML < 60
Integrated	60 ≤ IDML < 80
Optimal	80 ≤ IDML ≤ 100

. The maturity level for each dimension is thus determined using these bands. The “final maturity level evaluated by the model” (FMML) is obtained as the mean of the IDMLs, as all the dimensions have the same weight.

$$\mathrm{FMML} ={\displaystyle \sum }_{i=1}^4\frac{IDM{L}_i}{4}$$

(2)

The results of this step allow the company to obtain a quick overview of the global maturity level of their operational processes. Taking into account the correspondence between the maturity classification levels and the FMML score, it is easy to deduce that a company with FMML < 40 is not in the position of getting success from the permanent usage of a simulation software.

4.2. Identification of the Core Problem

One of the most important stages in selecting simulation software packages (as well as in many other decision-making problems) is to clearly state the problem, or class of problems, that the enterprise would like to address by means of the software. In addition, defining the problem also includes stating what the simulation tool is expected to do.

Once the need for purchasing a simulation software has been established, several factors have to be considered. These factors include, among others, the intended simulation purpose, the existing constraints within the company, the main types of problems to be simulated and some key information regarding the modellers and potential users. Other issues to be taken into account are organizational constraints. Financial constraints might include the hardware available for use by the simulation software, and the budget available for software purchasing, installation and maintenance costs, purchasing additional hardware, training of personnel, etc. Another constraint is the time available for software evaluation, selection and implementation [40]. Determining the models that are likely to be developed can further help reduce the list of possible software candidates for evaluation. Three main types of models can be distinguished: discrete-event, continuous or those that combine discrete and continuous elements. The issues relating to the persons involved in software selection, modelling and the use of future models also need to be addressed. Preferably, the same employees should be involved in the process of software selection and modelling. If possible, a team approach should be used.

4.3. Definition of the General Goals and Requirements

Once the core problem has been identified, in stage 3 of the software selection process the group responsible for executing the evaluation and selection of the software has to define the general goals of the choice based on the overall organizational requirements. The objectives are:

-

areas of the application and usage of the software;

-

particular aims of the organizational unit that will use it;

-

identification of the functional and non-functional requirements of the software [51].

This phase must be carried out in the most accurate, complete and detailed way possible as it will be crucial for selecting the most appropriate software package for the specific application the company needs. Indeed, the list of desired functionalities and features of the software indicates which the most important evaluation criteria are for the company and will determine which criteria will have a greater weight in the application of the selection method. Therefore, these functional requirements will be the discriminant in the evaluation of the candidate software packages. For example, for a specific application a company could consider as more important software features such as graphics, 3D animation and GUI. The same company, for a different application, could consider as more important different features such as statistical reports and integration with other technical systems.

4.4. Preliminary Screening of the Simulation Software Packages Available on the Market

This phase includes preliminary investigation on the market for availability of software packages that might be suitable candidates, including high-level investigation of major functionalities and features supported by the software package. Helpful sources for preliminary investigation could be web-based resources including the vendor’s website, professional association catalogues and other third party reports [59]. The purpose of this initial survey is to shorten the list of software products that can be considered for evaluation and subsequent selection [45]. For example, if the systems to be simulated comprise both discrete and continuous elements, then all packages that cannot support both functionalities could be eliminated. At this stage, several other sources of information have to be consulted. Vendors of software products that seem to be candidates for software evaluation should be contacted and asked for assistance, as well as for providing as much information as possible, in addition to software demonstrations and written material. Other sources of literature related to software being considered ought to be examined and other software users contacted, if possible. The expected outcome of this step is a short list of candidate simulation software packages for the next detailed evaluation.

4.5. Evaluation and Selection of the Best Simulation Software

This subsection describes the effective evaluation and selection methodologies, which represent stage 5 of our proposed framework. As already mentioned, the MCDM methods suggested for application are AHP integrated with BOCR analysis and BWM.

4.5.1. The Integrated BOCR-AHP Methodology

Based on the integration of BOCR analysis and AHP, this methodology is the core of the developed evaluation framework and includes six ordered steps to select the best software from a short list of candidates and rank them.

The AHP, one of the most popular MCDM approaches, introduced by [79], is an effective tool for dealing with complex decision-making, and may aid the decision-maker to set priorities and make the best decision. By reducing complex decisions to a series of pairwise comparisons, and then synthesizing the results, the AHP helps capture both subjective and objective aspects of a decision. The AHP functional steps are described in [80]. The AHP breaks down the problem as a hierarchy. In the hierarchical structure on which the elements are placed, the objective (goal) to be achieved is first established, and then split by defining the criteria needed to evaluate the alternatives. The AHP generates a weight for each evaluation criterion according to the decision-maker pairwise comparisons of the criteria. For a fixed criterion, the AHP assigns a score to each option according to the decision-maker pairwise comparisons of the options based on that criterion. Finally, the AHP combines the criteria weights and the options’ scores, thus determining a global score for each option, and consequent ranking. The global score for a given option is a weighted sum of the scores it obtained with respect to all criteria.

BOCR analysis is a strategic planning technique used to help a decision-maker identify benefits, costs, opportunities and risks related to business venture or project and identify the internal and external factors that are favourable or unfavourable in achieving those objectives [81]. The four parameters which the technique examines are:

-

Benefits: characteristics of the business/project that give it an advantage over others;

-

Opportunities: elements in the environment that the business/project could exploit to its advantage;

-

Costs: characteristics of the business that place the business/project at a disadvantage compared to others;

-

Risks: elements in the environment that could trouble the business/project.

Figure 5 briefly summarizes the six steps to be followed to perform an integrated BOCR-AHP analysis for simulation software evaluation and selection, specifying which elements represent the inputs and outputs of each step.

Step 1: Determining a set of decision criteria. The first step consists of defining the set of criteria to be adopted by a company for identifying the best simulation software for a specific application. Based on the literature review and by means of two brainstorming sessions, a three-level hierarchy with 9 criteria and 36 sub-criteria was developed, according to the scheme proposed in Appendix B. Appendix B also provides a description of each criterion/sub-criterion and suggests a possible scale to evaluate their relative importance.

Step 2: Developing of BOCR structure. To embody the BOCR analysis in the simulation software selection process, criteria should be therefore organized into four groups, covering benefits, opportunities, costs and risks. In the BOCR structure, the top level includes the merit nodes (i.e., the benefits, opportunities, costs and risks perspectives), whereas the second level consists of four sub-hierarchies, one for each perspective, containing the nodes known as control criteria. Figure 6 represents the developed BOCR structure with the classification of the criteria defined in the previous step.

Step 3: Computing the weight of the criteria. This is a typical step of the AHP, which has been slightly modified here to take into account the BOCR analysis. Not all the criteria will have the same importance. One of the steps in the AHP process is to derive the relative priorities (weights) for the criteria. In order to compute the weights for the different criteria, AHP starts by creating a pairwise comparison matrix A of the criteria (or sub-criteria). The matrix A is a n*n real matrix, where n is the number of evaluation criteria considered. Each entry $a_{ij}$ of the matrix A represents the importance of the ith criterion compared to the jth criterion. If $a_{ij}>1$, then the ith criterion is more important than the jth criterion, while if $a_{ij}<1$, then the ith criterion is less important than the jth criterion. If two criteria have the same importance, then the entry $a_{ij}$ scores 1.

The entries $a_{ij}$ and $a_{ji}$. satisfy the following constraint:

$$a_{ij}\times a_{ji}=1$$

(3)

Obviously, $a_{ii}=1$ for all i. The relative importance between two criteria is measured according to a numerical scale from 1 to 9. For further details, the reader is referred to [80].

In a BOCR-AHP methodology, pairwise comparisons should be carried out in each sub-hierarchy of each merit. The number of pairwise comparisons depends on the number of factors to be compared and scores:

$$\frac{n\left(n-1\right)}{2}$$

(4)

Once judgments have been entered, it is necessary to check their consistency. The matrix A is symmetrical, reciprocal and consistent if:

-

The reciprocity relationship is valid:$a_{ij}=\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$a_{ji}$}\right.$ for each value of i and j;

-

The transitivity relationship is valid:$a_{ij}=a_{ik} \times a_{jk}$ for each value of i, j and k.

If such conditions occur the matrix is perfectly consistent.

We now need to calculate the overall priorities or weights of the criteria. There are two types of methods available for this purpose: the exact and the approximate. Although we will not show the exact method in detail here, the general idea is very simple. Raise the comparison matrix to powers (e.g., raise the matrix to the power of two, raise the resulting matrix to the power of two again, and so forth) a few times until all the columns become identical. This is called the limit matrix. At this point, any of the matrix columns constitutes the desired set of priorities. This calculation can be done in a spreadsheet, but it is currently done very easily using AHP-based software packages. We will rather use one of the approximate methods due to its simplicity. The most commonly used method requires the normalization of the matrix at an early stage, summing the columns of the matrix and dividing all the elements of the matrix by the sum of the column in which they are located. Then approximately calculate the eigenvector of each row (the priority vector), simply calculating the average value of each row (adding the values of the rows and dividing them by the number of columns). The principal eigenvalue is calculated by multiplying each element of the priority vector by the number resulting from the sum of the matrix columns. The resulting values are then added up together. The consistency index is always used to check the consistency. From an analytic point of view, once the matrix A is built, it is possible to derive from A the normalized pairwise comparison matrix $A_{norm}$ by making equal to 1 the sum of the entries on each column, i.e., each entry ${\overline{a}}_{ij}$ of the matrix $A_{norm}$ is computed as

$${\overline{a}}_{ij} = \frac{a_{ij}}{{{\displaystyle \sum }}_{i=1}^n{a}_{ij}}$$

(5)

Finally, the criteria weight vector w is built by averaging the entries on each row of $A_{norm}$, i.e.,

$$w_i = \frac{{{\displaystyle \sum }}_{j=1}^n{\overline{a}}_{ij}}{n}$$

(6)

The components of the criteria weight vector meet the following constraint:

$${\sum }_{i=1}^n{w}_i = 1$$

(7)

Step 4: Computing the score of the alternatives. This step consists in deriving the relative priorities (preferences) of the alternatives with respect to each criterion. For this purpose, a pairwise comparison of all the alternatives against each criterion included in the decision-making model should be carried out. The matrix of option scores is a m × n real matrix S, where each entry $s_{ij}$ represents the score of the ith option with respect to the jth criterion. To derive these scores, a pairwise comparison matrix $B\left(j\right)$ is first built for each of the n criteria, j = 1,…,n. $B\left(j\right)$ is a m × m real matrix, where m is the number of options (alternatives) evaluated. Each entry $b_{ih}\left(j\right)$ of the matrix $B\left(j\right)$ represents the evaluation of the ith option compared to the hth option with respect to the jth criterion. If $b_{ih}\left(j\right)>1$, then the ith option is better than the hth option, while if $b_{ih}\left(j\right)<1$, then the ith option is worse than the hth option. If two options are rated as equivalent with respect to the jth criterion, then the entry $b_{ih}\left(j\right)$ scores 1. The entries $b_{ih}\left(j\right)$ and $b_{hi}\left(j\right)$ meet the following constraint:

$$b_{ih}\left(j\right)\times b_{hi}\left(j\right)=1$$

(8)

and $b_{ii}\left(j\right)=1$ for all i.

The same two-step procedure for the pairwise comparison matrix $A$ should thus be applied to each matrix $B\left(j\right)$. To be more precise, each entry in the matrix is divided by the sum of the entries in the same column, and then it averages the entries on each row, thus obtaining the score vectors $s\left(j\right)=1,\dots ,m$. The vector $s\left(j\right)$ contains the scores of the evaluated options with respect to the jth criterion. Finally, the score matrix $S$ is obtained as

$$S=\left[s\left(1\right)\dots s\left(n\right)\right]$$

(9)

i.e., the jth column of $S$ corresponds to $s\left(j\right)$.

Once the weight vector w and the score matrix $S$ have been computed respectively with the (6) and (9), the AHP obtains a vector v of global scores by multiplying S and w. The ith entry $v_i$ of v represents the global score assigned by the AHP to the ith option.

The pairwise comparisons of the alternatives shall be performed for each criterion in each hierarchy of each perspective—Benefits, Opportunities, Costs and Risks—and we will obtain a vector v for each merit B, O, C, R.

At the end of this phase we will obtain the scores $B_i$, $O_i$, $C_i$ and $R_i$ which represent the synthetic results of the alternative i under merit B, O, C and R. It is expected that the best alternative gets the highest priority for Benefits and Opportunities merit and that the worst alternative gets the highest priority for Risk and Cost perspectives.

Step 5: Ranking of the alternatives. Once the alternatives are prioritized with respect to B, O, C, and R in their respective hierarchies (including criteria and eventual sub-criteria), finally their scores in each merit are combined using some formulae to compute a final global score of each alternative. Five formulas (multiplicative, additive, probabilistic additive, subtractive and multiplicative priority powers) were suggested by [81] as approaches to calculate the overall priorities of the alternatives by gathering the scores ($B_i$, $O_i$, $C_i$, $R_i$) of each alternative under each merit with the corresponding priorities (b, o, c, r). For further details, the reader is referred to [81].

Step 6: Sensitivity analysis. The overall priorities (global score of the alternatives) will be heavily influenced by the weights given to the respective criteria. It is useful to perform a “what-if” analysis to see how the final results would have changed if the weights of the criteria had been different. This process is called sensitivity analysis and constitutes the sixth step in our proposed methodology. Sensitivity analysis allows to evaluate the robustness of the original ranking and what main drivers influenced it to the highest extent. This is an important part of the process and, in general, no final decision should be made without performing a sensitivity analysis. When embodying the BOCR analysis in the AHP approach, changes in the criteria weights can be motivated, for instance, by a variation of the relative importance of the four perspectives b, o, c, r. Moreover, a different formula can be used to compute the overall performance of each candidate software package [23]. The implementation of the BOCR analysis also allows to establish priorities among the factors and to carry out sensitivity analyses which make it possible to evaluate the robustness of the results or the changes in the ranking of the alternatives generated by a variation in the relative importance of the positive/negative aspects of the problem.

4.5.2. The BWM Approach

The BWM approach, proposed by [24], includes eight ordered steps to select the best alternative (simulation software package in the case under examination) from a short list of possible candidates. As the BWM is a MCDM evaluation method, some steps are the same as those of the AHP methodology, which is currently the most widely used MCDM approach.

Figure 7 briefly summarizes the steps to be followed to implement the BWM approach for simulation software evaluation and selection, highlighting the inputs and outputs of each step.

Step 1: Determining a set of decision criteria. For this description, see “Step 1” of methodology described above, with particular attention to Appendix B.

Step 2: Determining the best and the worst criteria. The description of this step is taken from [24]. The decision-maker identifies the best and the worst criteria in general. This should be done for the first-level criteria (general characteristics, visual aspects, logical aspects, etc.) and for each group of second-level criteria (sub-criteria). If more than one criterion is considered to be the best or the worst, one can choose arbitrarily among them.

Step 3: Determining the preference of the best criterion against all the others. This step involves identifying the preference of the best criterion (B) over all the other criteria using a 9-point scale (numbers between 1 and 9, where 1 = B is equally important to j; 9 = B is extremely more important than j) [80]. The resulting Best-to-Others vector would be:

$$A_B=\left(a_{B1},a_{B2},\dots ,a_{Bn}\right)$$

(10)

where criterion j is a generic criterion and $a_{Bj}$ indicates the preference of the best criterion B over criterion j. It is clear that $a_{BB}=1$.

Step 4: Determining the preference of all the criteria against the worst. This step determines the preference of all the criteria over the worst criterion (W) using a 9-point scale (numbers between 1 and 9, where 1 = j is equally important to W; 9 = j is extremely more important than W) [80]. The resulting Others-to-Worst vector would be:

$$A_W={\left(a_{1W},a_{2W},\dots ,a_{nW}\right)}^T$$

(11)

where $a_{jW}$ indicates the preference of the criterion j over the worst criterion. It is clear that $a_{WW}=1$.

Step 5: Finding the optimal weights for the criteria. The optimal weight for the criteria is the one where for each pair of $w_B/w_j$ and $w_j/w_W$, we have $w_B/w_j=a_{Bj}$ and $w_j/w_W=a_{jW}$ where $w_B$ is the weight of the best criterion, $w_j$ the weight of a generic criterion j, and $w_W$ the weight of the worst criterion. To satisfy these conditions for all j, we should find a solution where the maximum absolute differences $\left|\frac{w_B}{w_j}-a_{Bj}\right|$ and $\left|\frac{w_j}{w_W}-a_{jW}\right|$ for all j are minimized. Considering the non-negativity and sum condition for the weights, the following problem results in

$$\min maxj\left\{\left|\frac{w_B}{w_j}-a_{Bj}\right|,\left|\frac{w_j}{w_W}-a_{jW}\right|\right\}$$

(12)

subject to

$$\begin{array}{c}{\sum }_j{w}_j=1\\ w_j \ge 0 for all j\end{array}$$

After solving problem (12), with the procedure following by [24] the optimal weights $\left(w_1^{*},w_2^{*},\dots ,w_n^{*}\right)$ and ${\epsilon }^{*}$ are obtained. For MCDM problems with more than one level, we should identify the weights for different levels following the BWM steps, after which we can multiply the weights of different levels to determine the global weights.

Step 6: Comparing software alternatives against each criterion using the aforementioned procedure. According to [24], “In some decision-making problems, we have an alternative value i with respect to criterion j $\left(p_{ij}\right)$. For instance, think of a car selection problem where fuel consumption is a criterion, and we have information about the fuel consumption of all the alternatives. In some decision-making problems, however, values $p_{ij}$ are not available. For example, think of the same car selection problem where colour is a criterion and there are cars with different colours (not values). In case of the latter problem, where values $p_{ij}$ are not available, the aforementioned procedure is also carried out for the alternatives (comparing alternatives against each criterion) to find $p_{ij}$ (the weight of alternative i with respect to criterion j)”. This is our case. Each software alternative should be evaluated with respect to each criterion to determine value $p_{ij}$.

Step 7: Calculating the overall scores of the alternatives. The overall scores of the alternatives need to be calculated as

$$V_i={\sum }_{j=1}^n{w}_j{p}_{ij}$$

(13)

Step 8: Ranking of the alternatives. The alternatives ranking is accomplished by ordering the global scores in decreasing order. Sorting the values of $V_i\forall i$, the best alternative is identified.

At the end of this step, the decision-maker should be able to select the most suitable simulation software for its specific application.

Step 9:Sensitivity analysis. Sensitivity analysis is a powerful tool to check the robustness of the model and eliminate bias in data collection and analysis [82]. In order to execute sensitivity analysis, the weight of the first-level criterion that got highest weight is varied from 0.1 to 0.9 and subsequently, the weights of all the other first-level criteria are varied [83]. The next step is to use these new main criteria weights in BWM methodology again to calculate new ranking of solutions in these new different conditions. The results are then compared and analysed.

4.6. Candidate Software Packages Ranking

At the end of the previous step, the selection methodologies will provide as a result a ranking of the candidate simulation software packages ordered according to the score given to them by the procedure. The “best” software will not necessarily be the optimum one, but it will emerge as the best trade-off between all the criteria considered, depending on the importance given to them. The sensitivity analysis will also highlight possible changes in ranking of the candidate software packages due to possible changes in the weights of the evaluation criteria and/or in the relative importance of the positive/negative aspects of the problem.

4.7. Best Software Solution

Once the decision-maker has selected the most suitable software for the specific application using one of the methodologies describes above, the next step is to negotiate a contract acceptable with the software vendor, by means of direct contacts with the supplier. The contract should specify what products and services will be provided, where and when they will be used, how the licence will be transferred to other parties, and how long the product will be used. Dates and obligations should be specified precisely to avoid any future misunderstandings. Where a suitable agreement cannot be achieved, nor an adequate level of support ensured, the chosen software (i.e., top-ranked alternative) will be abandoned and the second-ranked alternative will be examined on the basis of results of the quantitative evaluation. Basically, this first means going back to the previous step, for the determination of different software products, then returning to this stage, for achieving an acceptable agreement with the (new) software supplier. In general, the purchase of the simulation software will be finalized if an acceptable agreement can be reached.

4.8. Implementing the Solution

At this stage, the software package is finally purchased. The next stage for the company is to set up training activities for the internal staff on the usage of the software. Once the staff have become experts in using the simulation software, the company can proceed with the implementation of the software solution by conducting tests and pilot activities followed by a phased rollout. After the implementation of the solution, the company will benefit from all the advantages deriving from the adoption of a simulation software and can explore all its possible usages.

4.9. Maintaining and Improving the Solution

When the solution has been implemented, the company will need to maintain the tool and ensure the possibility of updating it to accommodate new needs or possibilities, which could bring improvement or revisions to the selected software.

5. Case Study

This section details the full application of the framework proposed in Section 4, for selecting the best simulation software in a real case. The aim of the application is to determine the most suitable simulation software to be used for Participatory Design, a well-established approach also known as co-operative design and frequently used in Scandinavia [84,85]; according to this approach, all stakeholders, and therefore also the end-users, take an active role in co-designing solutions for themselves, to help ensure that the end product meets the needs of its intended user base.

The scene of the application is a mid-sized enterprise (called Company A), located in the South of Italy, and whose main business is the production of machined components, 3D printing and additive solutions for the aerospace and automotive industries. The entire evaluation phase has been carried out in close contact with the company’s managers, who acted as the key decision-makers for the application in question.

5.1. Stage 1. Self-Assessment of the Digital Readiness of the Company Using Maturity Model

The first stage in the implementation of the proposed procedure consists of evaluating the current maturity level of the company, so as to judge its suitability for the implementation and usage of a simulation software. To this end, the questionnaire shown in Appendix A and described earlier in the paper was used. Responses were obtained thanks to two meetings and some interviews with the company’s representatives, which took a couple of days. Applying the computational procedure (Equations (1) and (2)) to the responses obtained, it was found that the targeted company scored 57.00 in Process (Defined maturity level), 55.00 in Technology (Defined maturity level), 58.00 in Organization (Defined maturity level) and 32.00 in People (Repetitive maturity level). Therefore, the overall score was 50.5, which classified the company at level 2, but very close to level 3. Managers became aware of improving people management to improve the digital readiness of Company A.

5.2. Stages 2 and 3. Core Problem, General Goals and Requirements

For the case under examination, the core problem is to identify the most suitable simulation software tool for Participatory Design. The general goals of the software were well known to the company’s managers, so that this step simply consisted in formalising them and was quite immediate in terms of time. More precisely, the idea of the company was to select a tool with: (1) good performance in terms of simulation of manufacturing scenarios; (2) ease of usage and of understanding; (3) excellent graphics capabilities; and (4) a good set of animations. These characteristics form the basis for the effectiveness of the software in recreating daily cases with a sufficient level of detail. Indeed, the aim of Participatory Design is to directly involve workers and stakeholders; hence, the level of detail required could be very high. At the same time, the desired software should allow quick model changes and analysis of the effects of these changes. This means that a high simulation speed is desirable. Furthermore, it should allow the import of CAD files of original machines, the import of background images of the layout, in order to represent the real process and the real workplace.

5.3. Stage 4. Preliminary Screening of the Simulation Software Packages on the Market

As per the previous step, managers of Company A had already made a market analysis for identifying a possible set of potential candidates and found three software packages, called S1, S2, and S3 for the sake of confidentiality. Each alternative reflects a particular class of simulation software packages: S2 is a discrete-event simulator, S3 is an agent-based simulator and S1 has a continuous engine.

5.4. Stage 5. Evaluation and Selection of the Best Software

5.4.1. The Integrated BOCR-AHP Methodology

The decision criteria set (Step 1—Section 4.5.1) and the BOCR structure (Step 2—Section 4.5.1) were utilized for this case. Then, in order to compute the criteria weights, the pairwise comparison matrices A of criteria under each merit B, O, C, R, using the intensity scale proposed by [80], were created; this required involving the company’s managers in a meeting lasting three hours approximately. The computational steps were instead carried out by the authors; in particular, the pairwise comparison matrices were used to calculate the vectors of the weights of criteria and sub-criteria under each merit. Moreover, the consistency of judgements was checked. The next step is to derive the relative priorities (preferences) of the alternatives with respect to each criterion and sub-criterion. To this end, the pairwise comparison matrices B of the alternatives for each sub-criterion were first determined using the Saaty scale. These matrices were used to calculate the scores of the alternatives for each sub-criterion under each merit B, O, C, R, using the procedure described previously. The consistency of the judgements was also checked. Then, the score vector of the alternatives for the remaining criteria was determined. Finally, the score of the alternatives under each merit, $B_i$, $O_i$, $C_i$ and $R_i$, was calculated;

Table 5. Score of the alternatives under merits B, O, C and R.

	${\mathsf{B}}_{\mathsf{i}}$	${\mathsf{O}}_{\mathsf{i}}$	${\mathsf{C}}_{\mathsf{i}}$	${\mathsf{R}}_{\mathsf{i}}$
S1	0.491	0.273	0.282	0.651
S2	0.233	0.296	0.507	0.134
S3	0.276	0.437	0.209	0.215

summarizes the outcomes of the computation.

In order to rank the alternatives, their overall score was computed by combining the outcomes obtained in each perspective ($B_i$, $O_i$, $C_i$ and $R_i$), following a multiplicative approach. By setting the same priority for each merit (b = o = c = r = 0.25), the global scores shown in

Table 6. BOCR-AHP global scores.

Alternative	Global Score
S1	0.730
S2	1.015
S3	2.684

were obtained.

According to the proposed methodology, the most suitable simulation software for Participatory Design turns out to be S3. The second-ranked alternative is S2, while S1 got the lowest score.

A sensitivity analysis was carried out on the results obtained to investigate the possible changes in the ranking of alternatives if different formulae (other than the multiplicative approach) were used to combine the score of the alternatives under each perspective ($B_i$, $O_i$, $C_i$,$R_i$). Moreover, four alternative scenarios were considered as far as the weights of the perspectives are concerned; more precisely, in each scenario one of the perspectives (at a time) was assigned a double importance value compared with the remaining one (i.e., 0.4 vs. 0.2). By combining these scenarios with the different approaches that can be used for the computation of the scores, 25 scenarios were obtained overall for the sensitivity analysis. The relating results, in terms of the changes in the ranking of the candidate software packages, are graphically shown in Figure 8.

In Figure 8, the scenarios are labelled with the first word that indicates the perspective assigned the highest relative importance (i.e., benefit, cost, etc.). Otherwise, base is used to denote the situation where the four perspectives are assigned the same importance. The remaining words describe the approach used for computing the scores of the alternatives. Looking at the scores, in the base-multiplicative scenario (which led to the results in Table 6), the ranking of the candidate software packages is as follows: S3>S2>S1. In general, such a ranking is quite robust, as it is observed in most of the scenarios considered (18 scenarios). In 6 scenarios the ranking is S3>S1>S2, while only in the risk-additive scenario, the ranking is S2>S3>S1. Overall, apart from this latter scenario, S3 is always found to be the top-ranked alternative.

5.4.2. The BWM Implementation

Pairwise comparisons of the software packages for each criterion are shown in

Table 7. Pairwise comparison of the software packages for each criterion (Note: red = worst criterion; green = best criterion).

Criteria	VC to Others			PS to Others			AL to Others
	VC	PS	AL	VC	PS	AL	VC	PS	AL
General Characteristics (GC)
Compatibility	1	1	1/5	1	1	1/5	5	5	1
Integration with other systems	1	1/3	1/3	3	1	1	3	1	1
Data security	1	1/3	3	3	1	5	1/3	1/5	1
Maximum simulation speed	1	1/9	1/5	9	1	5	5	1/5	1
Efficiency	1	1/5	1/3	5	1	3	3	1/3	1
Supported languages	1	1/3	1/5	3	1	1/3	5	3	1
Visual Aspects (VA)
Icons	1	3	1/5	1/3	1	1/7	3	7	1
Graphical background images	1	1	1	1	1	1	1	1	1
3D objects	1	7	3	1/7	1	1/5	1/3	5	1
3D animation	1	9	7	1/9	1	1/7	1/5	7	1
Visual quality	1	7	5	1/7	1	1/3	1/5	3	1
Virtual reality	1	5	7	1/5	1	3	1/7	1/3	1
Logical Aspects (LA)
Incorporate-merge models	1	1/3	1/3	3	1	1	3	1	1
Coding	1	1	1	1	1	1	1	1	1
Attributes and variables	1	1/3	1/3	3	1	1	3	1	1
Routing rules	1	3	1	1/3	1	1	1	3	1
Queueing rules	1	1	1	1	1	1	1	1	1
Input (IN)
Data Input	1	1	1/5	1	1	1/5	5	5	1
Statistical information on input data	1	1	1/3	1	1	1/3	3	3	1
Output (OU)
Data export	1	1/3	1/3	3	1	1	3	1	1
Statistical information on export data	1	3	3	1/3	1	1	1/3	1	1
Export animation	1	1	1	1	1	1	1	1	1
Ease of use (EU)
Graphical user interface	1	3	1/3	1/3	1	1/5	3	5	1
Graphical model construction	1	1	1	1	1	1	1	1	1
Real-time viewing	1	1/5	1/5	5	1	1	5	1	1
Mobile application	1	9	9	1/9	1	1	1/9	1	1
Support and training (ST)
Manual	1	1	1	1	1	1	1	1	1
Tutorial	1	1	5	1	1	5	1/5	1/5	1
Online support	1	1/7	1	7	1	7	1	1/7	1
Demo version	1	1/3	1/3	3	1	1	3	1	1
Updates	1	1	1	1	1	1	1	1	1
Specialized training	1	1	1	1	1	1	1	1	1
Vendor (VE)
Vendor strength	1	1/9	1/4	9	1	5	5	1/5	1
Costs (CO)
Acquisition cost	1	3	1/3	1/3	1	1/5	3	5	1
Implementation cost	1	1	1	1	1	1	1	1	1
Cost of updates	1	1	1	1	1	1	1	1	1

. Aggregation (based on simple average) to determine the overall weights for the criteria and sub-criteria was used.

As can be seen from the table above, in the first-level criteria set, Ease of use is identified as the best criterion, and Vendor as the worst. Moreover, for example, in the group General Characteristics, the Best criterion is Maximum simulation speed, while the worst criterion is Supported languages. As for Visual Aspects category, the best criterion is 3D animation, the worst criterion is Graphical background images. Finally, in the Logical Aspects group, the best criterion refers to Coding, while the worst to Queueing rules.

At the end of the process, the following global scores were obtained:

$$\left(\begin{array}{c}S1\\ S2\\ S3\end{array}\right)=\left(\begin{array}{c}0.315\\ 0.283\\ 0.401\end{array}\right)$$

According to the BWM, the most suitable simulation software for Participatory Design is S3. Finally, a sensitivity analysis was performed to check the robustness of the model. To this end, the weight of the best first-level criterion, Ease of Use was varied from 0.1 to 0.9; then, the weights of the remaining first-level criteria were varied as well. A total of ten different runs were performed.

Table 8. Variation in weight value alter varying Ease of Use (EU) weight value.

Run	GC	VA	LA	IN	OU	EU	ST	VE	CO
Original	0.126	0.126	0.076	0.126	0.076	0.313	0.054	0.027	0.076
Run1 (0.1)	0.165	0.165	0.099	0.165	0.099	0.100	0.071	0.036	0.099
Run2 (0.2)	0.147	0.147	0.088	0.147	0.088	0.200	0.063	0.032	0.088
Run3 (0.3)	0.129	0.129	0.077	0.129	0.077	0.300	0.055	0.028	0.077
Run4 (0.4)	0.110	0.110	0.066	0.110	0.066	0.400	0.047	0.024	0.066
Run5 (0.5)	0.092	0.092	0.055	0.092	0.055	0.500	0.039	0.020	0.055
Run6 (0.6)	0.073	0.073	0.044	0.073	0.044	0.600	0.031	0.016	0.044
Run7 (0.7)	0.055	0.055	0.033	0.055	0.033	0.700	0.024	0.012	0.033
Run8 (0.8)	0.037	0.037	0.022	0.037	0.022	0.800	0.016	0.008	0.022
Run9 (0.9)	0.018	0.018	0.011	0.018	0.011	0.900	0.008	0.004	0.011

shows the weights of all the main criteria after a variation in the weight of Ease of Use. The new ranking of solutions in the ten different scenarios is shown in

Table 9. Ranking of solutions during sensitivity analysis (criterion EU variation).

Solutions	Original	Run1	Run2	Run3	Run4	Run5	Run6	Run7	Run8	Run9
S1	0.315	0.322	0.319	0.316	0.312	0.309	0.306	0.303	0.300	0.297
S2	0.284	0.287	0.285	0.284	0.282	0.281	0.280	0.278	0.277	0.275
S3	0.401	0.391	0.396	0.400	0.405	0.410	0.414	0.419	0.423	0.428

.

5.5. Stage 6. Candidate Ranking

The ranking returned by the application of the BOCR-AHP approach is S3-S2-S1, and thus S3 is the most suitable simulation software. When applying the BWM, the ranking of the alternatives is S3-S1-S2, which confirms the findings of the previous approach, as S3 still turns out to be the most suitable simulation software.

5.6. Stages 7, 8 and 9. Software Purchase, Implement, Maintain and Improve Solution

These stages form the practical part of the approach and deal with the purchase of the software package, its installation and maintenance at the targeted company. At the time of writing, the purchase and implementation of the selected software at Company A are still ongoing; hence, these stages could not be fully covered by this study and will be analysed in future research.

6. Conclusions

This paper has proposed an original and structured methodology to help the managers of an enterprise in selecting the most suitable manufacturing simulation software package for their needs, taking into account all the positive and negative aspects of such a decision. The methodology embodies a maturity model that is expected to help companies (especially SMEs) understand if they can fully exploit digitalization. Overall, the framework is a structured approach, composed of nine steps that could be customised when needed, depending on the specific application and/or on the industrial problem. In particular, the fifth stage is the true evaluation and selection of the software and is carried out by means of two approaches, namely:

-

the new integrated BOCR-AHP methodology. It consists of six ordered sub-steps and should help select the best software and provide a ranking of alternatives;

-

the BWM approach. It includes eight ordered steps to select the best simulation software from a list of candidates.

In the case under examination, the short list of potential candidates was made of three software packages (S1, S2, and S3). As mentioned before, each alternative reflects a particular class of simulation software packages: S2 is a discrete-event simulator, S3 is an agent-based simulator and S1 has a continuous engine. According to the BOCR-AHP approach, the most suitable simulation software for the problem in exam is S3. The second-ranked alternative is S2, followed by S1, which got the lowest score. The ranking with BWM confirms the finding of the previous approach, and, in particular, S3 still turns out to be the most suitable simulation software. This suggests robustness of the proposed methodologies.

From a theoretical perspective, this framework has the fundamental advantage of being easy to implement and not requiring significant efforts. Indeed, the company’s involvement is needed in the first step of the approach, to reply to the questionnaire regarding the digital readiness self-assessment, and when applying the MCDM methodologies, for gathering data and opinions from experts. Moreover, the computational procedure is quite easy, as it can be performed using an Excel spreadsheet.

From a practical perspective, because the analysis is based on a real enterprise, the results are expected to be useful to managers facing similar problems in real scenarios. Moreover, reusable software, framework and approaches can be used towards the development of Industry 4.0. To be more precise, this means that if there are similar demands, the framework for simulation software selection or codes can be reused to some extent without developing it again from scratch [85]. Because we have presented an exhaustive case study, the proposed approaches could be easily replicated in other companies whose criteria and sub-criteria for simulation software selection are similar to those presented in this study. The fact that some stages of the evaluation framework can be implemented in Microsoft Excel is also interesting, because this general purpose software is known and widespread. This is expected to encourage the application of the framework in practice. Finally, managers can use the proposed framework for monitoring the performance of the purchasing process for a simulation software in a continuous improvement perspective (Industry 4.0 era), where simulation is central in operations management.

A general weakness of this study is that it is hard to evaluate the correctness of the results provided by the framework, as similar methodologies almost lack in the literature. In addition, the two MDCM approaches suggested for application in the framework have never been applied to the problem of selecting simulation software, thus preventing the possibility of direct comparisons of the outcomes.

Starting from this study, several future research directions could be undertaken. First, it is important to highlight that, as the hierarchical structure can be updated, criteria and software candidates can be added or removed at any time, meaning that different frameworks could be built starting from that presented in this paper. As a further research direction, the proposed framework could be generalised, with the purpose of applying it to different fields or for the selection of other types of software tools. Moreover, from a technical point of view, this work could be extended by embodying an integrated BOCR-BWM evaluation methodology, which would combine the advantages of the BWM method over AHP, with the advantages of a BOCR analysis.