A Systematic Selection Process of Machine Learning Cloud Services for Manufacturing SMEs

Abstract

Small and medium-sized enterprises (SMEs) in manufacturing are increasingly facing challenges of digital transformation and a shift towards cloud-based solutions to leveraging artificial intelligence (AI) or, more specifically, machine learning (ML) services. Although literature covers a variety of frameworks related to the adaptation of cloud solutions, cloud-based ML solutions in SMEs are not yet widespread, and an end-to-end process for ML cloud service selection is lacking. The purpose of this paper is to present a systematic selection process of ML cloud services for manufacturing SMEs. Following a design science research approach, including a literature review and qualitative expert interviews, as well as a case study of a German manufacturing SME, this paper presents a four-step process to select ML cloud services for SMEs based on an analytic hierarchy process. We identified 24 evaluation criteria for ML cloud services relevant for SMEs by merging knowledge from manufacturing, cloud computing, and ML with practical aspects. The paper provides an interdisciplinary, hands-on, and easy-to-understand decision support system that lowers the barriers to the adoption of ML cloud services and supports digital transformation in manufacturing SMEs. The application in other practical use cases to support SMEs and simultaneously further development is advocated.

1. Introduction

As part of the fourth industrial revolution and the associated digitalization of production, data, things, and processes are becoming more and more interconnected [1]. The increasing connectivity is based on the Internet of Things (IoT), which is characterized by integrating technology-enabled physical objects into a cyber-physical network [2]. The integration of cyber-physical production systems (CPPS) allows for the extraction of process data and, consequently, lays the foundation for a smart, interconnected, and sustainable manufacturing ecosystem [3]. Ongoing digitalization and the implementation of CPPS in factories lead to the generation of large amounts of data [4,5]. Connecting machines with IoT technology and services connecting physical processes with digital services enable data processing and analytics. In this context, methods for analyzing large and heterogeneous datasets are vital competencies necessary for a more efficient production [6]. By enabling machines to extract, process, and send data, large quantities of datasets can be made available for applications based on artificial intelligence (AI) [7]. AI techniques, especially machine learning (ML), are suitable for realizing intelligent systems, ensuring continuous process optimization, and transforming more sustainable energy usage in manufacturing [8]. In theory, AI, which is often considered as automation of rational behavior, facilitates various use cases such as quality improvement, process control, demand planning, or logistics [9,10]. In practice, integrating AI systems still poses major challenges for manufacturing companies, especially in the fields of data quality, data processing, model selection, and cybersecurity [11].

Particularly for small and medium-sized enterprises (SME), implementing new digital solutions, such as AI systems, is often associated with challenges and, therefore, is not widespread [12]. SMEs are defined by the European Commission as companies with fewer than 250 employees and a turnover of less than €50 million or €43 million in revenue, although other definitions exist worldwide [13]. They can further be divided into micro-, small-, and medium-sized enterprises. In this paper, we only consider small and medium-sized enterprises with more than ten employees, following the definition of the European Commission. With more than 99% of EU employees working in SMEs, SMEs are considered the backbone of the European economy and are particularly important [14]. Besides a lack of knowledge in data science, SMEs often lack a sufficient infrastructure to implement new technologies [15]. To drive digitalization and enable the use of new digital tools, SMEs can leverage cloud-based solutions [16]. Cloud computing provides on-demand availability of IT resources such as infrastructure connecting company-related resources with cloud providers, enabling access to cloud-based AI services for analyzing data. In addition to the many advantages of cloud computing, SMEs face numerous challenges when migrating to the cloud [17]. They include difficulties assessing the suitability of different cloud providers and the integration of cloud services into the company’s IT and enterprise architecture [18]. On the one hand, SMEs are challenged to identify criteria that need to be considered when selecting a cloud provider or service. On the other hand, there is a lack of practical and easy-to-use selection processes supporting the multi-criteria decision problem. Moreover, the vast variety of cloud providers and services is confusing, and a successful selection without assistance is complex. A decision support system (DSS) can help SMEs identify a cloud solution that fits the company’s needs [19]. To change towards a more sustainable production by increasing energy efficiency and flexibility, SMEs in the manufacturing sector have to adapt to new technologies such as the ML services that numerous cloud providers supply. Although the starting point is well-known, companies still find it challenging to migrate to the cloud and implement cloud-based services [18]. However, there is a lack of studies facilitating this specific selection process. Current research is limited to publications that identify and evaluate the requirements of companies regarding cloud computing, whereas an applicable DSS is missing. To close this gap, the purpose and objective of this study are to develop a tool that enables small and medium-sized manufacturing companies to select an appropriate cloud-based machine learning service. Thereby, we assume as our hypothesis that SME manufacturing companies can systematically select a cloud-based machine learning service.

We, therefore, identify the various requirements of SMEs on the topic of cloud computing. For this purpose, related publications evaluating the current state of the art are investigated. As a result of this process, we derive a multilevel criteria catalog, summarizing the various requirements of SMEs regarding cloud computing. The derived multilevel criteria catalog is combined with a systematic selection process based on the analytic hierarchy process (AHP) initially introduced by Saaty [20]. Consequently, a DSS, facilitating the selection of a suitable ML cloud service following a design science research (DSR) approach, is developed. To ensure reliable results, the artifact is validated by conducting a case study of a German manufacturing SME that aims for more sustainable production by selecting a suitable cloud-based ML service that enables energy efficiency and flexibility measures.

With our designed DSS, we contribute to the theoretical body of research and practice by providing an approach that enables manufacturing SMEs to systematically select ML cloud services based on a comparable starting point. Although applying the AHP to facilitate multi-criteria decision problems known from other research fields, an exaptation is created, extending design knowledge to a new and relevant problem [21]. Moreover, the extension of design knowledge into the field of cloud computing is nontrivial and adds value for the following reasons: (1) Although adopting cloud services in an industrial environment is established in theory, practical approaches on how to select a suitable ML cloud service are still scarce since this research paper is among the first to develop a practical DSS for this exact case study. By providing the developed artifact, a selection of cloud services and a contribution to the digital transformation of manufacturing SMEs is possible. (2) Applying the AHP and the derived multilevel criteria catalog enables SMEs to systematically select the right cloud provider related to the defined requirements. (3) We provide an interdisciplinary and hands-on artifact by merging knowledge from different domains such as manufacturing, cloud computing, and ML, with practical aspects using IS research methods. Through the structured and easy-to-understand DSS, the reduction of existing barriers in practice is achievable, therefore increasing the acceptance for the adoption of ML cloud services by manufacturing SMEs.

This paper is organized as follows: Section 2 presents the findings of a literature review outlining the underlying topics of cloud computing and ML for SMEs, thereby elaborating on existing approaches and comparing state-of-the-art concepts. In Section 3, the paper’s research method is presented. Section 4 comprises the designed artifact, a four-step DSS facilitating manufacturing SMEs’ ML cloud service selection. Section 5 demonstrates the designed artifact, with a practical case study of a German SMEfor the utilization of the DSS to select a suitable ML cloud service. Based on the findings from the case study, we discuss the developed artifact and its limitations in Section 6, before the conclusion in Section 7.

2. Related Work

2.1. Cloud Computing for SMEs

Today’s companies are subject to constant change caused by volatile markets and the associated increase in global market pressure [22]. In times of advancing digitization, small and medium-sized manufacturing companies are increasingly interested in leveraging modern technologies such as AI to increase value creation in their operations and thus remain competitive [23]. Since SMEs generally often lack the resources, infrastructure, and knowledge to introduce AI Systems, the use of cloud-based services from companies such as Google, Amazon, Microsoft, and IBM offers a reasonable alternative [18,24,25,26,27].

In manufacturing, the adoption of cloud computing offers the potential to improve traditional processes, thus moving into an era of smart manufacturing with more agile, scalable, and efficient processes. The basic idea behind cloud computing is to enable demand-oriented access to various computing resources with high reliability, scalability and, availability in a distributed environment [25]. Cloud-based services and products can be flexibly booked, scaled, and rereleased with little effort [28]. Three different service delivery models can be distinguished [29]. Infrastructure-as-a-Service (IaaS) enables the utilization of IT infrastructure eliminating the need for investing in building and managing IT systems [17]. Platform-as-a-Service (PaaS) provides users with an environment that enables the development, testing and deployment of applications, accelerating application development by simplifying many process steps [30]. Software-as-a-Service (SaaS) offers a complete product managed and run by cloud providers. Since essential know-how or other resources are not sufficiently available, SMEs often fail to implement technologies such as AI. Since cloud providers recognize this problem, several cloud-based AI platforms now exist on the market to support companies in implementing AI use cases [31]. In the context of this work, cloud-based AI platforms are Machine Learning as a Service (MLaaS) solutions. As with SaaS or IaaS, MLaaS solutions provide services necessary for ML on demand. MLaaS solutions provide storage, computing capacity, algorithms, and more, over the cloud [23].

While large companies have quickly adopted cloud computing, many SMEs are still contemplating migrating to the cloud [32,33,34]. Even though cloud computing has been around for many years, the migration to the cloud still implies various challenges such as data privacy, compliance, reliability, and performance issues that SMEs need to consider [35,36]. Moreover, overcoming overarching issues such as the resistance towards new technologies, missing internal skills, and the doubt about the benefits of cloud computing are of great importance, too [18]. Thus, SMEs do not yet realize the full potential of ML because of existing barriers and challenges to implementation. This translates into an implementation gap in practice and provides research opportunities that are not yet studied sufficiently. The advantages of cloud computing include the rapid integration of new systems and the reduction of capital expenditure based on the usage-based billing of many cloud services [18]. Other benefits include increased availability and performance of IT systems contributing to better productivity and flexibility of the corresponding organization (Metzger et al. 2011).

2.2. Existing Approaches for Cloud Service Selection

Adopting new technologies such as cloud computing is a complex and challenging course [37]. While research acknowledges this issue, the existing literature generally focuses on deriving requirements and criteria for adopting cloud solutions [18,26,35,38,39]. Even though these studies are dedicated to providing a vital assessment foundation, they lack to introduce a practical decision support system, which enables systematic selection of a suitable cloud service based on the underlying requirements and criteria. Consequently, the resulting uncertainty due to a lack of knowledge when evaluating different providers is a hurdle for risk-averse decision-makers in SMEs [40,41,42]. Furthermore, some studies focus on generic criteria, whereas others identify specific criteria for SMEs. Repschläger et al. [19] derive a methodology to support the decision-making process for cloud customers using the AHP. The procedure includes four phases: (1) structuring the cloud provider selection problem; (2) measurement and data collection; (3) determining normalized weights; and (4) finding a solution to the selection problem. By applying this process, the authors derive 62 criteria regarding cloud provider selection and prioritize them based on common and non-critical processes and services such as human resource management. However, this is in contrast to the critical processes present in manufacturing, which have special requirements, for example, in terms of reliability and compliance with logistical targets [7]. Further limitations of the study conducted by Repschläger et al. [19], include an unspecific, and thus unpractical, assortment of criteria complicating the assessment of suitable cloud solutions for SMEs and the absence of a comprehensive system enabling the systematic selection of cloud services from end-to-end. Moreover, the current literature depicts that research focuses on generic cloud services and neglects the specific requirements necessary for ML cloud services, representing a barrier for SMEs. Furthermore, the procedure of evaluating and rating cloud services and decision-making based on a practical case is still pending, which necessitates a DSS that details the end-to-end process of ML cloud service selection for SMEs.

2.3. ML in Manufacturing

Continuous improvement and optimization of processes are key requirements of manufacturing companies to stay competitive [43]. Due to the integration of smart devices and machines in modern production sites, large amounts of data are generated that need to be stored, processed, and analyzed to extract value from the data thus creating a beneficial advantage [44]. Today, technologies related to AI or ML are increasingly utilized to analyze large and heterogeneous datasets [45]. ML is a subfield of artificial intelligence which enables computer systems to recognize correlations from data, thereby making human-like decisions without defined rules [46]. ML is based on the generalization of knowledge from data and can be realized with different methods such as classification, clustering, regression, and anomaly detection [47]. The basis for a successful implementation of ML models is suitable algorithms and large amounts of high-quality datasets. Depending on the quality of the available data, the prediction accuracy of ML models varies [48], thus requiring use of case-specific ML models [49]. In general, three categories of ML can be distinguished, two of which are common in cloud computing [50]. These categories can be distinguished according to how the models learn from the data. In supervised learning, models are trained with historical data that has already been classified by an external source and used for reproducing classifiers [31]. Unsupervised learning is characterized by the context that no outcome values are known. Thus, the ML algorithms process input data to gain insights by themselves [51]. While unsupervised learning is often used in grouping similar cases with an unclear definition of classes, supervised learning enables classifying cases according to predefined classes [10,52].

To ensure the long-term success of SMEs in the manufacturing sector, efficient and flexible energy use is crucial [53,54]. Some key incentives for contributing to the sustainable development of the manufacturing industry include rising energy costs, ever-stricter environmental statutes, and ethical and moral obligations [55,56,57]. To overcome these challenges and enable more sustainable manufacturing, an increasing level of ML is used [58]. The analysis of datasets utilizing ML enables numerous application opportunities and great optimization potential for industrial processes [59,60]. Existing solutions can be made more efficient and effective by deploying ML-based applications while new solutions are provided [7]. In manufacturing, ML can enable time and cost savings and increased quality and waste reduction. Typical use cases range from demand-side management (DSM) to process control, condition, monitoring, and predictive maintenance, enabling the continuous improvement of key performance indicators in manufacturing [8,61,62]. To supply high-quality reliable products, ML can be utilized to facilitate predictive model-based quality inspection, significantly reducing inspection volumes and generating economic advantages for manufacturing companies [63,64]. In order to leverage the diverse ML use cases in manufacturing SMEs and the associated efficiency gains, it is important that SMEs can identify their optimal service provider in a structured manner. However, ML applications are not yet widespread in manufacturing SMEs, and AI experiments and use cases are sporadic [7].

3. Research Method

Following the work of Gregor and Hevner [21], we organized research in this paper by the design science approach in information systems, combining research methods from engineering, economics, social science, and computer science [65]. This problem-solving approach is based on relevance and rigor, combining practical needs from a business environment with theoretical knowledge, thus contributing to IS research [66]. In line with this approach, our paper aims to contribute to the research gap by developing an artifact, a DSS facilitating the selection of an ML service for manufacturing SMEs. Non-existing DSS characterizes the relevance of our research for ML cloud service selection in manufacturing SMEs. To achieve rigor, the reliance on the appropriate application of theoretical foundations and methodologies is necessary. By merging theory and practice, the developed relevant artifact is based on the framework illustrated in Figure 1.

We adopted the design science research methodology (DSRM) initially introduced by Peffers et al. [67], consisting of the following six phases: problem identification, the definition of objectives, design and development, demonstration, evaluation, and communication. The DSRM is subsequently tailored to fit our specific artifact. Consequently, all steps are not included strictly as proposed. In accordance with the DSRM, Figure 1 provides a custom end-to-end overview of our research design. The first phase requires defining the specific research problem founded on the business environment and application domain and justifying the value of the proposed artifact (1). As outlined in the introduction, the adoption of cloud computing offers huge potential for SMEs by enabling the integration of new digital solutions such as ML, consequently setting the foundation for more sustainable manufacturing procedures. Therefore, in practice, a DSS that facilitates the systematic selection process of ML cloud services is highly needed. Our research addresses this gap, thus serving as a basis for SMEs’ successful selection of cloud services in the manufacturing industry. In the second phase (2), the knowledge foundation is identified, enabling our artifact’s development, based on the relevant literature and methods. Based on the work of Gregor and Hevner [21], the existing knowledge foundation and appropriate methods such as the AHP and expert interviews are used, thereby contributing an exaptation to the IS research. The sample size of the questionnaire is six, including expertise ranging from AI and manufacturing to project management in SMEs. The survey was conducted utilizing a structured questionnaire, which was sent to the corresponding experts by mail. The design and development phase includes creating and demonstrating the artifact (3) [67]. To support this process and at the same time derive relevant requirements and criteria enabling the decision process, the semi-structured literature research is conducted in the databases Google Scholar, ScienceDirect, Scopus, SpringerLink, Researchgate, and IEEE Xplore, searching for “cloud computing”, “SME”, “vendor selection”, “evaluation criteria”, “requirements”, “framework”, “criteria catalog”, “cloud services”, and “ML cloud services”. Based on the extracted criteria and the resulting criteria catalog, the designed artifact following the AHP approach is described in the next section. After several iterations of designing and optimizing our artifact, we conducted a case study to demonstrate our artifact. We applied our DSS to solve the underlying problem of systematically selecting an ML cloud service for SMEs. By communicating our results to researchers and other appropriate audiences, such as practicing professionals in manufacturing SMEs, the artifact aims to add to the existing knowledge base, thus enabling integration of the DSS in the business environment (4).

Analytic Hierarchy Process

The selection of an ML cloud service is a complex process in which many different influencing factors must be considered. While various methods enabling a systematic decision-making process can be found in literature, one approach proven to be useful is the AHP [19]. In contrast to other methods such as the utility analysis, the AHP offers benefits due to its increased transparency of the decision-making situation and the possibility that quantitative and qualitative data can be considered. Furthermore, the AHP can easily be understood and applied by operating managers, enabling decision-makers to reach a consensus [20]. The AHP has been developed for solving multi-criteria decision problems following the approach that a problem can be structured hierarchically to simplify complex decision-making. For this purpose, the criteria of the decision problem are reconciled and presented as a hierarchy. Elements of this hierarchy can be divided into groups and compared pairwise on each level. The results are translated into pairwise comparison judgment matrices (PCJM), enabling the determination of normalized weights, thus indicating the importance of the criteria. By utilizing the AHP, decision-makers can systematically break down the underlying decision problem, determine the priorities of the criteria, and compare numerous providers effectively to select the best suited [68].

4. Designed Artifact

In the context of this work, the approach of the AHP was adopted for the underlying objective: to investigate the suitability of cloud providers and their ML services for optimizing energy efficiency and flexibility. By utilizing DSR, a system (Figure 2) is designed following Godse and Mulik [69] and Tam and Tummala [68]. In their studies, the authors use the AHP to solve a complex provider selection problem. The approach involves the creation of a hierarchy that breaks down the decision problem into several levels. Based on the hierarchy, an evaluation of providers is conducted where different evaluation methods are being used. This is followed by pairwise comparison of two elements of each hierarchy level, enabling the criteria’s weighting and determining their importance. Finally, based on the evaluation results and the weighted criteria, the decision-making process takes place. The described four-step process is the starting point for complex decision problems and can be applied to many scenarios. In the context of this work, the described approach is used to design an artifact that enables the selection of suitable ML cloud services for SMEs. The four-stage DSS for machine learning cloud service selection is shown in Figure 2.

The presented artifact is customizable regarding the case offering an interdisciplinary approach that decision-makers of various SMEs can adopt. Whereas the results of steps 2, 3, and 4 are highly dependent on the implementing company and underlying case study, the structuring of the decision problem is generally valid for all SMEs. It can, therefore, be adopted for different use cases. For applying the DSS, the role of the decision-makers needs to be assigned to set assumptions and model the preferences accordingly.

Decision Process

In our artifact, we suggest a four-step DSS following the AHP introduced by Saaty [20]. First, the decision problem is structured (Step 1). During this process, a complex problem is decomposed and modeled as a hierarchical catalog (cf.

Table 1. Four-level criteria catalog for the machine learning cloud service selection (based on [19]).

Level 1: Goal	Level 2: Target Dimension	Level 3: Requirement Category	Level 4: Evaluation Criteria
Machine Learning Cloud Service Selection	IT Security	Security architecture	Data center security
			Cloud security
		Compliance	Data residence
			Compliance certifications
		Data protection	Conformity with the GDPR
	Reliability	Trustworthiness	Vendor reputation
			Vendor transparency
		Service promise	Service level agreements
		Redundancy	Geo-Redundancy
	Cloud management	Support	Technical support
			Community support
		Service	Free trial version
	Flexibility	Interoperability	Frameworks and SDKs
			Developer tools (IDEs)
		Portability	Data migration
			Data portability
	Costs	Payment method	Payment model
			Billing model
		Pricing model	Pricing
			Price transparency
	Performance	Usability	Service design
			Usability
			Customizability
		Functionality	Service functionality

) consisting of the goal, criteria, and subcriteria. The goal of our problem is the selection of a fitting ML cloud service. This goal is placed on the first level of the four-level hierarchy. The second level contains target dimensions. The target dimensions can be broken down and comprise different requirement categories placed on the third level. The fourth hierarchy level comprises evaluation criteria enabling the assessment of cloud providers and the associated services.

Based on the criteria derived in the initial step of the decision process, the providers can be evaluated (Step 2). This step differs from the usual AHP approach in that an independent rating scale is used. As suggested by Liberatore et al. [70], a five-point rating scale of outstanding (O), good (G), average (A), fair (F), and poor (P) is adopted. The respective priority weights for O, G, A, F, and P are determined as 0.503, 0.261, 0.134, 0.068, and 0.034, following the pairwise comparison approach introduced by Saaty [20]. The main reason for adopting this rating scale is the extensive scope of evaluating numerous criteria. The evaluation of cloud services involves many details consisting of many criteria. Therefore, adopting pairwise comparison to evaluate and rate cloud services regarding every criterion is too difficult and time-consuming. The use of the rating scale introduced by Liberatore et al. [70] can eliminate the difficulties, as the rating of the cloud services can be conducted without direct comparison. The next step of the decision process comprises weighting the criteria (Step 3). This step aims to compare the importance of the numerous criteria of the hierarchy (cf. Table 1), determining the importance of the respective elements regarding the attributes of the next higher hierarchy level. With this, the different criteria are plotted in matrices and compared in pairs. Following the approach of Saaty [20], this pairwise comparison is conducted for every hierarchy level, resulting in local weights. After the local weights of all hierarchy levels have been determined, the global weights of the evaluation criteria can be calculated. Therefore, the local weights of the target dimensions are multiplied with the local weights of the requirement categories. The results are then multiplied with the local weights of the evaluation criteria on the third level. The decision-making is the final step of the DSS (Step 4). In this phase, the results of the upstream steps are combined, enabling the selection of a suitable ML cloud service. To do so, the global weights of the evaluation criteria are offset against the results of the provider rating resulting in provider-specific results. Finally, the results can be compared to identify a suitable cloud service.

5. Case Study (Validation)

In the following, the functionality of the DSS is shown (Figure 2) by validating our artifact by conducting a case study of a medium-sized German manufacturing company that produces magnesium-casting parts. As part of the strategy, the company wants to implement energy efficiency and flexibility measures, thereby migrating to the cloud to enable DSM with the help of an ML cloud service. The company’s focus is on ML applications to predict future energy consumption on different time horizons, detect anomalies in energy consumption, and identify individual consumers in energy consumption via methods of non-intrusive load monitoring to initiate subsequent steps towards a more energy-efficient and flexible production. As part of a pre-selection process, three relevant cloud providers and their respective ML services were identified with the market share criteria, the scope of the cloud service portfolio, and the extent of the free trial version being the focus. Within the context of this process, the providers’ Amazon Web Services (AWS) and Microsoft Azure (Azure), as well as Google Cloud Platform (GCP) and their ML services Amazon SageMaker, Azure ML, and Google AI platform, were found to be suitable for the framework of the case study.

5.1. Decision Problem (DSS Step 1)

To solve the decision problem of selecting a fitting ML service, it is necessary to define a hierarchical catalog of criteria. Various scientific studies deal with the adoption of cloud solutions and have identified superior and inferior criteria for this process. Following the semi-structured literature review described in Section 3 of this paper, various scientific studies were identified that derive selection criteria for SMEs (cf. [16,18,35,38,71]). Through the exchange with six AI and manufacturing experts from research and practice, the criteria were evaluated to check for completeness and relevance for the framework of this study, making the different characteristics of the eligible cloud providers and ML services as tangible as possible. While some criteria have a clear quantitative character, others can only be measured on a subjective and qualitative level. Yet, they are still considered to provide a holistic view.

Based on many preliminary studies and expert interviews, we were able to identify factors that can be classified as important. Although many different evaluation criteria could be identified, not all of them are considered in the context of this study as only those criteria will be adopted that show great importance for the objective of this work, accordingly, focusing on the selection process of machine learning cloud services for manufacturing SMEs. Based on the literature analysis, a four-level hierarchy emerges, which is presented in Table 1. In total, 6 target dimensions, 14 requirement categories, and 24 evaluation criteria were identified.

The above-identified four-level hierarchy consists of relevant criteria and subcriteria used for the subsequent steps of the decision process. Six target dimensions mainly influence the selection of cloud services. Everything related to the protection and safety of the services and data is considered in the “IT-security” target dimension. The selection of cloud services is linked to rigorous requirements in privacy, data protection, and compliance. Companies need to ensure that their data and applications in the cloud meet the security requirements [17]. The “reliability” target dimension covers the availability and quality of cloud services. By migrating processes to the cloud, companies run the risk that these processes are linked to the reliability of the corresponding cloud provider [29]. Moreover, the permanent availability of the data and applications must be guaranteed [38]. The “cloud management” dimension comprises aspects the service provider carries out to enable a smooth operation of the various services [35]. This includes aspects necessary for the cloud management and maintenance of the relationship between client and provider [72]. When migrating to the cloud, companies seek “flexibility”. The use of cloud computing enables companies to be more agile, thus responding quickly to changing capacity requirements and market pressure [73]. The economic aspects are represented by the target dimension “costs”. The decisive factor here is whether cloud services offer cost advantages over traditional IT solutions. The last dimension comprises the “performance” aspect. When selecting a provider, knowledge of the scope and performance of the cloud services is crucial [74]. Companies need to know how well their applications perform on the different services and whether the resulting performance meets their expectations [39].

5.2. Service Evaluation (DSS Step 2)

To rate the above-mentioned cloud providers and services, thus enabling the decision-making in step 4, an evaluation regarding the most specific criteria on the fourth level of the hierarchy (cf. Table 1) is performed. Following a two-step approach, necessary information regarding the evaluation criteria (e.g., the usability of AWS SageMaker) is gathered and assessed, enabling cloud service rating based on the established scale of outstanding (O), good (G), average (A), fair (F), and poor (P). The first step of the evaluation includes extensive Internet research, where the websites of the various cloud providers are examined. The available information is analyzed, providing a foundation for rating the objective criteria such as data center security. In a second step, the associated ML services are evaluated using the free trial contingents to examine exemplary data, thereby gathering information on the service, providing a basis for rating the subjective criteria such as usability. While the second evaluation step is more time-consuming and requires more effort by the decision-makers, it is necessary to gather information about criteria that the Internet research does not provide. The two-step evaluation process mainly refers to the examination of the ML services. Since some aspects, such as the data center security, are provider-related criteria, assessing the ML services is not always possible. Due to this circumstance, the cloud provider itself is examined and rated in some instances. However, since the ML services are hosted on the cloud providers’ infrastructure and our research follows a holistic approach, evaluating these criteria is relevant and can be considered part of the examination and the subsequent decision-making procedure. Since the service evaluation is partially based on a subjective and qualitative assessment and is, therefore, susceptible to an evaluation bias, the results are not listed in this section but considered in the decision-making process (Step 4).

5.3. Criteria Weighting (DSS Step 3)

The decision to select a cloud service is highly dependent on the context-specific customer requirements reflected in the prioritization and weighting of decision criteria [19]. In the context of this work, two employees from the SME with domain expertise weighted the criteria regarding a cloud service selection based on the described scenario. To overcome problems of assessing decision matrices, the experts were first trained on the AHP principles before mailing them the questionnaires with short definitions of each criterion, enabling the criteria catalog’s weighting. Based on the underlying case study, the experts independently performed pairwise comparison, thereby providing numeric ratings for each of the criteria based on the scale implemented by Saaty [20]. After the experts edited the decision matrices, the corresponding PCJM’s were evaluated, resulting in local weightings of every hierarchy element. The corresponding matrices can be found in Appendix A and are the basis for weighting the criteria catalog (cf.

Table 2. Priority weighting of the criteria catalog based on expert interviews.

Target Dimension	Local Weight	Requirement Category	Local Weight	Evaluation Criteria	Local Weight	Global Weight
IT Security	0.272	Security architecture	0.29	Data center security	0.17	0.0134
				Cloud security	0.83	0.0655
		Compliance	0.06	Data residence	0.5	0.0082
				Compliance certifications	0.5	0.0082
		Data protection	0.65	GDPR conformity	1	0.1768
Reliability	0.172	Trustworthiness	0.6	Vendor reputation	0.5	0.0516
				Vendor transparency	0.5	0.0516
		Service promise	0.2	Service level agreements	1	0.0344
		Redundancy	0.2	Geo redundancy	1	0.0344
Cloud management	0.042	Support	0.75	Technical support	0.5	0.0158
				Community Support	0.5	0.0158
		Service	0.25	Free trial version	1	0.0105
Flexibility	0.118	Interoperability	0.83	Frameworks and SDK	0.25	0.0245
				Developer tools (IDE)	0.75	0.0735
		Portability	0.17	Data migration	0.83	0.0166
				Data portability	0.17	0.0034
Costs	0.228	Payment method	0.5	Payment model	0.17	0.0194
				Billing model	0.83	0.0946
		Pricing model	0.5	Pricing	0.25	0.0285
				Price transparency	0.75	0.0855
Performance	0.168	Usability	0.83	Service Design	0.22	0.0307
				Usability	0.45	0.0627
				Customizability	0.33	0.0460
		Functionality	0.17	Service functionality	1	0.0286

).

For example, all six experts rated the importance of the target dimensions on the first level of the criteria catalog, with IT security earning a local weight ($LW$) of 0.272, which was arrived at by assessing the PCJM (cf.

Table A1. Pairwise comparison judgment matrices of the machine learning cloud service selection.

Level 2—Target Dimensions
IT Security = I|Reliability = R|Cloud Management = C|Flexibility = F|Costs = C|Performance = P
	I		R	C	F	C		P	LW
I	1.00		5.00	3.00	3.00	1.00		1.00	0.270
R	0.20		1.00	5.00	3.00	1.00		1.00	0.172
C	0.33		0.20	1.00	0.20	0.20		0.33	0.045
F	0.33		0.33	5.00	1.00	1.00		0.33	0.119
C	1.00		1.00	5.00	1.00	1.00		3.00	0.227
P	1.00		1.00	3.00	3.00	0.33		1.00	0.167
CI = 0.174 CR = 0.140
Level 3—Requirement Categories
Security Architecture = S|Compliance = C|Data Protection = D					Trustworthiness = T|Service Promise = S|Redundancy = R
	S	C	D	LW		T	S	R	LW
S	1.00	7.00	0.33	0.295	T	1.00	3.00	3.00	0.600
C	0.14	1.00	0.11	0.057	S	0.33	1.00	1.00	0.200
D	3.00	9.00	1.00	0.649	R	0.33	1.00	1.00	0.200
CI = 0.041 CR = 0.07					CI = 0.000 CR = 0.000
Support = SU/Service = SE					Interoperability = I/Portability = P
	SU		SE	LW		I		P	LW
SU	1.00		3.00	0.750	I	1.00		5.00	0.830
SE	0.33		1.00	0.250	P	0.20		1.00	0.170
CI = 0.000 CR = 0.000					CI = 0.000 CR = 0.000
Payment method = PA/Pricing model = PR					Usability = U/Functionality = F
	PA		PR	LW		U		F	LW
PA	1.00		1.00	0.500	U	1.00		5.00	0.830
PR	1.00		1.00	0.500	F	0.20		1.00	0.170
CI = 0.000 CR = 0.000					CI = 0.000 CR = 0.000
Level 4—Evaluation Criteria
Data center Security = D|Cloud Security = C					Data residency = D/Compliance Certifications = C
	D		C	LW		D		C	LW
D	1.00		0.200	0.170	D	1.00		1.00	0.500
C	5.00		1.00	0.830	C	1.00		1.00	0.500
CI = 0.000 CR = 0.000					CI = 0.000 CR = 0.000
Vendor Reputation = VR|Vendor Transparency = VT					Technical Support = T|Community Support = C
	VR		VT	LW		T		C	LW
VR	1.00		1.00	0.500	T	1.00		1.00	0.500
VT	1.00		1.00	0.500	C	1.00		1.00	0.500
CI = 0.000 CR = 0.000					CI = 0.000 CR = 0.000
Frameworks and SDKs = F|Developer Tools (IDE) = D					Data migration = DM|Data Portability = DP
	F		D	LW		DM		DP	LW
F	1.00		0.33	0.250	DM	1.00		5.00	0.830
D	3.00		1.00	0.750	DP	0.20		1.00	0.170
CI = 0.000 CR = 0.000					CI = 0.000 CR = 0.000
Payment Models = P/Billing Models = B					Pricing = P|Price Transparency = PT
	P		B	LW		P		PT	LW
P	1.00		0.20	0.170	P	1.00		0.33	0.250
B	5.00		1.00	0.830	PT	3.00		1.00	0.750
CI = 0.000 CR = 0.000					CI = 0.000 CR = 0.000
Service Design = S|Usability = U|Customizability = C
	S			U		C			LW
S	1.00			0.33		1.00			0.220
U	3.00			1.00		1.00			0.450
C	1.00			1.00		1.00			0.330
CI = 0.068 CR = 0.117

) that comprises the mean of the six ratings according to the procedure introduced by Saaty [20]. Subsequently, the global weights ($GW$) of the evaluation criteria were calculated following equation 1 described below:

$$G{W}_{Evaluation Criteria}=L{W}_{Target Dimension}\times L{W}_{Requirement Category }\times L{W}_{Evaluation Criteria}$$

(1)

Exemplarily, the$GW$ of data center security (0.0134) was calculated by multiplying the $LW$ of IT security (0.272) with the $LW$ of security architecture (0.29) and the $LW$ of data center security (0.17). Accordingly, higher values of weights refer to a higher importance of the respective criteria. Based on the results of the expert interviews and the evaluation of the corresponding PCJMs, Table 2 provides the weighted criteria catalog:

The results illustrated in Table 2 indicate the importance of the superior and inferior criteria regarding the case study. On a higher level, represented by target dimensions, we observe that the selection of cloud-based ML services for this case study is mainly influenced by IT security aspects, suggesting that for SMEs in manufacturing, the protection of sensitive process data is of great importance. Furthermore, cloud security and conformity with the General Data Protection Regulation (GDPR) are considered critical for companies located in the European Union, which supports the findings of a survey conducted by Pols and Heidkamp [16]. Moreover, the importance of the specific costs, the associated billing model, and the price transparency of the ML services is high, which can be attributed to the necessity to estimate the cost-saving potential of the IT department [19]. According to the experts, an important aspect is the reliability of ML services, which manifests itself in the reputation and transparency of the provider and the promised service reliability defined in service level agreements. In conformity with the existing literature [7,75], the results of the expert interviews indicate that the integration of AI systems, such as anomaly detection in energy consumption, is closely linked to the service performance and associated criteria such as service usability. According to the experts’ assessment, from a flexibility point of view, this is a widespread perception. While developer tools (IDEs) are important for this case study, other criteria, such as data portability or data migration, rank on the bottom of the priority list. According to the results, cloud management criteria such as available free trial versions are irrelevant.

5.4. Decision Making (DSS Step 4)

The last step of the decision process, illustrated in

Table 3. Decision matrix enabling the identification of the most-suited ML cloud service.

Target Dimension	AWS SageMaker	Azure ML	GCP AI Platform
	Score	Score	Score
IT Security	0.0508	0.0508	0.0473
Reliability	0.0658	0.0783	0.034
Cloud management	0.0147	0.0173	0.0095
Flexibility	0.0578	0.0356	0.0503
Costs	0.0868	0.0664	0.0249
Performance	0.03	0.0664	0.0344
Σ Score	0.3059	0.3148	0.2004
Normalized Score	0.3725	0.3834	0.2441

, comprises the decision-making, thus identifying the most suitable cloud-based ML service to satisfy the goals and objectives of the case study. Since the results are based on an individual application example, which is reflected in the weighting of the evaluation criteria (cf. DSS Step 3) and the evaluated ML services (cf. DSS Step 2), the result is only meaningful for this specific case study. It, therefore, must be adapted if the objectives change or other ML services should be considered. To conduct decision-making, the results of the criteria weighting (cf. Table 2) and the service evaluation results are offset with each other. Therefore, the global weight of each evaluation criteria is multiplied by the weighted service score identified in the evaluation process. For reasons of clarity, the partial results of each evaluation criteria are added up and assigned to the respective target dimension. Accordingly, the various scores are summed up and normalized, enabling decision-makers to identify the most suitable ML cloud service for this case study by indicating the service with the highest score.

Based on the normalized scores illustrated in Table 3, Microsoft’s Azure ML had the highest score of the three services and, therefore, is the most fitting for meeting the specific goals described in this case study. To achieve a more flexible and efficient energy use in production by applying DSM, the SME should prioritize the use of Azure ML. The second-best ML service is AWS SageMaker, which performed almost as well as Azure. Therefore, GCP’s AI Platform has the lowest overall score and is least suitable for this case study. When aggregated, the results provide an accessible overview of which cloud service offers the best solution for each target dimension. As detailed in Table 2, IT security aspects as well as cost, reliability, and performance features are driving factors when identifying a suitable ML service based on the presented case study. As indicated in Table 3, Azure ML offers the best solution for three of the four most influential target dimensions, namely reliability, performance, and IT security. It is considered the most suitable in this context. Especially in terms of IT security, which is a key requirement for companies in manufacturing, AWS SageMaker is on the same level as Azure ML, convinces with the best cost and flexibility rating, and, thus, can be considered a good alternative. In terms of the ranking illustrated in Table 3, Google’s AI Platform is inferior to its competitors in all respective target dimensions and, therefore, cannot be recommended for the SME.

6. Discussion

This section discusses the case study results before evaluating the developed DSS in general and deriving limitations and prospects for further research. Three focal points were identified. First, because of the specificity of the underlying case study and the corresponding weighting of the criteria as well as the lack of studies, when identifying a suitable ML service for manufacturing SMEs, the intention is not to conduct a holistic discussion on whether Azure ML is a widespread and satisfactory solution for similar application domains. As illustrated within the results in Section 5 and Table 3, the ML cloud services Azure ML and AWS SageMaker perform nearly similarly well, with a slight advantage for Azure ML in our specific case study. AWS SageMaker even outperforms Azure ML within several target dimensions regarding the scores given. This indicates that in other SMEs with another weighting of the different underlying criteria, AWS SageMaker might be the service provider to choose. Due to the relatively close race between Azure ML and AWS SageMaker, it is relevant that the data used for evaluation is thoroughly and comprehensively collected so that a robust decision can be made. However, it also shows that Azure ML and AWS SageMaker generally perform well. Second, the cloud providers and ML services found to be relevant are evaluated based on the four-level criteria catalog (cf. Table 1). While conducting the case study, converting the service providers’ information into reasonable scores was not easy, even though the criteria are clearly defined and quantifiable. To properly evaluate the different cloud service providers, collecting as much information as possible is necessary. Deriving information from the providers’ websites might not be sufficient. Contacting sales consultants and using the available free testing contingents is advised. Third, future-oriented decisions in SMEs must be aligned with the strategic course. Thereby, relevant stakeholders within the company must be identified and involved in the decision-making. Experts from IT, manufacturing, and strategic positions might be suitable to account for interdisciplinary requirements. This leads to point four, as the status quo regarding automatization, digitalization, and existing knowledge on ML and AI as a fundamental starting point to incorporate ML cloud services into existing information systems.

The integrated four-step system provides an interdisciplinary approach that can be adopted by decision-makers of numerous SMEs in the manufacturing industry. As mentioned in Section 2, Repschläger et al. [19] derived a methodology to support the decision-making process for cloud customers identifying evaluation criteria and prioritizing them based on common and non-critical processes, not detailing how to use the weighted criteria to identify and select a suitable cloud provider. Our approach differs in that we contribute a DSS that facilitates an end-to-end approach for manufacturing SMEs, specifying every step necessary to identify a suitable solution, thereby detailing the processes of evaluating and rating cloud services and decision-making based on a particular case study. By applying our artifact in a case study, we demonstrate that it is possible to identify a suitable cloud-based ML service based on company-specific preferences. Furthermore, the structuring of our decision problem (cf. Table 1) is not based on generic criteria applicable for non-critical processes and systems known from Repschläger et al. [19] and others but rather focuses on specific criteria of SMEs regarding the use of cloud-based ML services considering the critical and complex requirements of manufacturing processes and systems. Since the priority weighting of the numerous criteria (cf. Table 2) is a difficult task that is further complicated by the fact that the differences between the various criteria are sometimes unclear and fluid, a precise definition is essential. Through exchange with AI and manufacturing experts as well as resulting iterations of adjusting the criteria catalog, it constantly improved the demarcation between criteria, thus simplifying the criteria weighting based on decision matrices.

Naturally, our study has some limitations and prospects for further research. First, it is possible that not enough differing opinions were considered in this process. In further research, more experts with differentiating viewpoints could be added to the third step of the decision process, thereby considering the admission and survey of various IT and manufacturing executives with an outside-in viewpoint to verify the relevant criteria identified in this paper. Additionally, the artifact could be validated with additional applications of ML besides DSM. Second, during the semi-structured literature review, 14 articles were identified and examined to provide a criteria catalog that is as complete as possible and considers all relevant criteria considering the requirements of manufacturing SMEs. To confirm the relevance and completeness, further efforts could be made in the form of a more detailed literature review. However, based on the experience gained when applying the DSS in the context of the use case and in line with the industry experts’ feedback, we are confident to have covered the most important criteria regarding the selection of ML cloud services by manufacturing SMEs. However, it must be considered that the selected evaluation criteria and their weightings are only valid for a limited time. Due to technological advances, changing requirements in SMEs, and externally specified constraints, the criteria catalog must be iterated accordingly in future research. Third, since some criteria can only be measured on a subjective and qualitative level, the service evaluation is prone to an evaluation bias, thereby greatly influencing the decision-making and suitability of the examined cloud services. Further applicants of the presented process need to conduct the second step based on their subjective perspective or, better yet, consult external experts to avoid evaluation bias based on existing knowledge about different services and solutions. Fourth, the practical success of the presented decision process is still to some extent uncertain and heavily dependent on the quality and accuracy with which the service evaluation and the criteria weighting is performed. Therefore, further improvements and specifications of the evaluation and weighting could be added, such as developing concrete standards for each criterion, facilitating the service evaluation. Fifth, consideration of more ML-specific topics such as explainability of ML algorithms, federated learning approaches, or their energy consumption might be analyzed in future research to cover a broader spectrum of criteria [62,76,77].

7. Conclusions

In this study, a four-step DSS for ML cloud service selection is developed, applying a process based on the AHP. This study proves that SME manufacturing companies can systematically select a cloud-based machine learning service (cf. hypothesis) by using the designed artifact as an end-to-end and easy-to-understand solution. More precisely, a semi-structured literature review was used to identify relevant criteria regarding the selection of cloud-based ML services by manufacturing SMEs. These were aligned with the feedback of industry experts. The identified criteria were shortlisted in a four-level criteria catalog summarizing the relevant requirements of manufacturing SMEs regarding ML cloud service selection. Subsequently, a case study was conducted, aiming to identify a cloud-based ML service that enables the implementation of DSM measures while considering the specific needs of the corresponding company. Based on a pre-selection process, the three cloud providers Amazon Web Services (AWS), Microsoft Azure (Azure), and Google Cloud Platform (GCP), as well as their respective ML services Amazon SageMaker, Azure ML, and Google AI platform, were identified to be eligible and subsequently assessed based on the evaluation criteria following a five-point rating scale. As part of the next step, experts weighted the criteria regarding the use case following the AHP approach. Based on the experts’ findings, the local and global weights of the criteria were derived, highlighting the relevance of IT security, cost, and reliability aspects for manufacturing SMEs. Based on the results, Microsoft’s ML service was well suited for meeting the specific goals and requirements described in this use case. To achieve a more flexible and efficient energy use by applying DSM measures, contributing to more sustainable manufacturing, the investigated SME should prioritize the use of Azure ML. However, it should be noted, that this is not a generally valid recommendation. Therefore, manufacturing SMEs interested in adopting cloud-based ML services should perform the described four-step decision process on their own, considering that the system developed in this paper offers a good starting point for further practical applications.