Evaluation of Cloud Services: A Fuzzy Multicriteria Group Decision Making Method

: This paper formulates the performance evaluation of cloud services as a multicriteria group decision making problem, and presents a fuzzy multicriteria group decision making method for evaluating the performance of cloud services. Interval-valued intuitionistic fuzzy numbers are used to model the inherent subjectiveness and imprecision of the performance evaluation process. An effective algorithm is developed based on the technique for order preference by similarity to ideal solution method and the Choquet integral operator for adequately solving the performance evaluation problem. An example is presented to demonstrate the applicability of the proposed fuzzy multicriteria group decision making method for solving the multicriteria group decision making problem in real world situations.


Introduction
With the rapid development of the information and communication technologies and the increasing growing of globalization, cloud computing has evolved as a popular and universal paradigm for service-oriented computing in which computing infrastructure and solutions are delivered as a service [1].Clouds are next-generation data-storage and computing systems with virtualization as the core, enabling available technologies to interconnect and manage distributed computers where resources are dynamically provisioned on the demand [2].The use of clouds greatly helps organizations create and maintain their competitive advantages and improve their overall performance in the marketplace.It often brings organizations with numerous benefits including (a) the reduction in the cost of managing and maintaining an organization's information technology systems, (b) the increased productivity, (c) the improved collaboration inside the organization, and (d) the provision for flexibility to individual employees in their work practices [3].
With the proliferation of a range of cloud services over the Internet, efficient and accurate service discovery and selection based on user-specific requirements has become a significant challenge for decision makers in various organizations [1,3].Often, there are numerous trade-offs between different functional and non-functional requirements fulfilled by different cloud services.This makes it difficult to evaluate the overall service level of different cloud services in an objective manner such that the required quality, reliability and security of an application can be ensured.As a result, it is critical for organizations to evaluate the performance of the available cloud service alternatives so that the most suitable cloud service can be selected for implementation.
Evaluating the performance of cloud service alternatives in an organization is complex and challenging.This is due to (a) the involvement of multiple decision makers in evaluating the available alternatives with respect to multiple, often conflicting criteria, (b) the presence of subjectiveness and imprecision inherent in the human decision making process [2], and (c) the need to adequately consider the interest of multiple decision makers in a comprehensive manner.
Much research has been done on the development of numerous methods for dealing with the cloud service performance evaluation and selection problem [3,5,6].Garg et al. [3], for example, use the analytic hierarchy process method to evaluate the performance of cloud service alternatives in an organization.This process comprises three main steps including (a) problem decomposition, (b) priority judgment, and (c) priority aggregation.In the first phase, a hierarchy structure of the evaluation problem is developed for modelling the relationship between the goals of the cloud service selection, the quality of service criteria, and the service alternatives.In the second phase, pairwise comparisons are conducted to determine the relative importance of the criteria and the performance rating of the cloud service alternatives.In the third phase, the overall rankings of these cloud services alternatives across all the evaluation criteria are determined based on an effective aggregation of the criteria weightings and the performance ratings in a specific situation.
Saripalli and Pingali [5] apply the simple additive weighting (SAW) method for dealing with the cloud service performance evaluation and selection problem.A Delphi method is used to assess the relative weightings for each criterion through an expert interview.The SAW method is then employed to determine the overall performance of each alternative across all the criteria based on the ratings of each alternative for finalizing the ranking of the cloud service alternatives.
Menzel et al. [6] present an integrated method using the analytic network process (ANP) and zero-one goal programming for evaluating the performance of cloud service alternatives.The ANP is used to obtain a set of suitable weightings for the evaluation and selection criteria involved.The information obtained from the ANP is then used in the zero-one goal programming method for determining the final ranking of the available cloud services alternatives.
The methods above have shown their applicability in solving various clod services evaluation and selection problem from different perspectives under various circumstances.There are, however, some specific issues and concerns that stop them from effective use in solving this kind of problems including (a) the failure to adequately handle the various requirements of the decision makers, (b) tedious and complex mathematical computation required, and (c) cognitively very demanding on the decision makers [7][8][9].
To overcome the shortcomings of these methods as above, this paper formulates the performance evaluation of cloud services as a multicriteria group decision making problem, and presents a fuzzy multicriteria group decision making method for evaluating the performance of cloud services.Interval-valued intuitionistic fuzzy numbers are used to model the inherent subjectiveness and imprecision of the performance evaluation process.An effective algorithm is developed based on the technique for order preference by similarity to ideal solution (TOPSIS) method and the Choquet integral operator for adequately dealing with the performance evaluation problem.An example is presented to demonstrate the applicability of the proposed fuzzy multicriteria group decision making method for solving the multicriteria group decision making problem in real world situations.

The Cloud Service Performance Evaluation and Selection Problem
Cloud computing is a key driver for the industry transformation which is happening in various sectors across the world.It enables individual organizations to have access to specific information technology systems and functionalities at a much lower cost.Cloud computing allows smaller organizations to rapidly expand their operations by giving them the ability to quickly and cost-effectively roll-out new products and services, and at the same time, service their customers across the world [10].Some of the other advantages of adopting cloud computing include (a) the reduction in initial capital expenditure [11], (b) minimal management [1], (c) optimized resources utilization [12,13], and (d) improved energy efficiency [14].In addition to that, Hoberg et al. [15] point out a number of other benefits of cloud computing including increased scalability, increased agility, reduction of information technology infrastructure complexity, reduction of operations costs, and improved alignment between businesses and information technology services.
Recent predictions by Gartner [16] and Manyika et al. [17] suggest that there is an upward trend in the adoption of cloud computing and it will be a multi-billion dollar industry in the coming years.A survey by KPMG [18] reveals that 81% of organizations are either planning their initial forays, are in the early stage of implementation, or have a full implementation of cloud computing.This is due to the benefits of cloud computing adoption including the cost savings, speed of deployment, scalability and better alignment between technology and business, decreased efforts in managing technology, and environmental benefits [2,4,14].
Due to such business benefits offered by cloud computing, many organizations have started building applications on the cloud infrastructure and making their businesses agile by using flexible and elastic cloud services [4].However, moving applications and data into the cloud is not a straightforward process.This is because numerous challenges exist to leverage the full potential that cloud computing has to offer.These challenges are often related to the fact that existing applications have specific requirements and characteristics that need to be adequately meet by cloud service providers.
With the growth of public cloud services offerings, it has become increasingly difficult for cloud services customers to decide which provider(s) can fulfil their requirements for quality cloud services [1].For example, each cloud service might offers similar services at different prices and performance levels with different sets of features.While one provider might be cheap for storage services, they may be expensive for computation.Given the diversity of cloud service offerings, it is an important challenge for organizations to discover the suitable cloud providers who can satisfy their requirements.There may be trade-offs between different functional and non-functional requirements fulfilled by different cloud services providers.As a result, it is not sufficient to just discover multiple cloud services.It is important to determine the most suitable cloud service through a comprehensive performance evaluation in a specific situation [3].
The performance evaluation of available cloud service alternatives with respect to a set of specific criteria is complex [19].This is due to the presence of the multi-dimensional nature of the evaluation process and the presence of vagueness of the decision making process [20].To effectively deal with this problem, an overall evaluation of individual cloud service alternatives is desirable.
In order to adequately measure the performance of the available cloud service alternatives, it is important to firstly define the suitable criteria for ensuring that the evaluation and selection process produces an accurate and effective outcome for specific organizations.This is because not every criterion is relevant to the requirements of a specific organization in a specific circumstance [7,9].
Security is one of the major concerns for organizations when they consider moving their business to the cloud environment.Security (C1) refers to the ability of the cloud service to protect the organizational data in terms of their confidentiality and privacy.There is a great deal of uncertainty and risks about how security at network, host, applications, and data levels can be achieved.Hosting data under another organization's control is always a critical issue which requires stringent security policies employed by the cloud service provider.Avram et al. [21] and Hung et al [22] believe that the support and maintenance provided by the cloud services provider and the security of the system are the main factors to be used for the selection of the most suitable cloud service.Godse and Mulik [8] state that a positive perception of the user on the level of security and privacy of a specific cloud service positively influences the actual selection of such a service in the real world.This means that cloud service providers should fortify both their applications and networks in terms of their security in order to protect the privacy of the users and the corresponding intellectual property as these services collect and compile an increasing amount of sensitive information.Garg et al. [3] point out that there is a significant relationship between the level of the security and privacy concern and the willingness to provide personal and sensitive information.
Julisch and Hall [23] state that due to the concept of resource pooling with other clouds, the clients' data is available both to the third party cloud and the cloud in use.As a result, the security of the data is a critical component in cloud services to ensure that the data is available and access is permitted only to authorized users.These views are shared by Park and Kim [24] who state that an acceptance of the cloud services is largely affected by the security, the perceived mobility, the quality of the service, the connectedness, and the satisfaction of individual users in the evaluation and selection process.There are many different solutions offered by cloud services for addressing the needs of different organizations.This means that it is important for individual organizations to understand how their applications perform on the different clouds and whether these application deployments meet their expectations [3,5,25].Performance (C2) refers to a set of parameters for measuring the quality of the service that the cloud provides.Martens and Teuteberg [26] show that organizations need to understand how their applications perform on different clouds and whether these applications meet their expectations and requirements for achieving the competitive position in the market place.Zeng et al. [27] claim that the performance of the cloud service is important for improving the operational effectiveness of an organization.
Service accessibility and usability (C3) refers to the ease of use of a cloud service for supporting the business operations.The easier to use a cloud service is, the faster an organization can switch to it [5,20].Limam and Boutaba [28] state that the cloud service needs to enhance the accessibility to many information resources which are locked either in proprietary or inaccessible desktop applications.Quinton et al. [29] believe that the usability of a cloud service should include multiple factors such as accessibility, learnability, and operability of the cloud.
Scalability (C4) refers to the ability of a cloud service to fit a problem as the scope of that problem increases.It depends on the automatic resizing and reconfiguration of cloud resources [6].The focus here is on how the cloud service has the ability to make a good use of available resources at different workload levels to avoid an excessive delay and unproductive consumption of organizational resources.Garg et al. [3] state that the scalability of a cloud service is an important quality measure for an organization who wants to move to the cloud.This is because the costs of using a cloud service increase, particularly at peak times if the cloud does not allow an application to scale well vertically.Saripalli and Pingali [5] believe that the cloud service should have the ability to be scaled up to easily meet the demand through replication and distribution of the requests across a pool or farm of available servers in a specific situation.
Adaptability (C5) reflects on the ability of the cloud service to adjust the services based on customers' requests [1,5,21,31].Menzel et al. [6] and Karim et al. [32] state that cloud services should be able to create a pool of resources that are flexible enough to handle many different sorts of applications.Such resources can be brought online or torn down to meet the demand of the organization in a given situation.
Giving the evaluation criteria identified as above, the available cloud service alternative have to be evaluated by multiple decision makers to determine the most suitable cloud service alternative for implementation.With the multi-dimensional nature of such an evaluation problem, the use of a multicriteria group decision making methodology is appropriate and necessary.

The Fuzzy Multicriteria Group Decision Making Method
Evaluating the performance of the available cloud service alternatives with respect to multiple, usually conflicting criteria in a specific situation is always challenging due to (a) the availability of multiple cloud service alternatives, (b) the multi-dimensional nature of the decision making problem, (c) the involvement of multiple decision makers, and (d) the presence of subjectiveness and imprecision involved in the decision making process [33].To overcome these concerns, this paper presents a fuzzy multicriteria decision making method based on the fusion of several concepts including (a) the TOPSIS method, (b) the Choquet integral operator, and (c) the intuitionistic fuzzy numbers.
Modelled as a multicriteria group decision making problem, the performance evaluation of available cloud service alternatives involves in (a) discovering all the alternatives, (b) identifying the evaluation criteria, (c) assessing the alternatives' performance ratings and the criteria weightings by individual decision makers, (d) aggregating the alternative ratings and criteria weightings for producing an overall performance value for each alternative across all the criteria, and (e) selecting the best alternative in the given situation [34].
To model the subjectiveness and imprecision of the human decision making process, interval-valued intuitionistic fuzzy numbers are used by the decision maker for assigning the weightings of the evaluation criteria.Interval-valued intuitionistic fuzzy numbers [35] are the generalization of the intuitionistic fuzzy numbers.The values of the membership function and non-membership function of interval-valued intuitionistic fuzzy numbers are represented as intervals rather than exact numbers.
In many real situations, it is often difficult to define the membership grade of an element because decision makers often do not agree on the same membership grade for an element.To effectively deal with this situation, hesitant fuzzy set is introduced [34] as a generalization of fuzzy sets.
In this paper, an interval-valued intuitionistic hesitant fuzzy Choquet integral operator [37] is introduced for dealing with the hesitant fuzzy multicriteria decision making problem in an efficient and effective manner.The procedure for the adoption of the interval-valued intuitionistic hesitant fuzzy Choquet integral operator in the fuzzy multicriteria group decision making process includes the following steps: Step 1. Construct an interval-valued intuitionistic hesitant fuzzy decision matrix .) ( denotes an interval-valued intuitionistic hesitant fuzzy element, and alternative Ai is evaluated by each decision maker Dk with respect to criteria Cj.The hesitant fuzzy decision making matrix for each decision maker Zk can be represented as in (1).
Peer-reviewed version available at Algorithms 2016, 9, 84; doi:10.3390/a9040084(1) Step 2. Obtain the overall interval-valued intuitionistic hesitant fuzzy decision matrix by averaging the fuzzy assessments made by individual decision makers as given in (1).(2) Step 3. Determine the fuzzy measures of the criteria set given as in (2) as follows: Step 4. Calculate the preference of one permutation to others based on the score function S(h) by using (4).
where #h is the number of the elements in h.
where (σ(1), Step 6.The interval-valued intuitionistic hesitant fuzzy positive-ideal solution (h + ) and interval-valued intuitionistic hesitant fuzzy negative-ideal solution (h − ) are given as in ( 6) and (7).Here B and C indicate the benefit and the cost criteria respectively.

[ ][ ] (
) ) ) + α and − α represent the largest and the smallest interval-valued intuitionistic fuzzy values respectively and are denoted as follows: between the alternative Ai and the interval-valued intuitionistic hesitant fuzzy positive-ideal solution (h + ) and the interval-valued intuitionistic hesitant fuzzy negative-ideal solution (h − ) respectively by using the interval-valued hesitant fuzzy Euclidean distance [23] by using ( 8) and ( 9) respectively.
Step 8. Compute the closeness coefficient value for each alternative across all the evaluation criteria by using (10) as follows where i = 1, 2, …, n.
Step 9. Rank all the alternatives Ai (i = 1, 2, …, n) based on the descending order of the closeness coefficient values.The larger the closeness coefficient value is, the more preferred the alternative Ai.

An Example
To demonstrate the applicability of the proposed fuzzy multicriteria group decision making method above for solving the cloud service evaluation and selection problem, a problem of evaluating and selecting the most suitable cloud service in a specific situation is presented.
Company A is an e-learning content and services provider with more than 50 employees located in Taiwan.The company's main activities include e-learning content development and delivery business through direct marketing.As the e-learning content is the most precious corporate asset, content security and piracy are the company's top concerns.Company A is particularly worried about its e-learning content being pirated on the Internet as it could cause a devastating business loss to the company.Up till now, Company A still has not found a suitable Digital Rights Management solution to resolve the data security and piracy issue on the Internet.As a result, Company A is seeking for a suitable cloud service that is capable of customizing a private e-learning platform to suit the company's specific purposes.
To start with the cloud service performance evaluation process, the team has identified several cloud service alternatives and the evaluation criteria through a comprehensive investigation.Four potential cloud service alternatives and five criteria are determined for evaluating the performance of the most suitable cloud service alternative.The five most important criteria that are relevant for the performance evaluation of cloud service alternatives are used including Security (C1), Performance (C2), Service accessibility and usability (C3), Scalability (C4), and Adaptability (C5).The proposed fuzzy multicriteria decision making method presented in Section 3 is therefore used for evaluating the performance of cloud service alternatives.The steps followed are illustrated in the following: Peer-reviewed version available at Algorithms 2016, 9, 84; doi:10.3390/a9040084 Step 1. Construct an interval-valued intuitionistic hesitant fuzzy decision matrix by each decision maker.In the evaluation process, it is necessary for the decision makers to provide their own evaluation in relation to all the available alternatives.Tables 1-3 show the evaluation of each decision maker Dk for all the alternatives Ai (i = 1, 2, …, n) with respect to the criteria Cj (j = 1, 2, …, m).Step 2. The overall interval-valued intuitionistic hesitant fuzzy decision matrix is obtained by averaging the fuzzy assessments made by individual decision makers as in (1).
Step 3. The fuzzy measures of criteria C is determined as shown below by using λ=0.5.Step 4. The preference of one permutation to others is obtained below by using the score function derived in (4).
Step 5: Aggregate all interval-valued intuitionistic hesitant fuzzy elements Step 6.The interval-valued intuitionistic hesitant fuzzy positive-ideal solution (h + ) and interval-valued intuitionistic hesitant fuzzy negative-ideal solution (h − ) are given as: Step 7. The distance between the alternative Ai from the interval-valued intuitionistic hesitant fuzzy positive-ideal solution (h + ) and the interval-valued intuitionistic hesitant fuzzy negative-ideal solution (h − ) are calculated by using ( 8) - (9).Step 8.By using (10), the closeness coefficient value of each alternative can be calculated as shown in Table 4.
Step 9.The ranking of each alternative can be determined based on the closeness coefficient obtained in Table 4. Table 4 also shows that alternative A 4 is the best performing cloud service alternative, as compared to the other alternatives as it has the highest closeness coefficient value of 0.739.

Conclusion
The cloud service performance evaluation process is challenging due to (a) the availability of multiple cloud service alternatives, (b) the multi-dimensional nature of the decision problem, (c) the involvement of multiple decision makers, and (d) the presence of subjectiveness and imprecision involved in the decision making process.This paper has presented a fuzzy multicriteria group decision making method for evaluating the performance of cloud services.The inherent subjectiveness and imprecision of the evaluation process is modelled by using interval-valued intuitionistic fuzzy numbers.An effective algorithm is developed based on the TOPSIS method and the Choquet integral operator for adequately dealing with the cloud services performance evaluation problem.The result shows that the fuzzy multicriteria group decision making method is capable of effectively and efficiently solving the multicriteria group decision making problem in the real world setting.

Fig. 1
Fig. 1 The hierarchical structure of the cloud service performance evaluation problem.

Table 1 .
The interval-valued intuitionistic hesitant fuzzy decision matrix for decision maker 1.

Table 2 .
The interval-valued intuitionistic hesitant fuzzy decision matrix for decision maker 2.

Table 3 .
The interval-valued intuitionistic hesitant fuzzy decision matrix for decision maker 3.

Table 4 .
The closeness coefficient of the cloud service alternatives and their rankings.