Although cloud computing has appeared since the early 2000s, many organizations were reluctant to implement it due to the technology’s lack of knowledge and trust [1
]. However, the cloud computing market has experienced rapid growth in recent years thanks to the impressive development of technology and the apparent ease of use, scalability, and security from the users. Due to the advantages of cloud computing, more and more public and private sector organizations choose to test cloud workloads and even migrate everything to the cloud [2
The COVID-19 pandemic was one of the reasons that boosted the increasing demand for cloud computing services. The coronavirus pandemic led to the sudden shutdown of almost every sector in many countries all over the world. Offices, business enterprises, and schools have been forced to shut down the offline channels and implement the online channel. People work from home; students participate in online classes; enterprises do business via the internet. That has resulted in greater demand for cloud solutions. In 2020, the market size of cloud computing was US$
219.0 billion, with a growth of 13.7% compared to the average year-on-year growth during 2017–2019, and it is projected to reach $
250 billion in 2021 and $
791 billion in 2028 [3
The advent of cutting-edge technologies such as artificial intelligence (AI) and machine learning (ML), as well as the increased investment by countries such as China, the United Kingdom, India, and the United States, are driving the growth of the cloud computing market. North America is the biggest market of cloud computing, with the market size reaching US $
78.28 billion in 2020, which is an increase of US $
8.69 billion, equal to 12.49% [3
]. This market is forecasted to account for about 40% of the global cloud computing market [3
As mentioned above, more and more organizations and firms have used cloud computing thanks to its benefits, such as decreasing operating costs and improving the flexibility of strategic decision making, thereby increasing the company’s lifespan and resilience [4
]. In addition to benefiting firms, the cloud computing sector also has an outstanding contribution to national economic development. In 2012, cloud computing contributed around US $
165 billion (1.04%) of added value to the US gross domestic product (GDP). It also supported nearly 1.7 million American jobs. This sector is expected to increase US GDP. In the next ten years, it will grow by approximately US $
2 trillion [5
]. This sector was found to contribute a total of €763 billion in added to value five European countries—Spain, France, Italy, Germany, and the United Kingdom—from 2010 to 2015 and created almost 2.4 million jobs during this period by promoting the expansion of an existing business, doing new business, reducing costs and indirect impact [5
]. These benefits mainly come from developing existing businesses, launching new businesses, cost savings, and side effects. The study of Deloitte [6
] in 2018 revealed that Google cloud services had a meaningful impact on the economic productivity of 14 selected countries, between $
300 million and $
1.2 billion in large countries and $
100 million to $
600 million in medium-sized countries. Moreover, cloud computing is also evidenced to reduce energy consumption [7
In the new era of communications, the benefits of cloud technology for business and economic development are increasingly obvious. Hence, the growth of this sector is very important. Therefore, the efficiency performance of cloud computing providers captured great attention. Thus, this study will focus on the performance efficiency of the top 10 cloud computing providers in the United States market—one of the biggest markets in terms of cloud computing—by applying the two-stage data envelopment analysis (DEA) model, which included the super slack-based measure model (Super-SBM) to evaluate the performance during the past period from 2017 to 2020 and the resampling past–present–future model to predict the efficiency in the future period from 2021 to 2024.
The rest of the research is structured as follows: Section 2
contains a detailed review of the literature; Section 3
introduces the research methodology; Section 4
introduces the results analysis, and the last section contains the conclusions and discussion.
2. Literature Review
There are different methods for analyzing the performance, such as the ratio analysis, the performance pyramid, the analytic hierarchy process (AHP), the stochastic frontier analysis (SFA), variable factor productivity (VFP), and the data envelopment analysis (DEA). In the past, the ratio analysis was used as the favored instrument for measuring the business performance through standard ratios groups such as activities ratios, profitable ratios, liquid ratios, etc. The ratio method is attractive because of its simple methodological perspective, and the results of ratio analysis are easy to interpret. However, this method has several notable limitations. The first notable disadvantage of ratio analysis is that it requires or assumes that all the decision-making units (DMUs) operate under conditions of similar return of scale [8
]. Another limitation is that each ratio group focuses on the different activities of the business organization, not the whole operation performance [9
Another popular method is the performance pyramid, which can avoid the limitation of the ratio method. A performance pyramid is an integrated performance method that can capture multiple perspectives [10
]. Each side of the pyramid represents a hierarchical perspective of success factors, controls, and process drivers. This model pays attention to the importance of influencing internal and external factors of the implementation and allows the construction of a performance monitoring system in the pyramid floors, starting with vision organization, using both financial and non-financial indicators [11
]. However, it takes a long time, and the results may not be good for the organization in any way. The model also restricts the leadership from seeing what it highlights and leaves no room for other decisions [12
The limitation of the ratio analysis method and the performance pyramid led to the application of more sophisticated methodologies [8
]. There are two basic approaches, the parametric and nonparametric ones, to measure efficiency parametric frontier models and nonparametric methods of efficiency analysis, which have become dominant in the field of efficiency analysis [13
]. Parametric and nonparametric approaches have been widely applied in measuring efficiency and achieved highly correlated results in most cases [14
]. In comparison, the nonparametric method is more straightforward than the parametric approach. Thus, its application has increased in recent decades, which is applied in measuring the efficiency of diversified management fields thanks to its various models that suit many different requirements of the researcher [15
One of the most common nonparametric methods is data envelopment analysis (DEA). DEA is a mathematical technique that uses linear programming strategies to convert inputs into outputs to estimate the overall performance of an organization or similar product [13
]. Since Charnes, Cooper, and Rhodes [16
] first introduced the DEA-CCR model in 1978, many different DEA models have been established to deal with the requirements of researchers, such as the Window model that can deal with the small set of DMUs developed by Charnes et al. [17
] in 1984 and the slack-based model (SBM) that can deal with undesirable output introduced by Tone [18
] in 2003. DEA was quickly regarded as a cutting-edge performance measurement instrument. Since then, a vast and substantial number of articles have appeared, including key theoretical discoveries and an enormous number of works on DEA applications, both in the public and private sectors, to evaluate the efficiency and productivity of their activities [19
]. For example, Zhang et al. [20
] used the slacks-based measure DEA model to examine the environmental efficiency of the 16 listed cement enterprises in China from 2008 to 2013. Xia et al. [21
] evaluated the environmental efficiency of China’s mining sector from 2007 to 2016 by the meta-frontier slacks-based measure method. The environmental performance of the top 20 industrial countries was estimated by using the data envelopment analysis model by Iqbal et al. [22
]. In another research, Behera et al. [23
] calculated the productivity change of the coal-based power plants in India by the Malmquist productivity index. Park et al. [24
] applied the DEA window and SBM DEA model to estimate the operational efficiency of the coastal ferry sector in South Korea. The application of DEA appeared to be popular in the high-tech sector. For example, Zhang et al. [25
] evaluated the efficiency of the hi-tech industry in China by applying a multi-activity network DEA. Wang et al. [26
] applied two-stage DEA in the efficiency evaluation of China’s high-tech industry. Bai et al. [27
] used the dynamic network SBM model in DEA to assess the efficiency of the high-tech zone in China in the post-financial crisis era. Li et al. [28
] measured the innovation efficiency of the semiconductor industry in China with the application of the new DEA model. Healthcare represents the main application area for DEA. The study of Kohl et al. [29
] reported that there were about 262 papers of DEA application in healthcare focusing on hospitals from 2005 to 2016. The literature proves that the DEA is a powerful method in measuring the efficiency of DMU performance in every sector.
In terms of forecasting methodology, the Grey system theory is considered to be one of the most widely used methods due to its computational efficiency [30
]. The Grey model has been applied widely to forecast the outputs or productions of different fields. Numerous previous studies that applied the Grey model can be found in the literature [31
]. Thanks to the advantages of the DEA and the Grey model, these two methods are usually combined to measure and forecast performance efficiency. The study of Wang et al. [35
] presented an integrated approach by combining the super slack-based model and the Grey model GM(1,1) to evaluate and predict the energy consumption efficiency of 25 different countries in the world. Chen and Chen [36
] applied the Malmquist productivity index (MPI) and the Grey model GM(1,1) to investigate and forecast the operation performance of the Taiwanese wafer fabrication firms. Carboni and Russu [37
] used a combination of the DEA with the Grey model to measure and predict the local economy and environmental efficiency in Italy. Wang et al. [38
] assessed the performance of major Asian airline companies by combining the Grey model GM(1,1) and the DEA Window model. However, this integrated method required sophisticated steps and a large database, which is necessary to make a base for forecasting by the Grey method.
In 2016, Tone [39
] proposed the model in DEA based on resampling that enables the researchers to evaluate and forecast the efficiency at the same time. In previous research, Wang et al. [40
] applied this DEA resampling model to estimate the macroeconomic performance of 17 economies (12 Asian developing countries and five developed countries) from 2013 to 2020. The research shows that DEA resampling is an effective model for predicting and measuring the performance of multiple decision-makers. Chiu et al. [41
] introduced the combination of the merger potential gains model and the resample past–present–future model to evaluate the performance of the financial industry in Taiwan. Wang et al. [42
] applied the newly developed DEA resampling model to evaluate the profitability of Vietnamese commercial banks. The results of these studies supported the feasible application of the new DEA method proposed by Tone [39
] not only in measuring but also in forecasting the efficiency of decision making without the need of combining two separate methodologies, such as DEA and the Grey model.
In recent years, measuring the efficiency of cloud computing service providers has newly captured researchers’ attention due to the significant development of technology and the consequence of the COVID-19 pandemic. For example, Azaidi et al. [43
] in 2019 proposed the network DEA method as a tool to evaluate the efficiency of the 18 cloud service providers. The results of this research indicated that DEA could effectively measure the efficiency of this industry. Another study by Azadi et al. [44
] in 2021 also assessed the sustainability of 30 cloud computing service providers with the application of DEA. One considerable research on this topic is the work conducted by Subirats and Guitart [45
]. They evaluated and forecasted the energy efficiency of cloud computing platforms. However, most of the related studies found in the literature focused on analyzing the benefits and the challenges of adopting cloud computing in organizations, the efficiency of cloud computing [46
], and many other studies concentrating on the advantages, difficulties, and opportunities related to cloud computing can be found in the literature. On the contrary, there are very few studies on the utility of DEA in cloud computing.
An increasing quantity of businesses and organizations around the world use cloud computing offerings to enhance their performance inside the aggressive marketplace. Measuring the performance helps to understand these providers’ behavior and provides a guideline to improve performance. However, it is challenging to evaluate service industry performance such as cloud computing because of the unbalancing service quality and resource usage, which is closely related to game theory. Game theory is known as the practice of simulating strategic interaction between two or more players [50
]. Game theory is used in many different fields to analyze players’ strategic decisions in two cases: cooperation and competition [50
]. In the technology field, Teng and Magoulès [51
] in 2010 tackled the resource allocation issue in cloud computing by game theory. Riahi and Riahi [52
] in 2018, Ghosh [53
] in 2020 evaluated the energy efficiency in wireless systems by applying game theory. Azadi et al. [43
], in 2019, measured the efficiency of cloud service providers using DEA with a game theory base.
4. Results Analysis
4.1. Efficiency Evaluation during the Past Period from 2017 to 2020
In this section, the Super-SBM model is applied to compute the efficiency scores and rank the efficiency of 10 DMUs. Table 4
presents the obtained scores and ranking of these DMUs over the period of 2017 to 2020, along with the illustration in Figure 2
As observed, the efficiency scores of each DMU experienced different trends. Some DMUs remained unchanged and remained stable, such as C1-Amazon and C4-Salesforce, while the rest of the DMUs showed a notable fluctuation in efficiency scores over the period of 2017 to 2020. The results divided these ten DMUs into three groups: the stable group, the increasing group, and the decreasing group.
The efficiency score of C1-Amazon remained unchanged over four years with a score of 1. The efficiency score of C4-Salesforce slightly changed over the period. However, the change was insignificant, approximately 1% over a 4-year period and a lower than 6% year-to-year change. It was also noted that C4-Salesforce had the highest score among the ten DMUs and ranked at the first position for all four years from 2017 to 2020. From the results, it can be seen that these two DMUs controlled well the inputs and outputs over the observed period.
While the scores of C1-Amazon and C4-Salesforce were stable, the efficiency scores of the remaining eight DMUs significantly fluctuated. C2-Alphabet had a fluctuation in score with a decreasing trend from 1.56 in 2017 to 1.35 in 2020, approximately a 13.5% decrease over the period, which led to a drop in ranking from the second position among ten to the third position. The efficiency scores of C3-Microsoft, on the other hand, significantly increased with the drastic change of 43.5% from 1.08 in 2017 and ranked at the 6th position to the score of 1.55 and become the second most efficient DMU in 2020. The scores of C5-ServiceNow experienced an increasing trend of 9.4% over the period of 2017 to 2020, with the score changing from 0.96 in 2017 to 1.05 in 2020. It should be noted that C5-ServiceNow was inefficient in 2017 and 2018; then, it turned to be efficient in 2019 with a score of 1.0 and increased to 1.05 in 2020, which thanked both the increase in gross profit and the decrease in the cost of goods sold and operating expenses. C7-Dropbox and C8-Shopify experienced the same trend with C3-Microsoft and C5-ServiceNow, with the scores increasing over a 4-year period. Scores of C7-Dropbox increased from 0.85 in 2017 to 1.01 in 2020, approximately 18.8%, and the score of C8-Shopify improved from 0.93 to 1.11, about 19.4%. These two DMUs also turned from inefficient to efficient.
On the other hand, C6-Splunk, C9-Atlassian, and C10-Twilio went through the decreasing trend of 41.3%, 10.4%, and 51.8%, respectively, due to the higher cost of revenue and operating cost and also the lower gross profit and total revenue. Among these three, C6-Splunk and C10-Twilio turned from efficient to inefficient.
4.2. The Choice of Replicas Illustration
This stage demonstrates the efficiency score of ten DMUs in 2020 and then compares the actual scores of DMUs in 2020 obtained by 5000 replicas with those obtained by 500 replicas. The variation of scores by using different replicas within a 95% confidence interval was obtained by applying the past–present model, and the results are presented in Table 5
. In the research, most of the results of 5000 replicas and 500 replicas were statistically negligibly small except for the scores of C10-Twilio with the difference of 0.31. Thus, 5000 replicas can be applied in the next stage.
Before conducting further analysis, the correlation analysis is assessed to ensure the appropriate inputs and outputs. The value of the correlation coefficient is always between (−1) and (+1), and if the value is near (±1), it means that there is a stronger linear relationship between factors. The results of correlation analysis reported in Table 6
confirm that the selected inputs and outputs in this study are suitable, which is proved by the high correlation between inputs and outputs.
4.3. Illustration of Selecting the Prediction Model
This section illustrates the forecasted efficiency of ten DMUs through three different prediction models in the past–present–future framework in DEA. The most appropriate prediction model is selected based on the comparison of forecasted results obtained by each prediction model with the actual efficiency scores (for the year 2020 only).
The Trend, Lucas weight, and the hybrid are three considered prediction models in this stage. The results of efficiency scores obtained by these models and the actual efficiency score of all DMUs in 2020 are compared. The comparison indicates that the results of these models were consistent with all actual efficiency of all ten DMUs were within the 95% confident interval of all three prediction models. However, the average difference between forecasted and actual scores is 24.7% with the trend prediction, 10.7% with the Lucas weight prediction, and 12.4% with the hybrid model. Therefore, the Lucas weight model is the most appropriate prediction model in this study, and it would be used to get the efficiency scores of ten DMUs in the future period from 2021 to 2024. Table 7
exposes the difference between the actual and predicted results in 2020 (predicted by the Lucas weight model).
4.4. Future Forecast Efficiency Evaluation over the Future Period of 2021 to 2024
The efficiency scores of ten DMUs in the future period of 2021 to 2024 are predicted by using the past–present–future model with the Lucas weight prediction. The obtained scores are displayed in Table 8
, along with the bar chart illustration in Figure 3
The obtained scores in 4 years tend to be a stable trend corresponding with the insignificant changes in efficiency scores of all DMUs from 2021 to 2024. From 2021 to 2024, ten DMUs can be categorized into two different groups: inefficient and efficient groups. The inefficient group includes C6-Splunk and C7-Dropbox, with the corresponding efficiency score smaller than 1. On the other hand, eight DMUs are included in the efficient group as the obtained efficiency scores of those are equal and higher than 1.
It is also notable that the average efficiency score from 2021 to 2024 is slightly higher than that from 2017 to 2020. However, the average efficiency score of C2-Alphabet and C6-Splunk from 2021 to 2023 decreased compared to those during 2017 to 2020, as can be seen in Figure 4
4.5. Efficiency Improvement over the Research Period of 2017 to 2024
In this section, the Malmquist productivity index is applied to investigate the efficiency changes over the period of 2017 to 2024. The change in efficiency is categorized into three groups: the catch-up index measuring the efficiency changes of each DMU, frontier-shift measuring the movement of the efficiency frontier over two periods, and the MPI, which is the production of catch-up and frontier-shift. Table 9
summarizes the obtained results.
According to the catch-up scores in Table 9
, three DMUs (C4-Salesforce, C6-Splunk, and C10-Twilio) showed a weakening in efficiency, with a catch-up score lower than 1. On the contrary, five DMUs (C3-Microsoft, C5-ServiceNow, C7-Dropbox, C8-Shopify, and C9-Atlassian) demonstrated progress from 2017 to 2024 with a catch-up score above 1. The rest two DMUs (C1-Amazon and C2-Alphabet) displayed no changes in the degree of their efforts attained for improving their efficiency. The results of the frontier-shift indicated the technological regress of C8-Shopify, C9-Atlassian, and C10-Twilio. The rest of the seven DMUs experienced technological progress with the corresponding frontier-shift score higher than 1. MPI is the production of catch-up and frontier-shift represents the total improvement over two time periods. The results in Table 9
indicated the efficiency improvement of seven DMUs from C1 to C7 with the MPI higher than 1. On the other hand, three DMUs (C8-Shopify, C9-Atlassian, and C10-Twilio) showed the regress with the MPI lower than 1.
5. Conclusions and Discussion
The rapid growth in the cloud computing sector, along with its importance, measuring and forecasting the performance of cloud computing providers was considered to be very necessary. Therefore, this study applied the Super-SBM and the past–present–future model in DEA based on the resampling technique to evaluate and predict the performance efficiency of ten cloud computing providers in the United States of America, which is one of the most developing markets in terms of cloud computing services.
After forming the research framework, the first step was taken is to select the appropriate indicators. Since there was no related study measuring the efficiency of cloud computing providers found in the literature, the inputs and outputs of this study were selected based on reviewing the previous studies that measured the performance of service providers.
The second step was to measure the performance efficiency of ten selected providers over the past and present period (2017–2020) by applying the Super-SBM. This step found that 30% of cloud computing service providers were inefficient due to the problem in controlling the cost of goods sold, operating cost, and the capacity to generate revenues of these providers. However, when observing the whole ten providers as one, the results indicated that the cloud computing sector was efficient with an average efficiency score of 1.19 over the period of 2017 to 2020. During this period, it was also notable that most of the providers fluctuated in terms of efficiency with the corresponding unstable efficiency scores.
After assessing the performance over the past and present periods, the efficiency in the future was predicted by applying the past–present–future model in DEA that was based on the resampling method. Since accuracy was the most severe concern in forecasting, choosing an appropriate number of replicas and a correct prediction model was considered first in this step. In order to ensure the right choice of the number of resampling, 5000 replicas and 500 replicas were conducted. The results of these two were similar in most cases. However, the results of 5000 replicas and 500 replicas in the case of C10 showed a significant difference. Thus, the results of 5000 replicas were chosen. The next step was to identify the most accurate prediction model among the three available models: trend, Lucas weight, and the hybrid model. The accuracy of these three models was checked by comparing the efficiency scores calculated from each prediction model with the actual scores obtained by the Super-SBM model. The one with the smallest forecast-actual ratio would be selected. After the comparison, the Lucas weight model was the selected prediction model with the lowest difference between the forecast and actual score (10.7%). The final step was to forecast the efficiency of these ten providers by using the Lucas weight prediction model. The obtained results showed stable efficiency over the future period of 2021 to 2024. From 2021 to 2024, all providers were forecasted to perform more stably than their performance from 2017 to 2020, with the corresponding efficiency scores slightly changed. It was also noted that the average efficiency score of ten providers in the future would be slightly lower than that over the past period from 2017 to 2020. This result might be explained by the significantly increasing demand for cloud computing services in 2019 and 2020 due to the COVID-19 pandemic.
In the next step, the DEA Malmquist productivity index was used to investigate the efficiency improvement during the research period of 2017 to 2024. The results of MPI revealed that during this research period, 70% of the selected providers showed efficiency progress with an MPI value higher than 1.
The results of this study provided valuable information and practical implications to the decision-makers. The measure of the past and present performance helped better understand how efficient the firm was and how it compared to other providers in the same field. The result of forecasted efficiency helps a firm to have an insight picture of how it would perform in the future. That can help the firm make the appropriate policies and strategies to be more efficient and gain competitive advantages. Additionally, this study will fill the gap in the literature as the first study that measures and predicts the performance efficiency of cloud computing service providers, which will provide a helpful reference for future studies. However, there are still many things that need to be done in this research area. For example, this study only focused on measuring the performance in general without in detail analyzing the performance of each provider. This study applied the new approach of DEA to measure and forecast efficiency at the same time. Thus, further research can apply another forecasting technique to compare the accuracy of the forecast model. Finally, economic efficiency is not the only criterion that needs to be considered regarding the performance of the decision-making unit. In fact, there are two major issues in the cloud computing industry related to how to develop services with high levels of quality of service and maximize the benefit for the service providers. To answer these two questions, the general evaluation of the economic efficiency of cloud computing providers is critical, but there is not enough. In this study, the performance efficiency was evaluated based on the assumption of the non-cooperative game among cloud computing providers in which providers selfishly choose the best strategy to maximize their payoff. However, it will be more practical if the study can analyze the cooperative behavior and conflict between providers based on the theoretical tool-game theory. Further research should focus on the resource allocation problems which arise in the cloud computing industry by applying non-cooperative and cooperative game theory.