Analysis of Complexity and Performance for Automated Deployment of a Software Environment into the Cloud

Round 1

Reviewer 1 Report

This paper provides an analytical understanding of moving to the cloud,
which is a topic that tends to be present in all enterprises that have
digitalized their activities. This includes the need to work with
software environments specific to various business domains, accessed as
services supported by various cloud providers. Besides provisioning,
other important issues to be considered for cloud services are
complexity and performance. This paper evaluates the processes to be
followed for the deployment of such a software environment in the cloud
and compares the manual and automated methods in terms of complexity. We
show that the manual deployment process is from 2 to 7 times more
complex than the automatic one, depending on the metrics applied. This
proves the importance of automation for making such a service more
accessible to enterprises, regardless of their level of technical
know-how in cloud computing. In addition, the performance is tested for
an example of an environment and the possibilities to extend to
multi-cloud are discussed.

Although the paper mainly focuses on IBM clouds, the authors also show
the differences required to ensure the deployment to other clouds like
Amazon or Azure and obtain a multi-cloud deployment platform.

My major concern is whether the authors could briefly explain how the
resource provisioning models like Amazon affect and are connected to
their modeling process. The authors show also briefly explain how the modeling process is connected to the actual resource usage process, e.g., when using cloud resources to process a type of workload in practical applications. 

Some minor comments include:

The font in the figures needs to be larger and the figures need to be
adjusted a little

Author Response

We carefully analyzed the recommendations of all our reviewers, and we found all of them very useful; they helped us improve our manuscript. The revised version contains supplementary explanations, especially in the Introduction and Discussion. There is also a minor change in the title in response to the reviewers’ comments. We attentively revised the English language and style. We used the "Track Changes" function for fragments or paragraphs added to the paper, but there were also other adjustments to respond to your recommendations. 

Our responses are available in the attached file.

Author Response File: Author Response.pdf

Reviewer 2 Report

This is an interesting research topic and the authors put a lot of effort to develop the research. However, the current version is not ready for publication. I will list a few points below to guide the author(s) to work on the article further:

• The abstract needs to be re-written, the current version does not invite the readers to read the article. It also does not fully fit with the content of the paper.
• The introduction needs to be restructured and extended. Some basic concepts and what the authors mean by them should be clearly defined and described in the introduction. What is the research problem, what are the research questions, what is the contribution of this article? What do you mean by complexity? What are the complexity metrics? What is Environment as a Service? Define them, explain the problem so readers can understand if the article is interesting for them to read or not. 
• Some sentences are just really difficult to understand, I strongly suggest proofreading. For instance, the first sentence (this is just one example, you need to read the whole article again and look at each sentence again) "In many business domains people are interested to have cloud-based solution..." What are these business domains? who are the people? what are the cloud-based solutions? Additionally, the sentence dramatically needs some work, do you mean a cloud-based solution or solutions? the continuation of the sentence is also vague "...that is easier to maintain in comparison with on-premises implementations" why is it easier? or do you really mean it is easier? (because later you argue that there are a lot of manual tasks and need for special knowledge) who says this? use references.
• Some information is presented in the wrong sections. For instance, the research methods section starts with a sentence which is the research problem really. It would have been great if you said this in the introduction and motivated the article's purpose this way.
• Complexity is a huge domain, the authors should really pay attention to using a well-known definition and instead of using only complexity in sentences explicitly tell what type of complexity they are talking about, my guess is it is software complexity but I am not sure. If they can do that early in the article then the research methodology section on complexity metrics would make more sense. There is no definition even in the related work section. 1000s of people work on the complexity from biology to computer science to engineering to mathematics. You should be clear on what you are focusing on.
• I want to underline the importance of clarity here again. For instance, you talk about Cyclomatic Complexity (CC). Where did you define this type of complexity in the article? I do not think you ever defined it really but you have presented how to calculate it on page 7. Before that, you used this term 5-8 times. How do you expect the reader to understand any of your points? You talk about what people think about CC you gave a formula about CC but you do not tell what it is. And probably this metric is one of the most important parts of your findings. This is one example, if you re-read the article with this eye, you will realize many other terms that suffer from the same. The first thing we want from an article is to be understandable for the readers of the journal. Help your readers to understand your research.
• Add a section about the limitations of the study, methodology, tools, findings, and so on. This study like all other research work is full of assumptions and limitations. I suggest listing them in one separate section and making it clear what your scope is and explaining what you have done to deal with these limitations.
• I think the EaaS is not really discussed in this article, you have used the term at the beginning but there is no definition of it, there are not enough analyses, there is not enough discussion about it either and in the conclusion you said it was the aim of the study to help non-specialist to use this approach. It just doesn't make sense. Either take this claim from the article or if you are aiming for this just explain it in more depth, clarity and show how you have done it,
• I also think that the title of the article needs to be changed. What does "deployment of a software environment into the cloud" means? "Analysis of complexity and performance metrics" sounds weird to me. Did you analyze the metric or you are assessing these metrics? The title talks about the performance and yes it has been discussed in a few places but is the performance really the focus of the article? I think the article focuses on complexity and evaluates/assesses three different types of/techniques to measure software complexity. Performance discussion needs to be clarified and motivated much better than the current way.
• The conclusions are not really enough and are quite vague. There is an analysis of the different complexity types/techniques and then it ends with concluding that a new platform is needed. Is this the conclusion of the paper? 
• The article also shortly presents a deployment platform. Is this your main contribution? I hope you understand where I am coming from. The reader is expected to read 18 pages, to come to a point where you present the automated deployment platform as the last thing and then you end the article by saying that a new platform is needed. But what was your research question? What was the answer? What did you find with this study? You should explain it in conclusion.

Author Response

We carefully analyzed the recommendations of all our reviewers, and we found all of them very useful; they helped us improve our manuscript. The revised version contains supplementary explanations, especially in the Introduction and Discussion. There is also a minor change in the title in response to the reviewers’ comments. We attentively revised the English language and style. We used the "Track Changes" function for fragments or paragraphs added to the paper, but there were also other adjustments to respond to your recommendations.    

Our answers to your comments are placed in the file attached.

Author Response File: Author Response.pdf

Reviewer 3 Report

The paper is well written and easy to read.
The authors evaluate the processes to be followed to deploy software environments in the cloud and compare the manual and automated methods in terms of complexity. They claim to show that the manual deployment process is from 2 to 7 times more complex than the automatic one, depending on the metrics applied. More strongly, they claim to prove the importance of automation for making such a service more accessible to enterprises, regardless of their level of technical know-how in cloud computing.

In fact, they do what they claim. However, there are some issues to be considered, and I pointed out some of them that I elected as important. First, by comparing manual and automated deployment processes presented in the paper, the difference is evident in task details. The main point here is the weight associated with each task, and consequently, the weights used to "prove" that the complexity of human tasks to automated approach is better. So what? Any automated process is proposed to facilitate the human task, but which are the consequences? I would like to see a deep discussion on that.

The second point is the shallow evaluation presented. Both CPU and memory consumptions for the Mongo and WebGME containers are presented, but the authors do not discuss how they were built. Also, the claimed evaluation was performed by only two users. These are important points to be considered and demand further discussion.

 

Author Response

We carefully analyzed the recommendations of all our reviewers, and we found all of them very useful; they helped us improve our manuscript. The revised version contains supplementary explanations, especially in the Introduction and Discussion. There is also a minor change in the title in response to the reviewers’ comments. We attentively revised the English language and style. We used the "Track Changes" function for fragments or paragraphs added to the paper, but there were also other adjustments to respond to your recommendations.    

Our answers to your comments are placed in the file attached.

Author Response File: Author Response.pdf

Round 2

Reviewer 1 Report

I am ok with the revisions.