Exploring the Benefits of Combining DevOps and Agile
Round 1
Reviewer 1 Report
Dear authors, thanks for reporting about your study. Please let me provide some feedback / suggestions.
The article "Exploring the Benefits of Combining DevOps and Agile".
has a good background about the process of development of software and the combined adoption of Agile and DevOps .
The study aims to explore the benefits of the combined adoption of both models with a qualitative methodology applied to 12 case studies from international software engineering companies.
Relevance: The topic is relevant and the field is currently still limited in its coverage.
In the background or literature review section of the article, the concepts and relationships of the methodologies are explained. However, the abstract references the waterfall model, a model that is not then mentioned in the rest of the article. I also miss references to the iterative development model (Craig Larman).
Methodology: I believe the methodology can be more detailed. Even though there is a Figure that shows the process that is involved in the analysis, it is not clear how you obtained the data in Table 3 from the data in Table 2. The relationship between indexes has different meanings in the cited case studies.
The original data analyzed comes from twelve case studies. These cases correspond to reports and press releases from commercial vendors of reference in this area. There is no reference to where this data comes from.
+ In Table 2, in the row description: does this come from the press releases or is it a summary made by the authors from the press releases? If so, how can you ensure that there is no bias in the data?
Results:
Table 3 presents the identification process of the themes associated with each case study. Each theme is identified by the acronym "BF" and a number is associated to differentiate each benefit. A brief description of how each benefit is understood in the case studies is also included.
How can you ensure that there is no bias in the analysis?
I.E. the benefit BF2, with the index - automation- has three different meanings, and it's not clear to me how these definitions were obtained.
Considering the limitation of the analysis (secondary source of information, no interviews, no metrics), it's important to establish the relationship of the data and their results and establish the limitations.
(BF2) Automation: the combined development and production process becomes more automated to meet market needs.
(BF2) Automation: continuous delivery and integration combined with fast releases lead to the automation of activities.
(BF2) Automation: contribution for the implementation of agile fluency model which focus on value, transparency, and alignment.
Maybe it would be worth establishing the limitation of the study, and the impact of this limitation, in a subsection correlating to future work.
I believe that a stronger case can be made if the methodology is detailed with the steps and considerations in the analysis of data. Considering the N and the source of information, it is important to note that it could be too early to generalize since more metrics are needed to characterize each company and the implementation of the process in their business.
Conclusions:
Please add a subsection with limitations and future work that gives importance to this content in the text.
Other aspects to consider:
Please note that when you use an abbreviation that you must write the meaning after the first appearance. I.E. DevOps (Development and Operations)
In line 155, "The application of DevOps still must deal with some problems and concerns that can limit its use". I find that it is important to summarize what the problems are and compare them with traditional methodologies.
In line 164 "There is an obvious relation between DevOps and Agile methods in software development." Avoid the use of the word obvious; what is obvious for you may not be for the readers.
In line 168 "DevOps = Agile + Lean + ITMS" , correct to: DevOps = Agile + Lean + IT service management (ITSM).
Figure 1: it is blurry.
Author Response
We appreciate the review suggestions and comments received by the reviewer. These elements are key to improving the final quality of the manuscript. Below we respond to each issue raised.
Review #1
Dear authors, thanks for reporting about your study. Please let me provide some feedback / suggestions.
The article "Exploring the Benefits of Combining DevOps and Agile".
has a good background about the process of development of software and the combined adoption of Agile and DevOps.
The study aims to explore the benefits of the combined adoption of both models with a qualitative methodology applied to 12 case studies from international software engineering companies.
Author’s response: Thank you very much for your constructive feedback and improvement suggestions and to have provided a good overview about the main objectives of this study.
Relevance: The topic is relevant and the field is currently still limited in its coverage.
In the background or literature review section of the article, the concepts and relationships of the methodologies are explained. However, the abstract references the waterfall model, a model that is not then mentioned in the rest of the article. I also miss references to the iterative development model (Craig Larman).
Author’s response: We agree with the reviewer that it is important to introduce the waterfall model in the literature review and explain the iterative development as proposed by Craig Larman. Accordingly, we have included the following new reference:
Larman, C.; Basili, V.R. Iterative and Incremental Development: A Brief History. Comp 2003, 36, 47-56. https://doi.ieeecomputersociety.org/10.1109/MC.2003.1204375
Methodology: I believe the methodology can be more detailed. Even though there is a Figure that shows the process that is involved in the analysis, it is not clear how you obtained the data in Table 3 from the data in Table 2. The relationship between indexes has different meanings in the cited case studies.
Author’s response: Thanks for the opportunity to better describe the relation between the data showed in Table 2 and Table 3. The twelve case study reports associated with each company presented in Table 2 have been thoroughly read and each identified benefit has been assigned a unique identifier. Each theme is identified by the acronym "BF" and a number is associated to differentiate each benefit. Common themes have the same acronym. A brief description of how each benefit is understood in the case studies is also included. Table 3 presents the identification process of the themes associated with each case study. Table 3 shows the themes for all the case studies mentioned in Table 2. Accordingly, we have restructured the first paragraph of the results section to describe this connection between both tables.
The original data analyzed comes from twelve case studies. These cases correspond to reports and press releases from commercial vendors of reference in this area. There is no reference to where this data comes from.
+ In Table 2, in the row description: does this come from the press releases or is it a summary made by the authors from the press releases? If so, how can you ensure that there is no bias in the data?
I believe that a stronger case can be made if the methodology is detailed with the steps and considerations in the analysis of data. Considering the N and the source of information, it is important to note that it could be too early to generalize since more metrics are needed to characterize each company and the implementation of the process in their business.
Author’s response: Thanks for the opportunity to clarify this point. The data was extracted from press releases of software development companies that operate in the international market. A brief description of each of these companies is given in Table 2. The data are from secondary sources and the authors have not made any changes to the press releases that represent the view of each manufacturer. No summarization of the press releases has been made. Consequently, the thematic analysis performed includes the entire content of the press release with the aim of finding the benefits of the combined adoption of agile and DevOps. It cannot be guaranteed that there is no risk of bias since the identified benefits come from press-releases from commercial companies in the area and that have commercial goals in the market. However, to minimize this risk the external and internal validity mechanisms were used. As external validity was considered multiple case studies of companies in different geographic areas and as internal validity was used the same identifier to associate similar benefits between the case studies.
Results:
Table 3 presents the identification process of the themes associated with each case study. Each theme is identified by the acronym "BF" and a number is associated to differentiate each benefit. A brief description of how each benefit is understood in the case studies is also included.
Author’s response: Thank you very much for your positive feedback regarding the process of presenting the results of this study.
How can you ensure that there is no bias in the analysis?
I.E. the benefit BF2, with the index - automation- has three different meanings, and it's not clear to me how these definitions were obtained.
Author’s response: Thanks for the opportunity to clarify this point. How each benefit is viewed in each case study is necessarily distinct. Automation does not have a single definition, and it is not the goal of this study to find a unique definition to describe it. On the contrary, the richness of a qualitative analysis is to understand how these benefits are perceived by each organization. For this reason, we leave a brief description of the perception of each benefit in the respective case studies. For example, we look at the benefit related to automation. This is a benefit common to eight case studies although its perception in each case study is not necessarily the same. Let's see:
In CS1: (BF2) Automation: the combined development and production process be-comes more automated to meet market needs
In CS2: (BF2) Automation: continuous delivery and integration combined with fast releases lead to the automation of activities
In CS5: (BF2) Automation: contribution for the implementation of agile fluency model which focus on value, transparency, and alignment
In CS7: (BF2) Automation: increasing code size and complexity encourages process automation
In CS9: (BF2) Automation: increase speed and agility to attend continuous requirements changes
In CS10: (BF2) Automation: implementation of a paradigm based in continuous integration, continuous delivery, and continuous deployment
In CS11: (BF2): Automation: shorten the development cycle by promoting the automation of repetitive tasks
In CS12: (BF2) Automation: better performance when compared against on-premise DevOps automation. Furthermore, it contributes to eliminates human errors
We described better our approach in the methodology section to explain the relevance to keep the vision of the benefits presented in each case study.
Considering the limitation of the analysis (secondary source of information, no interviews, no metrics), it's important to establish the relationship of the data and their results and establish the limitations.
(BF2) Automation: the combined development and production process becomes more automated to meet market needs.
(BF2) Automation: continuous delivery and integration combined with fast releases lead to the automation of activities.
(BF2) Automation: contribution for the implementation of agile fluency model which focus on value, transparency, and alignment.
Maybe it would be worth establishing the limitation of the study, and the impact of this limitation, in a subsection correlating to future work.
Author’s response: We have clarified the adopted methodology in the manuscript by describing the approach of extracting, identifying, and analyzing the different themes that corresponds to the benefits by combining DevOps and Agile. We have followed your recommendation and include a new sub-section in the conclusions section that address the main limitations of this study and presents some relevant future research directions.
Conclusions:
Please add a subsection with limitations and future work that gives importance to this content in the text.
Author’s response: Thanks for your suggestion. Accordingly, we have included the 6.1 subsection.
Other aspects to consider:
Please note that when you use an abbreviation that you must write the meaning after the first appearance. I.E. DevOps (Development and Operations)
In line 155, "The application of DevOps still must deal with some problems and concerns that can limit its use". I find that it is important to summarize what the problems are and compare them with traditional methodologies.
In line 164 "There is an obvious relation between DevOps and Agile methods in software development." Avoid the use of the word obvious; what is obvious for you may not be for the readers.
In line 168 "DevOps = Agile + Lean + ITMS" , correct to: DevOps = Agile + Lean + IT service management (ITSM).
Figure 1: it is blurry.
Author’s response: Thanks for your pertinent observations. We have performed all these corrections in the manuscript. Additionally, the quality of Figure 1 was improved using the AI Image Enhancer software.
Author Response File: 
Author Response.docx
Reviewer 2 Report
The paper is focused on exploring the benefits of combining and DevOps. The authors summarised known facts and described current trends in software engineering. The topic is explained clearly, with enough relevant references.
It is not so usual to divide the abstract into parts using colons. The abstract does not emphasise what the novelty of the paper is. The authors identified the benefits already known when Agile and DevOps came to the market.
I consider the introduction section a good university textbook that clearly introduces the students to the current software engineering topic. I can't entirely agree with the authors about the research gap they found. In my opinion, the methodology used in this paper has not brought any new findings of the process of combining DevOps and Agile.
The authors used qualitative methodology to explore the benefits of the combined adoption of DevOps and Agile. This methodology is very simple. I suggest using the more robust methodology and applying it to the large dataset of the case studies.
The authors provide quite a detailed discussion about the findings. I appreciate that they also tried to identify the weaknesses of the study.
Please, let me summarise the strengths and weaknesses of the paper:
Strengths
- clear writing style suitable for the students of introductory courses of software engineering,
- using relevant references,
- application of simple and very flexible thematic analysis,
- discussion about the results.
Weaknesses
- the authors did not explain the novelty and uniqueness of the paper insufficient details,
- the authors analysed twelve case studies, which they did not describe in enough detail,
- the quality of these case studies is questionable; the authors mentioned that the case studies could represent PR or vendor-specific press releases, which prefer positive views of the phenomenons they provide,
- the applies methodology is straightforward,
- the authors identified twelve benefits, while some of them were mentioned very rarely between the case studies,
- the concussion can not be generalised and requires more sophisticated research and methodology,
- I recommend that authors try to publish this study at the conference and improve it after receiving feedback from the large community.
Finally, I recommend the authors improve the overall quality and novelty of the paper using a more robust methodology for qualitative and/or quantitative research, find more case studies without the marketing or PR background, realise systematic survey between the software companies, which tried to implement DevOps and Agile and use some statistical methods to be able to generalise the results.
Author Response
We appreciate the review suggestions and comments received by the reviewer. These elements are key to improving the final quality of the manuscript. Below we respond to each issue raised.
Review #2
The paper is focused on exploring the benefits of combining and DevOps. The authors summarised known facts and described current trends in software engineering. The topic is explained clearly, with enough relevant references.
Author’s response: Thank you very much for your positive feedback and suggestion to improve the final quality of this manuscript.
It is not so usual to divide the abstract into parts using colons. The abstract does not emphasise what the novelty of the paper is. The authors identified the benefits already known when Agile and DevOps came to the market.
Author’s response: Thank you very much for your suggestion. We have restructured the abstract to avoid its division into parts. We have also presented the novelty of this study. The novelty of this study lies in the systematization of the benefits of the combined adoption of Agile and DevOps considering multiple perspectives of the software engineering business environment.
I consider the introduction section a good university textbook that clearly introduces the students to the current software engineering topic. I can't entirely agree with the authors about the research gap they found. In my opinion, the methodology used in this paper has not brought any new findings of the process of combining DevOps and Agile.
Author’s response: Thank you very much for your opinion regarding the relevance of the introduction section. We would like to clarify the research gap addressed by this study. We recognize that there are a large number of studies that address the use of Agile and DevOps separately. The combined use of both has received less attention from the scientific community. The studies conducted are mostly conducted by software engineering consulting firms that highlight the benefits of their combined adoption, but do not allow in a comprehensive way to characterize this phenomenon in the area, since they represent individual views of these implementations. This study addresses this limitation by performing an analysis based on multiple case studies (there are 12 in total), which provides a better understanding of this phenomenon and highlights the main benefits of the combined adoption of Agile and DevOps. The qualitative methodology used in this study is the most appropriate because it allows to understand this phenomenon in the organizations where it occurs and enables to identify the various benefits and explore the relationship between them.
The authors used qualitative methodology to explore the benefits of the combined adoption of DevOps and Agile. This methodology is very simple. I suggest using the more robust methodology and applying it to the large dataset of the case studies.
Author’s response: Thank you for the recommendation. We believe that the most important thing is to choose a methodology that is adjusted to the objective of the study, regardless of whether it is simple or complex. The apparently simple qualitative methodology hides a methodological process to reduce the risk of bias as shown in Figure 1. A total of 12 case studies from software engineering companies located in different geographical areas were considered. This number is in our opinion sufficient to draw meaningful and scientifically relevant conclusions. We recognize that it would also be relevant, and as future work, to consider a larger dataset that would allow a quantitative study to be conducted. We give this suggestion as future work.
The authors provide quite a detailed discussion about the findings. I appreciate that they also tried to identify the weaknesses of the study.
Author’s response: Thank you very much for your positive feedback regarding the pertinence of discussing the findings and weaknesses of this study.
Please, let me summarise the strengths and weaknesses of the paper:
Strengths
clear writing style suitable for the students of introductory courses of software engineering,
using relevant references,
application of simple and very flexible thematic analysis,
discussion about the results.
Author’s response: Thank you very much for your feedback regarding the main strengths of this manuscript.
Weaknesses
the authors did not explain the novelty and uniqueness of the paper insufficient details,
Author’s response: We have improved the abstract and introduction sections to explain the novelty of this study. We believe that the adopted approach supported by multiple case studies avoids the individual limitation of each company's vision and reduces the risk of bias, as well as comparing, group, and systematizing the main benefits of the combined adoption of both methodologies.
The authors analysed twelve case studies, which they did not describe in enough detail,
the quality of these case studies is questionable; the authors mentioned that the case studies could represent PR or vendor-specific press releases, which prefer positive views of the phenomenons they provide,
the applies methodology is straightforward,
the authors identified twelve benefits, while some of them were mentioned very rarely between the case studies,
Author’s response: We have provided more detail regarding the adopted methodology. The data comes from secondary sources and the authors have not made any changes to the press releases that represent the view of each manufacturer. No summarization of the press releases has been made. It cannot be guaranteed that there is no risk of bias since the identified benefits come from press-releases from commercial companies in the area and that have commercial goals in the market. However, to minimize this risk the external and internal validity mechanisms were used. As external validity was considered multiple case studies of companies in different geographic areas and as in-ternal validity was used the same identifier to associate similar benefits between the case studies.
We agree that some benefits are more relevant than others and this information is provided in the column “Ranking” of Table 4. We have explained in the results section how the data provided in table 4 should be interpreted.
The concussion can not be generalised and requires more sophisticated research and methodology.
I recommend that authors try to publish this study at the conference and improve it after receiving feedback from the large community.
Finally, I recommend the authors improve the overall quality and novelty of the paper using a more robust methodology for qualitative and/or quantitative research, find more case studies without the marketing or PR background, realise systematic survey between the software companies, which tried to implement DevOps and Agile and use some statistical methods to be able to generalise the results.
Author’s response: The main objective of the qualitative methodology is not the generalization of results, but the perception and knowledge of a phenomenon in the context in which it occurs. The use of 12 case studies allowed us to identify the main benefits of the combined adoption of Agile and DevOps and provided a good understanding about the less relevant benefits that emerge only in specific organizational contexts. We consider that these results offer mainly practical contributions relevant for software companies. As a future work, we welcome the reviewer's suggestion to conduct a quantitative study based on a large dataset to address the challenge of generalizing the results. These recommendations and incentives are important to motivate our research team to continue work in this area but should not invalidate the publication of this study in view of the practical contributions of this study to software companies.
Author Response File: Author Response.docx
