Integrating Scrum and DevOps for Very Small Entities in South America: Design of an Implementation Guide

Abstract

This article examines the challenges very small entities (VSEs) face as defined by the ISO/IEC 29110 standard, particularly their reliance on individual expertise and insufficient adherence to established software engineering practices and international standards, which can lead to increased rework. The article proposes a guide that harmonizes the Scrum and DevOps frameworks to address these challenges and facilitate their adoption in software development processes. An evaluation of the guide by experts indicates significant progress in integrating agile practices within VSEs. The methodology involved understanding both frameworks, modeling a harmonized development process, and creating a characterization tool to assess the current state of VSEs concerning best practices. Also, the guide includes an evaluation instrument to measure the adoption of its suggestions before and after implementation. While expert evaluations confirm the guide’s clarity and practical value, they suggest minor navigation and content clarity improvements. Ultimately, the guide aims to enhance VSEs’ development efficiency and improve software quality.

1. Introduction

Software development companies can be classified as very small entities—VSEs (up to 25 employees), small entities—SEs (25 to 50 employees), medium-sized entities—MSEs (up to 250 employees), and large entities—LEs (more than 250 employees) [1]. This classification has characteristics that allow an understanding of how such companies work, their advantages and disadvantages over other classifications, and what problems they constantly face [2]. Likewise, the ISO/IEC 29110 standard allows an understanding of how it could help to improve the software development process (SDP) under the adoption of practices suggested in software engineering [3]. However, the implementation of this standard and other options proposed in the literature is affected when it is necessary to adopt agile frameworks [4]. This is because such frameworks indicate that implementation can be done, but not necessarily how, leaving everything to be explored through the trial and error of different options [5]. This is very costly and, in some cases, unfeasible for VSEs [5].

VSEs constitute a significant portion of the software development industry, accounting for more than 70% [4], a situation that holds across Latin American countries. For this reason, they stand out more when reviews are made about the different success and failure factors of SDP, such as chaos reports, where it is still not possible in more than 50% of cases to achieve coherence between the planning, development, and results of projects of various sizes that lead to their success [6].

Most of these companies have empirical processes, guided by the experience of the founders and development teams, more focused on coding than on formal processes that give relevance to stages such as requirements analysis, software design, quality, and even deployment, which is consistent with the findings in [6]. Reprocessing is constant due to problems in understanding the impact of applying the recommendations of a formal process guided by software engineering practices. This negatively affects the results regarding planning and compliance with objectives agreed upon with clients. Additionally, there is a considerable impact on people constantly under role overload, demotivation, and frustration from having to solve the various errors that arise after the deliveries, as indicated in [5].

To address the issues above, VSEs try to implement frameworks such as Scrum and DevOps that promote communication, quality, automation of repetitive activities, and retrospective reviews to facilitate companies’ continuous improvement based on understanding what they do well and what they can improve [5]. However, agile frameworks have a philosophical approach where there are no standards or norms that guide the processes beyond recommending practices that can improve development results, which represents a great difficulty for their implementation [7]. In addition, some are not prepared to take this step towards agility because their SDP is still empirical [7].

This paper addresses a critical need in the software development industry by presenting the design of a guide for implementing harmonized practices in VSEs, drawing on the insights of Scrum and DevOps. This guide offers a structured approach that leads to more consistent and reliable software development outcomes by adopting industry-proven practices from both frameworks. Additionally, the guide is evaluated by experts for refinement before dissemination, which further ensures its relevance and applicability and adds credibility to its recommendations.

This research aims to address the following primary research question (RQ): Is the design of an integrated Scrum and DevOps implementation guide for very small entities (VSEs) considered adequate by software engineering experts?

To support a comprehensive evaluation, the study also investigates a secondary sub-question: What specific areas of the guide require improvement before its application in industrial settings?

The paper is organized as follows: Section 2 covers the background, Section 3 outlines the motivation scenario, Section 4 details the methodology, Section 5 presents the results, Section 6 presents the discussion, and Section 7 presents the conclusions and future work.

2. Background

A software development process (SDP) is made up of the following stages [8]:

• Analysis: The objective of this stage is to understand the solution’s context and determine the system’s functional characteristics to be built. The information is consolidated into an artifact that serves as a deliverable at this stage. This stage is directly related to planning and quality because it allows defining functional test scenarios to check the correctness of what has been built and handling exceptions and alternate flows.
• Planning: The findings of the analysis stage are assessed with a focus on understanding how long it will take to carry out the development project. Here, estimation and prioritization techniques are applied.
• Design: This stage addresses how the project will be executed, resulting in the software architecture document. This document outlines the quality attributes of the architecture, tackling common software development challenges related to system behaviors under various conditions. Solutions to these challenges are derived from design patterns (proven solutions integrated into development frameworks across programming languages). The interconnections among these decisions are formalized through UML diagrams, providing a structured representation of the architecture.
• Development: The results of the analysis and design stages are shaped through coding. The delivered outcome is the source code.
• Quality: This stage validates that the system responds adequately to different stimuli through a preventive approach. Quality controls are implemented before saying that the functionality has completed its construction, or corrective, in which test cases are implemented to check correctness and the handling of exceptions after the functionality has been achieved.
• Deployment: The final goal is to take the version that has passed the quality check, generate the deployable unit, and put it into the final environment (development, quality, or production).

These stages are integrated with Scrum and DevOps in an agile SDP that implements the recommendations given by the frameworks. Below are detailed the possible elements to consider in both Scrum and DevOps, the motivation for the proposal, and the integration of the frameworks through the implementation guide of software development practices.

2.1. About Scrum

Scrum is recognized as an incremental, iterative framework composed of three roles: the product owner (PO), who oversees the maximizing of the value of the delivery by being transparent about the scope and helping to define the functional needs of the system so that everyone shares the same vision of the product; the scrum master (SM), who has the greatest knowledge of Scrum and helps with its implementation; and the scrum development team (SDT), which develops the work [9].

Additionally, it has five events: the first is the sprint, which is considered a time box with a duration of 1 to 4 weeks in which the product is developed through partial deliveries added together until the completeness of the project is reached. The second is the sprint planning meeting, which takes place at the beginning of a sprint and allows the team to be clear about what will be done and how it will be done. The third event is the daily scrum, which is an informative meeting that allows them to be aware of the progress and impediments in the execution of the sprint, aiming at transparency and constant communication between all participants, in addition to the control of project risks. The penultimate event is the sprint review, a meeting inviting stakeholders to validate the fulfillment of agreed commitments. Its goal is to get early feedback and detect if there are improvements that need to be included as soon as possible to ensure satisfactory delivery. The sprint retrospective is the last event. It is a meeting where the team recognizes what they are doing well and what they can improve to create strategies to implement those improvements in the following sprints.

Finally, Scrum focuses on delivering three artifacts [9]. The first is the product backlog (PB), which aims to group the user stories to allow the SDT to build those functionalities. It is the result expected from the previously mentioned stage of analysis and the fundamental input of the planning. The second is the sprint backlog (SB), obtained by applying estimation and prioritization techniques to the PB to decompose that artifact into minor backlogs. SBs are the commitments agreed upon for each sprint and allow greater control over delivery. Finally, the increment, resulting from the construction completed in each sprint, undergoes quality assurance and deployment in the test environment before the fully developed product is ultimately deployed in the production environment. Figure 1 summarizes the above.

2.2. About DevOps

DevOps is a framework that fosters collaboration between those who carry out software development and those in charge of the infrastructure and operations that support all the systems and services of companies at the hardware level [10]. This collaboration of all members as part of a single team aims to promote better communication channels, automate repetitive tasks, reduce rework, and promote continuous organizational improvement using quality as a transversal axis throughout the development process [11]. It aims to understand and adopt best practices for software development and operation, which generate sufficient, detailed, and valuable information for decision-making on what aspects may need to be improved within the development process or after putting it in a quality or deployment environment. This allows for gradual improvement depending on the practices to be implemented. Additionally, DevOps divides its suggestions into eight areas [12].

For this reason, DevOps is more than just practices and tools [12]. It becomes a mindset that revolves around quality, continuous feedback, and the generation of helpful information to promote continuous improvement as part of the organizational culture.

Table 1. Association between phases and practices. Adapted from [13].

Phase	Description/Purpose	Featured Practices
Code	The solution defined in the analysis and design is developed.	Control version.
Build	Development teams use the programming language’s instructions to determine whether their source code can build a deployable unit that they can release and test.	Look to automate this step by using dependency managers. The first constructions are local but automatically replicated in quality and production environments.
Test	Checking the results of development with various testing approaches that may be manual in their beginnings but seek to become automated when the adoption of the practice is more advanced.	Unit tests (UT), manual code inspections, static code analysis (SCA), manual and automated functional type tests, load, volume, stress tests, safety tests, and fault tolerance tests stand out.
Release	The goal is to automatically construct a drop-down unit to verify that the changes have not negatively altered the incremental version syntactically.	The most practical is continuous integration (CI)
Deploy	The drop-down unit is automatically put into the final environment when coding increases, as the version manager approves.	The most prominent practice is continuous deployment (CD)
Operation	Ensuring stable, organized, replicable operating environments with rapid recovery from possible failures.	Practices associated with managing infrastructure configuration in a stable and replicable way using containers and infrastructure as code.
Monitor	Obtaining information about development and infrastructure and identifying problems associated with performance, security, and failure response.	Use monitoring and logging tools (logs) and define and use performance metrics and key performance indicators (KPIs).
Plan	Achieving strong collaboration and communication between everyone involved during the software development cycle and beyond.	The most common practices are fostering a culture of shared responsibility, holding regular feedback meetings, and promoting effective communication channels.

details the phases, their purposes, and suggested practices:

3. Motivation Scenario

The difficulties in VSEs are that they tend to have heterogeneous organizational cultures, low diversity of development technologies, little practical and helpful documentation for development teams, and an overload of responsibilities as the same person has to function in several roles, which has an impact on the high turnover of people who enter and leave the company as a result of the stress generated by this way of working [12]. Additionally, the lack of practices that promote quality as the central axis of the development process creates non-compliance with the commitments planned and agreed upon with customers, affecting budgets and cash flow [14], which can become an obstacle to investing in continuous improvement processes [5]. In other words, because little software engineering is understood and applied within their SDP, predictions of their results are not reliable, and they struggle to succeed in achieving adequate planning, building customer loyalty, and generating constant results that guarantee high-quality products [15].

The VSEs then face the problem of asking themselves: What should they adopt, and how should they do it? [16].

Some relevant literature reviews have been conducted, such as [17]. These have helped us to understand the practices that have led to the success of projects, and by utilizing the findings of [13], where the integration of common practices for Scrum- and DevOps-based VSEs is addressed, the construction of the implementation guide proposed and evaluated in this article by experts has been achieved. Implementing Scrum and DevOps in VSEs brings numerous advantages, such as making them more efficient in the development process, thanks to the automation of repetitive tasks by integrating practices and tools [18]. Also, the possibility of better software quality through continuous testing, constant monitoring, and more efficient delivery cycles that allow them to respond quickly to the needs of the market and customers is highlighted [19]. In addition, Scrum and DevOps foster greater collaboration and communication between development and operations teams, reducing silos and facilitating problem-solving, which together optimizes development and operation processes, allowing VSEs to be more competitive in the market [20].

While the integration of Scrum and DevOps has been recognized as a promising approach to improving software development processes, especially in environments requiring agility and continuous delivery, the literature reveals several approaches to this integration. For instance, Ref. [21] explores the benefits and challenges of combining DevOps and Agile practices, highlighting improvements in collaboration, automation, and feedback loops, but also noting the need for tailored guidance to address organizational context and resource constraints. Similarly, Ref. [22] discusses the convergence of Scrum and DevOps, emphasizing that while the frameworks are complementary, successful integration requires clear mapping of roles, events, and practices to avoid overlap and confusion.

Despite these valuable contributions, most existing guidelines and case studies previously mentioned in this section focus on medium-to-large organizations or provide general recommendations that may not be directly applicable to VSEs. VSEs face distinct challenges, such as limited personnel, overlapping roles, and constrained resources, which are often not addressed in broader integration frameworks. The lack of practical, context-specific guidance can hinder the effective adoption of integrated frameworks in smaller organizations [21].

The present work distinguishes itself by proposing an integrated implementation guide specifically designed for VSEs in Latin America, considering their unique operational realities. Unlike previous studies, this guide offers a step-by-step, interactive approach that aligns with the ISO/IEC 29110 standard and is validated by regional experts. This targeted approach not only bridges the gap identified in the literature but also provides actionable recommendations that facilitate the adoption of both Scrum and DevOps practices in a resource-constrained context.

4. Method

Building on the identified research gaps, this study tests the following hypotheses through expert evaluation of the proposed guide:

• Null hypothesis (H₀): The designed guide is not perceived by experts as an adequate or useful tool to facilitate the understanding and adoption of Scrum and DevOps practices in VSEs, or it presents significant deficiencies that require substantial improvements before it can be recommended for industrial application.
• Alternative hypothesis (H₁): Experts will perceive the designed guide as an adequate and useful tool to facilitate the understanding and adoption of Scrum and DevOps practices in VSEs. Furthermore, experts are expected to identify specific areas for improvement that can be addressed prior to its industrial application.

The method for constructing the guide comprises the following phases: (I) modeling the software development process that integrates Scrum and DevOps; (II) creating a characterization instrument that allows knowing the current state of the VSEs regarding the suggested practices; (III) creating the guide; and (IV) evaluating the guide.

4.1. Modeling a Software Development Process That Integrates Scrum and DevOps

The authors first focused on integrating the development process, Scrum, and DevOps, and creating a graphic representation of the existing relationship [5]. This step aims to facilitate VSEs’ understanding of the desired development process and provide guidance. The steps taken to create the model are described below.

• Review of which Scrum and DevOps practices are used in each phase of the development cycle. This mapping identifies the elements to be integrated.
• Create a graphical facilitation that represents the development process that integrates Scrum and DevOps.

4.2. Creating a Characterization Instrument

Since Scrum focuses its recommendations on the analysis, design, and management of the team’s work, most of the practices found in documents tend to be associated with development within the sprint. Therefore, the most significant impact within this cycle to promote quality is more related to DevOps and its suggestions. This is key to developing both the assessment instrument and the guide.

To help effectively implement the suggestions proposed by DevOps tailored to the unique needs of each VSE, it is necessary to identify the different characteristics of companies and understand their strengths and options for improvement. In Section 2.2 (“About DevOps”) and Table 1, the authors define the eight DevOps phases—code, build, test, release, deploy, operate, monitor, and plan—based on established sources, e.g., [13,23] to ensure terminological consistency. The rationale for using these phases is based on their widespread adoption in both academic and industrial DevOps frameworks, as detailed in the cited works.

For the above, and based on [24], the authors developed an Excel form containing questions for each DevOps phase, with an answer value that determined the phase’s classification and practice. It is important to note that it is managed in this way because, on some occasions, they do not want to share their information. Therefore, this makes it easier for the company to always own the information, enabling them to apply the evaluation before using the guide without fear of exposing sensitive information.

4.3. Creating the Guide

Based on what is defined in Section 4.1 and Section 4.2, the Genially tool was used to create the guide as interactive digital content in Spanish, since the focus of this work is exclusively aimed at Spanish-speaking countries in Latin America.

This tool helps design interactive presentations aimed at fostering creative ways of communicating [25]. It has been proven in different academy contexts, such as software development education [26], training programs in the distance format [27], and digital literary reading practices [28], as well as in contexts of training professionals in companies, for the development of professional skills in a special professional program [29] and to implement the organizational justice model.

It allows work based on three basic principles that can be used to create content: animation, interactive, and integration [28]. Animation can be used to configure entry or exit digital content, as it is used in the menu, and to create animated images, such as enlarging the image size to better appreciate the details in it. The interactive principle is used for navigation through the content, seeking to organize information in a way that prevents a single page from becoming saturated with content. Finally, the integration principle is used to access external links and information. In this case, it was used for link references to official information about the frameworks and tools. Additionally, it allows its sharing on the web, downloading the entire site to be hosted by another host if necessary, or downloading in PDF format for special cases.

An organized, clear, and step-by-step presentation of information is essential for guiding readers through the implementation guide of integrated software development practices in VSEs based on Scrum and DevOps. The guide begins with a general menu outlining its components, including the introduction, objectives, and work team, as well as sections to explore Scrum and DevOps further. Additionally, it offers options for integrating these methodologies through a software development process (SDP) and for implementing best practices in software development.

DevOps encompasses practices lacking a specific implementation order [10]. However, certain practices are interdependent, requiring the implementation of some before others. This guide suggests using the eight phases of DevOps—coding, construction, testing, release, deployment, operation, monitoring, and planning—as a flexible framework. This structure allows for grouping practices by phase, facilitating information organization, and formulating questions to guide practice adoption. A summary of these relationships is illustrated in Figure 2.

4.4. Evaluating the Guide

To apply the expert evaluation, a real-time Delphi method was adapted [30]. This is a shorter variant of the traditional Delphi method, which takes place during a meeting and uses mechanisms to summarize the answers given immediately. This method anonymizes expert responses, provides controlled feedback, and facilitates iterative consensus-building [31]. The steps proposed to follow the method are defining the coordinator group and expert group, creating a questionnaire for the experts, sharing the guide and the form, collecting data, and analyzing data.

For the first step (coordinator group definition), Margarita Varela-Ruiz et al. (2012) [30] recommend creating a group of two to six people who are in charge of studying and refining the working protocol (selection and recruitment of experts, timetable, etc.), studying and approving the list of experts, drawing up questionnaires, encouraging the participation of experts, analyzing the responses of the rounds, and managing the process. In this case, the authors form the coordinator group.

The coordinator group carries out the second step (expert group definition). Experts are those who assume the responsibility of issuing judgments and opinions, which is fundamental for the application of the method. The criteria for selection in this case depended on the nature of the topic and the purpose of the study. A traditional approach was considered for their selection (software engineers specialized in the implementation of best practices based on Scrum and DevOps), considering the level of knowledge, experience, publications, and prestige in their field. Additionally, the number of experts also depends on the objectives and budget of each study [30]. In general, it is considered that there should be no fewer than seven experts, and the maximum is around 30. In this case, the target population was 15 experts in the field of software engineering, adoption of agile practices, and process improvement from Latin America who not only have long academic careers in the subject but have also worked in collaboration with companies supporting the implementation of agile practices in software development processes. Once the selection of experts was made, a formal invitation was created through email, outlining the objective of the study, the steps of the method, the questionnaire to be solved, the number of questions to be answered, the time to answer them, the duration of the process, the potential usefulness of the results, and the benefit of their contribution to the research carried out. The coordinator group, responsible for managing the process, did not participate in the expert panel.

In the next step, creating the questionnaire, the validation instrument designed by the authors followed the recommendations of [32], which aim to identify key questions on aspects such as clarity in the wording, understanding, and relevance of both the question and the suggestions. Also, the authors evaluated aspects of the usability of the guide associated with its ease of use and navigation [33], in addition to the helpfulness and effectiveness of the suggestions [34]. Usability testing is crucial for an implementation guide, especially for VSEs adopting Scrum and DevOps, as it ensures the guide is easily understood, navigated, and applied despite potential resource limitations and varying expertise levels. By providing direct feedback on the guide’s information architecture and practical applicability in real-world scenarios, usability testing helps overcome barriers to implementation, increases adoption and compliance, reduces errors and rework, and validates expert feedback, leading to improved processes and business outcomes by creating a usable resource [35]. The coordinator group carried out this.

The questionnaire consists of nine questions: six closed, aligned with specific evaluation criteria and indicators (the first, second, third, fifth, seventh, and eighth), and three open-ended (the fourth, sixth, and ninth), which are complementary. The relationship is presented below in

Table 2. Relationship between question, evaluation criteria, evaluation, and indicators.

Question	Criteria for Evaluation	Description	Indicators
First question: How would you rate the ease of navigation of the guide in the Genially tool?	Navigability.	It aims to assess the degree of usability of the guide in terms of its navigability.	Very easy
			Easy
			Neutral
			Hard
			Very hard
Second question: To what extent did you find the information provided in the guide to be clear and understandable?	Clear and understandable information.	It aims to evaluate how clear and understandable users perceive the information presented in the guide.	Very clear and understandable
			Clear and understandable
			Unclear and incomprehensible
			Very unclear and incomprehensible
Third question: Do you consider the questions posed at each stage relevant to implementing agile practices?	Relevance of the questions.	It aims to determine whether users find the questions presented at each stage pertinent and useful for effectively implementing agile practices.	All questions are relevant
			Most questions are relevant
			Some questions are relevant
			Few questions are relevant
			They are not relevant
Fifth question: Do you find the suggestions at each stage helpful in addressing the challenges?	Helpfulness of suggestions.	It aims to assess whether users perceive the suggestions provided at each stage as useful and effective in helping them face the challenges they encounter.	Very useful
			Useful
			Neutral
			Nothing useful
			Not very useful
Seventh question: What do you think about the clarity and effectiveness of the diagrams presented in the guide?	Clarity and effectiveness of the diagrams.	This question aims to evaluate users’ opinions on the clarity and effectiveness of the diagrams included in the guide.	Very clear and effective
			Clear and effective
			Neutral
			Unclear and ineffective
			Nothing is clear and ineffective
Eighth question: On a scale of 1 to 10, how would you rate the guide’s overall effectiveness in implementing agile practices in VSEs companies?	Effectiveness of using the guide. Mean (SD)	This question aims to gauge users’ perceptions of the guide’s overall effectiveness in facilitating the implementation of agile practices within VSEs.	On a scale of 1 to 10

.

In the next step, the guide was shared with the expert by email, as outlined in the second step. A review was made without any guidance other than the context of a research project seeking to facilitate the adoption of the various practices. The Google form was shared at https://forms.gle/cHi3eXKGyHUaVa4H6, (accessed on 1 April 2024) where the experts could leave their comments freely, and a meeting session of no more than half an hour was requested, where they presented all their findings individually.

The final steps were collecting and analyzing the data and generating the quantitative and qualitative results of the expert review, and were carried out by the coordinator group.

5. Results

The results for each of the stages of the method are presented below.

5.1. A Software Development Process That Integrates Scrum and DevOps

The traditional software development cycle, which is focused on application development, is made up of the phases of analysis, design, development, quality, and deployment [8]. These phases frame the steps that a development team must take to execute a project. The analysis allows the functional needs of the project to be identified, delimiting the scope of the solution and defining the vision of the product by means of an artifact. Here, Scrum provides the product backlog as an artifact, which contains the user stories that make up the solution. To build it, stakeholders, the product owner, the scrum master, and the software development team interact to consolidate the information. With the document completed and validated, the project team can estimate and prioritize the user stories to create the plan to be executed. Scrum indicates that, when creating the roadmap, the product backlog can be broken down into smaller lists of user stories to be executed per sprint. These are called sprint backlogs. These artifacts (the product backlog and sprint backlogs) serve as inputs for the subsequent phases of design, development, and quality.

The approach to be used in the design phase is collaborative. That is, all members of the software development team participate in the technical decisions that guide the system’s implementation. The artifact built at this stage is called a software architecture document (DAS).

A sprint runs the development phase to perform an incremental, iterative deployment. It is important to note that the execution of the development sprint begins with a ceremony proposed by Scrum called the sprint planning meeting, where what will be done, and how, is reviewed. Therefore, it receives as input the sprint backlog to be carried out and the DAS of the solution. During each day of the sprint, a daily scrum meeting is also held, where the aim is to determine the progress and impediments to progress so that the team can always know the status of the project and take corrective action in case of any problems arising. At the end of each sprint, two events are held. The first is called a sprint review, which is a meeting where teams present the user stories engaged in the sprint backlog already implemented in the solution. This is carried out with the aim of obtaining approval for delivery and obtaining feedback that allows the solution to be improved if required. The second event is the sprint retrospective, where internally, the team encourages continuous improvement based on a retrospective technique that allows them to determine what they do very well to continue it and what they must adjust to do better. Additionally, it is proposed that before the development sprints, a zero sprint, or preparation sprint, be carried out. This is where DevOps takes a relevant role because it is where all the practices to be used by the development team are configured. In this case, it is proposed that versioning, CI, AEC, automated functional tests, continuous monitoring, and non-functional tests be implemented. To achieve this, the tools that allow the use of the practices are configured, and the archetype is used to check its operation (the first version of the code, where the structure is implemented and a functional transactional example). All these tools will be available to the team, generating constant information that allows each developer to know the quality of what they are building in a preventive way.

The integration between Scrum and DevOps in a software development process is shown in Figure 3. Based on this figure, it can be highlighted that the impact of Scrum on VSEs is related to the process stages of analysis, planning, and design. Additionally, it helps us to understand and organize the work to be executed during the development, quality, and deployment stages, and to obtain early feedback in controlled periods called sprints. The impact of the DevOps part is more associated with the stages of development, quality, and deployment, generating enough quality controls and metrics that record how the team is doing its work, which aspects are compelling, and which are not, to propose how to improve them.

5.2. Characterization Instrument

For the characterization instrument, an Excel file is organized that contains the relationships between the DevOps phase, the practice, the question, and a classification mechanism that allows both the practice and the stage to be classified based on [24]. The instrument is divided into a presentation, which contains an introduction to the document, its purpose, and the instructions for use, and details of the code, build, test, release, deploy, operate, monitor, and plan phases.

The objective of this document is to determine which area the company wants to review freely and to ensure that they know, based on that information, its status in the face of practice, thus generating an opportunity for guidance through the implementation guide. Additionally, the participants own the document, so they do not need to share their information they apply to the guide.

The recommendation for using the guide is that the characterization instrument be completed before using it; in this way, the previous state of the company is known in the various areas that the guide can impact, thus making it possible for those who use it to centralize their effort strategically. After selecting an area and implementing the guide’s recommendations, it is necessary to reapply the characterization instrument to measure the impact again. The main menu shows the characterization instrument in the “INSTRUMENTO DE CARACTERIZACIÓN” option, as seen in Figure 4.

5.3. The Guide

The guide is available at https://view.genial.ly/65d908e6cf8e4e0014acb0a4 (accessed on 1 April 2024). It was built in Spanish with the Genially tool, which allows you to create interactive content such as web pages or PDFs. The guide starts with a presentation page and an option menu described next:

The first option, “INTRODUCCIÓN” (Introduction), serves as a guide to using this document effectively, explaining its purpose and providing instructions. By clicking the + button, a pop-up window with relevant information will appear, aiding in the navigation of the guide.

The second option, “OBJETIVO” (Objective), outlines the guide’s aim.

The third option, “EQUIPO DE TRABAJO” (Work Team), is pivotal in establishing the guide’s credibility. It introduces the researchers who contributed to the guide’s construction, each with a box containing their photo, full name, and a +info button. Clicking the button reveals a short biography of each researcher.

The fourth option, “INSTRUMENTO CARACTERIZACIÓN” (Characterization Instrument), allows users to download and learn the instrument that will enable them to determine the characteristics and opportunities the company has in implementing practices suggested by DevOps.

The fifth option, “INFORMACIÓN DE SCRUM” (Scrum Information), presents graphic facilitation of Scrum’s roles, events, and artifacts. All this is within Scrum’s harmonizing relationship with the stages of the SDP. The objective of this section of the guide is for companies to understand how Scrum works and what its interaction is at a high level between its various components.

The sixth option, “INFORMACIÓN DE DEVOPS” (DevOps Information), features an image with an interactive button that allows users to view detailed descriptions of each of the eight phases that make up DevOps. When clicked, the button opens a pop-up window providing an in-depth explanation of one or more phases and their objectives.

The seventh option, “PROCESO DE DESARROLLO” (Software Development Process), explains how Scrum and DevOps can be integrated into an SDP. The representation indicates that Scrum significantly influences the analysis, planning, and design stages by promoting practices related to product management, collaborative work, and proactive team communication. It helps organizations understand how to approach the development, quality, and deployment stages, where DevOps contributes through various suggested practices from the literature. These practices include creating archetypes with unit tests, versioning, continuous integration, static code analysis, continuous deployment, automated functional tests, continuous monitoring, and non-functional testing. Additionally, it recommends infrastructure management practices such as using containers and infrastructure as code. Figure 3 visually facilitates this integration.

The last option in the guide, “PRÁCTICAS ARMONIZADAS (INTEGRADAS)” (Harmonizing Practices (Integrated)), contains the relationship between DevOps’ proposed phase, question, and suggestion.

Initially, the DevOps representation is used to evidence the phases. This allows one to start without limiting oneself to a single order and freely choose any phase. Each phase has an interactive button that leads to the specific group of questions for that section, as seen in Figure 4.

5.4. Evaluating the Guide by Experts

Fifteen experts from Latin America completed the instrument.

Table 3. Assessment by the panel of experts.

Question	Criteria for Evaluation	Categories of Responses	Evaluation by the Experts (n = 15)
First question	Navigability: n (%)	Very easy	5 (33.3)
		Easy	5 (33.3)
		Neutral	3 (20)
		Hard	1 (6.7)
		Very hard	1 (6.7)
Second question	Clear and understandable information: n (%)	Very clear and understandable	2 (13.3)
		Clear and understandable	11 (73.3)
		Unclear and incomprehensible	2 (13.3)
		Very unclear and incomprehensible	0 (0)
Third question	Relevance of the questions: n (%)	All questions are relevant	8 (53.3)
		Most questions are relevant	4 (26.7)
		Some questions are relevant	3 (20)
		Few questions are relevant	0 (0)
		They are not relevant	0 (0)
Fifth question	Helpfulness of suggestions: n (%)	Very useful	3 (20)
		Useful	9 (60)
		Neutral	2 (13.3)
		Nothing useful	1 (6.7)
		Not very useful	0 (0)
Seventh question	Clarity and effectiveness of the diagrams: n (%)	Very clear and effective	4 (26.7)
		Clear and effective	8 (53.3)
		Neutral	3 (20)
		Unclear and ineffective	0 (0)
		Nothing is clear and ineffective	0 (0)
Eighth question	Effectiveness of using the guide: mean (SD)		7.8 (1.42)

SD: standard deviation.

presents the results of the quantitative responses to the questionnaires sent to the experts.

From Table 3, it can be highlighted that most experts found the guide to be easy or very easy (66.9%) to navigate, 86.6% had a positive view of the clarity and comprehensibility of the information, and 80% considered most or all of the questions relevant and that the suggestions were useful or very useful. In addition, this same percentage considered that the diagrams were clear or very clear and effective. Finally, the overall guideline score on average was 7.8 (±1.42).

Concerning the open-ended questions, the qualitative responses are detailed below.

To expand on the information in the third question, the fourth question (Which ones do you not consider relevant and why?) was designed to compile relevant suggestions to improve the questions proposed by the practice implementation guide. Some experts recommended modifying the following question to have a Likert scale of 3 or 5 values and to be worded differently: The instrument question: are incident response workflows automated to facilitate rapid incident detection, response, and resolution? Should it be: What percentage of your workflows have automated incident responses to facilitate rapid incident detection, response, and resolution? Poor (less than 20%), Low (21% to 40%), Medium (41% to 60%), High (61% to 80%), and Very High (81% to 100%). This also allows for defining the level of each company better. The scales should not be generic but dependent on the question.

The sixth question (Which ones do you not consider relevant and why?) gathers observations based on the fifth question to refine suggestions that can increase the adoption of the guide as a tool for ownership of practices in VSEs. On the one hand, it is recommended that all diagrams be left in Spanish and future guides be created in other languages if required, because the target audience is currently Spanish-speaking Latin American companies. Including complementary elements with audio can be a handy way to expand on the explanations or suggestions, especially in appropriating Scrum elements, DevOps stages, and explanations of very long diagrams. On the other hand, property-based testing could be interesting to evaluate. However, it is not a practice referenced in the previous literature review to identify practices. However, the recommendation of various authors has led to the inclusion of unit tests as a minimum seal of quality, with an approach that allows the responsibility of quality to be immersed in development as a requirement for the completeness of functionalities and as a mechanism to identify if changes made have affected functionalities built by other developers.

The last question (If you have any additional comments about the guide, please share them here. We would greatly appreciate any suggestions for improvement) aims to collect extra suggestions that allow for further improvements. It mentions improving the usability of the guide, to help the authors determine how it might be possible to simplify navigability, include aids to generate an initial pilot, and implement it in a case study with companies. Additionally, it is necessary to refine the questions and adjust the navigability for a more intuitive use of the guide and its content, thus facilitating the understanding of its content. As the experts suggest, it is probably more relevant to the tool than complementary information. On the other hand, it is evident that the content has been evaluated well, and it is possible to make a new version of the guide by applying the suggested adjustments.

6. Discussion

The findings of this study highlight the effectiveness of the proposed implementation guide designed for integrating Scrum and DevOps practices for very small entities (VSEs) in South America, which allows us to verify the alternative hypothesis raised by this work. The guide addresses critical challenges faced by VSEs, such as managing technical debt, speeding up product delivery, maintaining high-quality standards that meet the market’s needs, and ensuring customer loyalty [36]. One possible solution for this is the use of best practices for software development based on DevOps [37], which implies the implementation of quality control points and continuous feedback to inform the development team of the quality stage of the project evolution [38]. This scenario allows collaborators always to know, during the increment of the deliverable, the actual status of the evolution of the project in terms of quality to avoid and reduce rework resulting from the need for adjustments after the completion of one or more functionalities during a development sprint [39].

Additionally, the guide provides actionable recommendations tailored to the unique needs of VSEs in South America. For example, its focus on automation tools and continuous feedback loops enables small teams to achieve greater efficiency despite resource constraints. The inclusion of a characterization instrument further enhances its practicality by allowing VSEs to assess their current maturity level and identify areas for improvement before implementing the suggested practices.

Moreover, the evaluation results indicate that the guide’s clarity and usability make it accessible even to organizations with limited experience in formal software engineering methodologies. However, minor improvements in navigation and content clarity were suggested by experts, highlighting opportunities to refine the guide further. Future iterations could incorporate interactive tutorials or real-world case studies to enhance their applicability across diverse organizational contexts.

Although in recent years, several studies have addressed the challenges and opportunities of integrating DevOps practices into software engineering education, an instrument such as the one presented in this work has not been created that allows guiding the implementation step by step through a model of questions and suggestions, which was also validated by experts previously. For example, Grotta and Prado’s DevOpsBL framework emphasizes project-based learning for academic purposes, where students engage with DevOps concepts through capstone projects that simulate real-world scenarios [40]. However, their approach is primarily tool-agnostic, focusing on the development of soft skills and team collaboration rather than prescribing specific technologies or step-by-step technical procedures. Additionally, experts did not validate it, and there was no proof in the industry.

Similarly, Rong et al. introduce DevOpsEnvy [41], a platform designed to support learning by providing students with a simulated DevOps environment. This enables learners to engage with workflows and collaborative dynamics that mirror those found in professional settings. The system offers a virtualized infrastructure where students can apply DevOps concepts in a controlled and safe context. Nonetheless, akin to DevOpsBL, DevOpsEnvy refrains from offering granular, technology-specific implementation guidance. Its primary emphasis is on replicating industry roles and processes rather than detailing concrete technical steps and understanding how the practices are linked with others to achieve their automation, and the software development improves. This represents a significant difference from this work. Additionally, the configuration presented in [41] only applies to a specific context, leaving out of its coverage the different needs that VSEs could face. This also differs from the guidance proposed in this paper, because the suggestions presented by the guide can be implemented in multiple contexts, focusing on understanding the practice and then on its implementation, and not limiting it to specific technologies.

On the other hand, Bruel and Jiménez [42] highlight the ongoing challenges in teaching and implementing DevOps. The need for meaningful feedback and the difficulty of balancing automation with goals is one remarkable point of their work. Nevertheless, they fall short of offering a granular, staged methodology for practices, tool adoption, and process automation.

The guide proposed by this work indicates that, prior to its use, the characterization instrument must be completed, not only to know the initial state of adoption of practices, but also to be able to create staggered goals that allow the gradual adoption of the suggestions. This has a positive impact on process improvement planning and allows VSEs to go at their own pace.

Likewise, Hobeck et al. [43] contribute further by comparing DevOps teaching approaches at two universities, focusing on the integration of CI/CD pipelines and the use of industry tools like Jenkins and Ansible. Their work demonstrates the value of exposing real-world scenarios as an example of how the practices and tools can be applied. Nevertheless, their methodology is largely centered on graduate-level courses, leaving out their application in companies. It does not provide a phased, incremental adoption model suitable for diverse educational settings, as the other works related before. In contrast to this prior work, the guide proposed in this paper offers a structured, step-by-step methodology that explicitly details the adoption of DevOps practices.

Threats to Validity and Mitigations

While this study provides valuable insights into the integration of Scrum and DevOps for VSEs in South America, several limitations and potential threats to validity should be acknowledged.

• The evaluation of the implementation guide relied primarily on expert reviews, which may not fully reflect its real-world effectiveness in diverse VSEs settings. Although the experts provided constructive feedback and identified areas for improvement, their assessments may be subject to subjectivity bias, as personal preferences and preconceived notions about Scrum and DevOps could influence their evaluations. To mitigate this risk, we employed structured evaluation forms and clear guidelines, but the potential for bias cannot be eliminated. Furthermore, we must also acknowledge the potential for social desirability bias where experts may be inclined to offer overly positive evaluations. To address this, experts were guaranteed anonymity and confidentiality to encourage honest feedback.
• The study’s focus on VSEs in South America limits the generalizability of the findings to other regions with different cultural, economic, and technological contexts. The guide’s effectiveness may vary depending on factors such as organizational culture, team size, project complexity, and available resources. Therefore, caution should be exercised when applying the guide in settings outside of the South American context. This poses a threat to external validity.
• The reliance on expert opinion, while valuable, represents a threat to internal validity. Improvements noted may not be causally linked to the guide and may be influenced by extraneous variables. Future studies could incorporate control groups or longitudinal designs to strengthen causal inferences.
• The validity of the characterization instrument is crucial for accurately assessing the maturity of Scrum and DevOps adoption in VSEs (construct validity). While the instrument was developed based on established frameworks and expert knowledge, further validation is needed to ensure that it accurately captures the intended constructs.
• The study did not assess the long-term impact of the guide on VSE performance. Future research should investigate the sustained effects of the guide on key metrics such as software quality, time to market, and customer satisfaction. Addressing these limitations and threats to validity in future research will further strengthen the evidence base for the effectiveness of the proposed implementation guide and enhance its practical applicability for VSEs worldwide.

7. Conclusions and Future Directions

Our guide is a practical tool based on integrating the suggestions given by the Scrum and DevOps frameworks. It allows for the nurturing of agile software development processes explicitly designed for the context of VSEs. It brings a systematic and refined approach to adopting agile practices, supported by expert assessment.

The results reflect an excellent start to the proposal, with straightforward content, practical questions, and suggestions. The experts considered the information provided in the guide to be clear and understandable, indicating that the content is well-structured and accessible. However, a minority found that some aspects still need greater clarity, suggesting that improvements should be made to ensure that all information is equally understandable to all users. Regarding ease of navigation, most experts considered the guide intuitive and easy to use, although a significant percentage found areas for improvement in navigation. This suggests that the guide could benefit from adjustments to its structure and functionalities to optimize the user’s experience. The diagrams in the guide were generally well-valued for their clarity and effectiveness.

Therefore, based on the experts’ opinions, the guide is potentially practical in helping very small software development entities (VSEs) adopt agile practices. By applying the experts’ suggestions, the next version will increase its effectiveness in disseminating and using it in the industry. Additionally, the guide provides significant benefits by offering a structured approach to adopting agile practices, specifically in DevOps. These benefits include increased efficiency in development processes, improved software quality, and faster delivery cycles. Adapting the guide to VSEs’ specific needs and implementing it in stages allows for additional flexibility that can increase its effectiveness.

In future work, it is recommended that, after refining the guideline according to the suggestions of the experts’ evaluations, the impact of using the proposed guideline be measured through a case study with at least five companies.