1. Introduction
The software architect is the professional responsible for building the software, considering internal and external factors such as the architectural standard and clients, respectively [
1]. For an architect to successfully develop software, they must know the aspects inherent to its creation, for example, the existing frameworks to achieve specific purposes to maintain quality software.
A framework is defined as a structure that serves as a basis for systematically building applications. There is also a need for frameworks used in software architecture to be able to meet such expectations external to agile methods, for example, the Behavior-Driven Development (BDD) framework, used in the life cycle of a system [
2].
BDD introduces a flexible methodology that allows monitoring the characteristics to be developed by elaborating requirements through user stories. These requirements are validated through scenarios that outline the convenience criteria for the behavior of the functionalities in question.
The BDD framework defines the software system’s behavior using the “given, when, then” pattern, expressed in high-level, domain-specific natural language and executed through automated tests. BDD can document non-functional requirements (quality requirements) based on the performance efficiency characteristic presented in the ISO/IEC/IEEE 25010 [
3] standard.
The ISO/IEC/IEEE 25010 standard describes the quality characteristics that can be employed to evaluate software quality, such as non-functional requirements. By aligning the characteristics of the ISO/IEC/IEEE 25010 standard with BDD examples, non-functional requirements can be documented and tested in a structured way. This technique ensures that software meets desired quality standards and provides a framework for evaluating the quality of software in use [
4].
The protocol used to direct this case study was formalized using part of the Goal-Question-Metric [
5] model defined below: “
To analyze the application of BDD,
with the purpose of automating the elicitation and testing,
in relation to non-functional requirements based on the characteristics of the ISO/IEC/IEEE 25010 standard,
from the point of view of those involved in the software product,
in the context of the performance characteristic”.
Table 1 presents the research questions proposed to achieve this purpose.
In order to maintain software quality, there is a need for the structures used in its development to be designed in such a way as to contribute to the delivery of the final product, aiming to achieve what the user expects from the product and reduce rework arising from flaws in these structures [
6]. As described in the case study below, we validate the proposed scenarios in a company that uses the BDD framework for test cycles. Thus, we ensure security in the quality of the product delivered according to testable scenarios for non-functional requirements in a real example.
This study has the following Sections:
Section 2, where we address studies related to this research;
Section 3, where we point out the main points related to the application of the ISO/IEC/IEEE 25010 standard and BDD;
Section 4, where we explain how we carried out the methodological steps of this study;
Section 5, where we discuss the user stories used in this research;
Section 6, where we present the data obtained;
Section 7, where we explain the contributions of this study concerning related works;
Section 8, where we discuss the threats that have been identified and mitigated; and, finally,
Section 9, where we present the contributions of this research to Software Architecture.
2. Related Works
Haoues et al. [
7] conducted a study to show which type of architecture could follow the quality characteristics expressed in the ISO/IEC/IEEE 25010 standard. The authors identified that it is necessary to think about quality throughout the entire software development life cycle to deliver the final product and maintain the quality characteristics of the ISO/IEC/IEEE 25010 standard. Therefore, the architecture used must aim to meet the non-functional requirements demanded to maintain the quality of what was requested as a final product; thus, when carrying out a case study using BDD to elicit non-functional requirements, it will be possible to identify difficulties about guaranteeing such requirements.
Estdale and Georgiadou [
8] explored the scope and interpretation of the quality models of the ISO/IEC/IEEE 25010 standard, presenting aspects such as non-functional requirements inherent to software development. Non-functional requirements directly improve when implemented in products through measurable characteristics that meet a fixed set of specifications, aiming to satisfy usage and consumption expectations. In this way, this case study will identify how BDD will seek to guarantee quality related to the performance efficiency characteristic of the ISO/IEC/IEEE 25010 standard.
According to Jarzkebowicz and Weichbroth [
9], there is a greater need for attention when dealing with the elicitation of non-functional requirements. Therefore, the authors conducted a systematic review followed by interviews to identify how non-functional requirements were elicited and documented. The authors discovered through the results of the studies that non-functional requirements are an essential part of the software development process; however, they are carried out differently, with each company eliciting the requirements differently. Therefore, understanding the elicitation of non-functional requirements is essential, as it can impact the entire execution of software development. Thus, using BDD to elicit non-functional requirements, we can better understand how to make it using this framework with the example of a software development company.
Olsson, Sentilles, and Papatheocharous [
10] also performed a systematic literature review that analyzed empirical research for non-functional requirements. They observed that the ISO/IEC/IEEE 25010 standard was mentioned in the articles analyzed by the authors, which had the function of directing the quality of the software during its development. However, researchers need to understand more when reporting possible solutions regarding non-functional requirements, which affects practitioners’ acceptance. Therefore, validating research through methods such as case studies can effectively identify whether what they publish is in line with the daily challenges in the industry. Thus, this case study can contribute to the use of BDD in identifying and automating non-functional tests to maintain quality and achieve the behavior expected by the software.
4. Methodology
Evidence-based software engineering is a methodology whose purpose is to deliver systems with greater reliability [
20] through, for example, case studies. The methodology used in this research on the application of BDD for non-functional requirements and automation, based on the performance efficiency characteristic of the ISO/IEC/IEEE 25010 standard, was conducted through a case study in collaboration with a company that develops software.
Case studies have been increasingly used in software engineering [
21] because they use empirical data collection that seeks to understand a specific population. Thus, software engineering is based on observing actions aimed at software and their respective responses [
22], as, in this way, it is possible to use the observed aspects to validate how the system should behave. One of the research purposes in software engineering focuses on how software engineers and other professionals in the field carry out development, operation, and maintenance in different situations [
23]. Therefore, according to Perry, Sim, and Easterbrook [
24], a case study is a scientific method carried out in a planned manner, from the elaboration of questions to the presentation of results.
This research is exploratory and descriptive [
23,
25], given the inherent need to elicit non-functional requirements through agile frameworks, in order to integrate quality validation. Some guidelines pointed out by Melegati and Wang [
21] and Peterson [
26] were followed, which involve a clear definition of research questions and goals, the appropriate selection of relevant cases for research, collection, and analysis of data from various sources, such as interviews, observations, and documentation. There is also a need to ensure the validity and reliability of the data through triangulation and member checking.
This research analyzed the case of a team that adopted BDD for non-functional requirements and test automation based on the performance efficiency characteristic of the ISO/IEC/IEEE 25010 standard. Experienced teams are understood to have more than one year of experience in adopting BDD; however, they still need to apply it to non-functional requirements. The case study focused on analyzing requirements elicitation in one of the products developed by a Brazilian software manufacturer. The tested product is for customer management and product pricing. This section describes the steps followed during the research, highlighting the team members’ participation in the process.
Below, for better understanding, a BPMN diagram of the steps of this research is presented in
Figure 1.
When the research topic was defined, the method that achieved the relationship proposed as a general goal between the ISO/IEC/IEEE 25010 standard and BDD was chosen, followed by the elaboration of research questions, the case study carried out, a list of threats to validity how we mitigate them, and, finally, conclusion.
4.1. Case Study
We applied this case study to a system with mixed CRM and people-management characteristics, so we chose it due to its relevance to the company. In addition, the systems only adopt BDD for functional requirements and do not cover non-functional ones, which, according to the quality team, lacks information for tests to be carried out, generating retests in several case. A multidisciplinary team was involved during the case study with the following team members:
A software architect responsible for defining the architecture and non-functional requirements;
Two developers in charge of implementing the acceptance scenarios;
A team lead, responsible for leading the process and helping with communication between team members;
A test analyst tasked with automating performance tests using Locust in Python;
A DevOps, responsible for configuring the infrastructure and deploying the system.
The software architect created user stories and acceptance scenarios that described non-functional performance requirements based on the characteristics of the ISO/IEC/IEEE 25010 standard, following the gherkin language. These user stories and acceptance scenarios were presented to the team and relevant stakeholders to gain their approval, ensuring a common understanding of the non-functional requirements.
The tests were automated after the user stories were written, and the acceptance and approval criteria were set by those involved. The test analyst was responsible for automating performance tests using the Locust tool in Python, ensuring that acceptance scenarios were translated into executable tests.
The tests were carried out, and the metrics available in the acceptance criteria were monitored using analysis tools. During the execution of performance tests, the multidisciplinary team, including the software architect, test analyst, and DevOps, tracked the results and metrics generated by monitoring and analyzing the results using Grafana and Jaeger tools.
4.2. Interview
Before executing the automation tests, one of the authors interviewed the team lead who used BDD in this case study. In a brief conversation, the team lead informed us of how the requirements were approved by interested parties to achieve the behavior expected by the software. Furthermore, he mentioned the importance of integration between the team, the process, and the quality of the software, aiming to obtain releases that meet what was requested and stating the importance of using BDD from the elicitation of the requirement to the execution of the test cases.
In order to guarantee the performance efficiency characteristic expressed in the ISO/IEC/IEEE 25010 standard, as well as its sub-characteristics, based on the elicitation of requirements, user stories were created with their respective acceptance criteria reported in the following section.
5. Case Execution
The execution of the case study is based on the functionalities and scenarios defined following the ISO/IEC/IEEE 25010 standard using BDD. The case study intended to evaluate the system’s performance efficiency concerning three sub-characteristics: time behavior, resource utilization, and capacity. User stories and acceptance criteria, following the gherkin language, a standardized and widely adopted language in BDD frameworks are described below. This standardization ensures the reliability and consistency of these concepts, instilling confidence in their use. Each subsection corresponds to one of the sub-characteristics addressed in this study in relation to the ISO/IEC/IEEE 25010 standard.
Figure 2,
Figure 3 and
Figure 4 below were extracted from Locust and are part of the BDD automation.
5.1. User Story: System Efficiency (Time Behavior)
Description: As a system administrator, the team seeks to ensure the system can handle 100 concurrent users for 7 days, ensuring efficiency is consistent during periods of high demand.
Scenario: Time behavior test.
Given that the system is working correctly and can accommodate 100 simultaneous users, as shown in
Figure 2.
Figure 2.
Number of users.
Figure 2.
Number of users.
When 100 simultaneous users access the system for 7 consecutive days and each user creates an opportunity every 5 min, as shown in
Figure 3.
Figure 3.
CPU and memory consumption during test execution, with low memory consumption and some peaks in CPU consumption.
Figure 3.
CPU and memory consumption during test execution, with low memory consumption and some peaks in CPU consumption.
Then the system must maintain an average response time of less than 5 s and not present critical errors during the test, and the use of server resources must remain below 80%, as shown in
Figure 4.
Figure 4.
Time to response.
Figure 4.
Time to response.
5.2. User Story: Resource Utilization
Description: As a system administrator, the team seeks to ensure that the system can handle 1000 concurrent users for 30 min, ensuring that server resource utilization remains below 85% during load testing.
Scenario: Resource utilization test.
Given that the system is working correctly and has capacity for 1000 simultaneous users.
When 1000 simultaneous users access the system for 30 min.
Then server resource utilization should not exceed 85% and the system should not experience critical errors during testing.
5.3. User Story: Capacity
Description: As a system’s administrator, the team seeks to ensure that the system can handle 2700 concurrent users for 30 min, ensuring that system capacity is sufficient to meet demand during peak usage.
Scenario: System capacity test.
Given that the system is working correctly and has a capacity for 2700 simultaneous users.
When 2700 simultaneous users access the system for 30 min.
Then server resource utilization should not exceed 90%, and the system should not experience critical errors during testing.
6. Results
Below are the answers to the research questions presented in
Table 1.
6.1. QP1—How Does BDD Address Difficulties in Ensuring Non-Functional Requirements?
The BDD framework intended for test automation, such as Cucumber and JBehave, was initially designed to address the system’s functional units, which resulted in significant challenges in the integration between user stories and their corresponding scenarios and subsequent automation. However, as evidenced in this investigation, it is possible to considerably mitigate the difficulties associated with the practical application of BDD in contexts involving non-functional requirements by adopting complementary tools.
In the present study, we opted for the combined use of Pytest-BDD in conjunction with Locust. While the former makes it possible to apply BDD in Python environments, the latter facilitates performance testing. As a result, an integration enables the validation of the acceptance criteria established by the BDD, focusing on evaluating the system’s performance. This approach demonstrates that adopting complementary tools, such as Locust, allows the framework adopted for BDD to validate the acceptance criteria efficiently, providing more excellent reliability to the results obtained.
Adopting BDD combined with complementary tools emerges as a complete and adequate procedure to satisfy the system’s non-functional requirements, allowing efficient compliance monitoring with the proposed scenarios.
6.2. QP2—How Can BDD Be Used to Ensure Quality Related to the Performance Efficiency Characteristic of the ISO/IEC/IEEE 25010 Standard?
Solis and Wang [
27] identified essential characteristics of BDD, highlighting scenario models and automated acceptance tests with mapping rules as preponderant elements. Such characteristics play a fundamental role in ensuring software quality. In this study, we adopted a set of tools based on user stories and acceptance criteria, enabling test automation and monitoring of user perspectives. To this end, we used Locust to evaluate system performance and Jaeger to analyze communication between services, as illustrated in
Figure 5.
Figure 5 presents Jaeger. This system shows communication between services, demonstrating performance and how each microservice communicates, allowing us to understand if there is an internal problem between services. This approach facilitates the work that links the need for performance that BDD pointed out and the possibility of verifying the behavior, not just of the response but also of the components linked to the response.
Figure 6 shows Grafana analyzing the containers of running services, allowing us to understand memory and processor consumption during tests, thus allowing us to cross-reference information from the point of view of the user with Locust, the service with Jaeger, and the server with Grafana, with a 360 view of the system’s performance.
The assessment of infrastructure components, monitored by Grafana, provided a comprehensive and end-to-end view of the system’s performance.
6.3. QP3—What Are the Benefits of Using BDD in Identifying and Automating Non-Functional Tests?
The application of BDD to ensure that the quality of a system’s characteristics offers a series of advantages, particularly highlighting the substantial improvement in communication between the architect and the product development team. Furthermore, its intrinsic integration with automation processes provides greater transparency and ease in validating the system, ensuring that it effectively fulfills the characteristics outlined during the design process.
As evidenced in
Section 4.1, BDD becomes more transparent and accurate in the elicitation of requirements when using the gherkin language. In this way, we achieved the behaviors expected by the software, achieving success in automating non-functional tests. Furthermore,
Figure 1 shows the importance of acceptance criteria meeting the tests to guarantee the quality initially proposed by eliciting the requirement.
7. Discussion
The execution of these test scenarios made it possible to evaluate the system’s performance concerning the sub-characteristics inherent to the performance efficiency characteristic of the ISO/IEC/IEEE 25010 standard and ensure that the defined non-functional requirements were met. Thus, scripts were created for testing in Python with Locust.
As mentioned by Haoues et al. [
7], to maintain software quality following the ISO/IEC/ IEEE 25010 standard, there is a need to focus on the quality aspect throughout the entire cycle of software life. In this way, BDD managed to present positive results from eliciting requirements, carried out objectively among team members, to the automation of tests to meet the proposed acceptance criteria.
According to Estdale and Georgiadou [
8], non-functional requirements present improvements regarding the functioning of the software. As they are considered more complex, it is expected to have difficulty in eliciting non-functional requirements. Through the gherkin language used by BDD, there was a better understanding of what to expect from the software’s behavior, so user stories were described with a clear purpose.
According to Jarzkebowicz and Weichbroth [
9], it is necessary to understand how non-functional requirements are elicited. Through this study, we perceived the necessary steps for the elicitation of non-functional requirements, as shown in
Figure 1, in order to realize the need to validate the requirement through user stories and their respective acceptance criteria to identify whether what was expected by the software was achieved or not.
Regarding the systematic literature review with a focus on non-functional requirements carried out by Olsson, Sentilles, and Papatheocharous [
10], the authors observed the need to carry out studies that validate what has been researched in academia in order to deliver answers to assertions consistent with the reality of the industry. In this way, this research contributed to understanding the elicitation of non-functional requirements with BDD in the current situation.
As North [
13] mentioned, BDD was created to mitigate problems arising from Test-Driven Development (TDD), so BDD is focused on the behavior of releases, while TDD focuses directly on testing. Thus, the author also states that BDD has the benefits of improving communication and readability, which are aspects perceived in this research in the elicitation of non-functional requirements, so there was the participation of the parts involved in the development of releases, such as what was described in
Section 4.
9. Conclusions
In this study, the BDD framework was adopted for the elicitation of non-functional requirements and the ISO/IEC/IEEE 25010 standard, specifically the performance efficiency characteristic with its three sub-characteristics, namely time behavior, resource utilization, and capacity. Each subcharacteristics was associated with a user story, accompanied by one or more acceptance scenarios, as exemplified in
Section 5. This approach made it possible to guide user testing, establishing links between efficiency characteristics and how requirements are being elicited through the BDD gherkin language. For the company using BDD to align non-functional requirements is crucial, as it facilitates product approval. Furthermore, it fosters closer integration between the Quality team and the product since BDD requirements are directly linked to the product.
Furthermore, as it is written in gherkin language, BDD contributed to eliciting non-functional requirements about BDD’s objectivity for writing focused on the system’s behavior. In this way, the use of BDD helped those involved in the requirements elicitation process to mitigate possible ambiguities in writing user stories and their respective acceptance scenarios.
The use of BDD as a framework in the software life cycle leads to improved communication on the part of the members involved in the process to improve the software development stages, from requirements elicitation to documentation and maintenance. This research brought a holistic approach to the system’s quality requirements by relating BDD to the ISO/IEC/IEEE 25010 standard. Through this study, BDD showed its effectiveness following the performance efficiency characteristic expressed in the ISO/IEC standard /IEEE 25010 as per
Section 6.
The obstacles encountered in this study were mitigated according to
Section 8 to guarantee the results’ quality. The research questions expressed in
Table 1 were answered according to
Section 6, in addition to the process used for the elicitation of requirements and the test automation being detailed, aiming to clarify the step-by-step process used so that other researchers can replicate it.
As BDD is a framework used from the elicitation of requirements to software implementation and maintenance, communication between stakeholders and the development team is necessary to contribute to the quality of the software delivered. Thus, when using BDD in software development, relevant people can present its importance so that BDD can achieve its goal proposed by Dan North: a framework that assists in communication and delivers releases according to the expected behavior. Therefore, this study involves people, processes, and techniques to maintain the system’s quality entirely.
As suggestions for future work, the methodology used in this research could be replicated in a new study with teams that use BDD using other quality characteristics of the ISO/IEC/IEEE 25010 standard. A survey could also be carried out with a focus on the quality characteristics of the ISO/IEC/IEEE 25010 standard with professionals who use BDD in order to identify whether software development is based on the quality characteristics expressed in the ISO/IEC/IEEE 25010 standard. Finally, an experiment could be carried out that compares the use of BDD to elicit non-functional requirements with and without the ISO/IEC/IEEE 25010 standard as a reference to observe the impact caused by the quality expressed in the standard.