An Action Research for Improving the Sustainability Assessment Framework Instruments †
- Second, a replication of our original empirical study supported by action–research methodology [24,25] to validate/evaluate SAF in terms of applicability, and perceived efficacy (ease of use and usefulness). The SAF validation results as a consequence of applying the SQ Model and the architectural Decision Map notation in a software project that focuses on the achievement of sustainability goals.
- Third, we propose some improvements that should be considered for increasing the acceptance of the SAF instruments (Decision Map and SQ model) in practice.
2. Related Work
3. The SAF Model
4. Research Method
4.1. Goal and Research Questions, and Variables
- How applicable is the framework to assess if the relevant sustainability–quality concerns have been covered in the software architecture at hand?
- How applicable is the SQ Model to identify requirements as sustainability–quality concerns from a software architecture perspective?
- How applicable is the Decision Map to frame the relevant sustainability–quality concerns of the software architecture at hand?
- How easy to use and useful is perceived the framework to assess if the relevant sustainability–quality concerns have been covered in the software architecture at hand?
- Applicability. In Method Engineering research, applicability is investigated based on the situation in which it is applied . According to Kitchenham et al. , applicability is defined as “focusing on specific cases in which a method is used”. Given that the applicability of a method can only be known when the method is used, action research has been used as a means to validate both SAF instruments to a specific case. Improvements to the framework that are realized as an outcome of implementing the action research were also considered as part of RQ1. Figure 4 shows the possible states that can be determined as a result of applying the SQ model (RQ1.1): Quality Attribute (QA) discovered (orange cell), QA covered (green cell), QA missing (red cell), and n/a when a QA is not observable (grey cell). These states are used in Section 5 and Tables 2 and 5 to report the results of the first research cycle (RC1).
- Perceived ease of use. This construct was originally defined by Fred Davis , which was instantiated to evaluate the framework as follows: the degree to which a person believes that using the SAF instruments would be free of effort.
- Perceived usefulness. It was also defined by Davis  and instantiated as follows: the degree to which a person believes that SAF will be effective in achieving its intended objective, which is the ability to provide guidance for decision making. To achieve this overall objective, we considered the following specific objectives:
- The SQ model should provide support for mapping the most relevant design concerns. We consider that the usage of the SQ model as a tool for supporting early identification of sustainability–quality attributes should enable the achievement of this objective.
- A created DM should be able to be understood by people not involved in the process. Considering that software sustainability is a broad topic that requires the participation of multiple stakeholders with different perspectives and backgrounds, the achievement of this objective is important to facilitate effective collaboration among the stakeholders.
- The SQ model and DM should help in analyzing the sustainability concerns of software architecture.
4.2. Research Context
4.2.1. The GIRO Project
- Company A: meat by-products treatment not intended for human consumption; in particular, dead animals collection in the northwest of Spain.
- Company B: management and valorization of organic waste transforming it in biogas.
- Company C: management of alarm systems in companies and homes.
- Company D: well-being services for elderly people residing in their own homes.
- Companies E and F: job safety analysis.
- Designer: Designing the instruments of the SAF framework.
- Helper: Using the instruments of the SAF framework to help the software company in getting awareness of the sustainability–quality requirements that were addressed in the project.
- Researcher: Drawing lessons learned about the instruments of the SAF framework (e.g., sustainability–quality model)
- In the first action research cycle (RC1), two participants were practitioners from the Database Laboratory and the software company (Enxenio), who used the SQ model to identify relevant QAs that were not originally considered in the project or missing QAs that were not present in the Model. The roles of these participants were software analyst and software project manager. Besides, a software architect from Enxenio contributed to solving some doubts in the QAs identification process.
- In the second action research cycle (RC2), one of the three participants involved RC1 participated in the construction of the Decision Map for the selected product. This participant was the most senior software analyst, with 21 experience years in the development of Geographic Information Systems (GIS) and 5 years in GIS-Mobile Workforce Management Systems.
4.3. Unit of Analysis
4.4.1. Procedure for the Sustainability–Quality Model
4.4.2. Procedure for the Decision Map
4.5. Experiment Instrumentation
5. Validation of the Sustainability–Quality Model
5.1. Applicability of the Sustainability–Quality Model
5.1.1. Quality Attributes Covered in the Project
5.1.2. Quality Attributes Discovered in the Project
5.2. Improvements in the Model
6. Validation of the Decision Map and SAF Evaluation
6.1. Applicability of the Decision Map
- Scoping sustainability design concerns: Only quality attributes were considered as part of the scope. This might be because the practitioner is led to focus only on the attributes that were present in the SQ model, and in doing so is not stimulated to explore other sustainability concerns that might be relevant for the project at hand. Moreover, we observed that the created DM included several quality attributes that were not directly related to the software architecture of the selected product (e.g., learnability, testability). This observation was also confirmed by the practitioner, who indicated “It is difficult to decide which attributes should be included. There are many, with four dimensions each”. “Sometimes I felt like I was giving structure to the attributes in general instead of describing how the attributes relate to the system”. This feedback confirms that we need to extend the SAF with an instrument that makes explicit the link between the concerns illustrated in a DM and the architecture design elements addressing such concerns.
- Framing the expected impact: Looking at the first version of the DM created by the practitioner, we found that most of the relevant design concerns were not appropriately located in the areas that correspond to the type of impact (namely, immediate, enabling, systemic). For instance, maturity (defined as reliability under normal operation) was considered as a design concern with enabling impact, while it could be measured immediately during the project. Similarly, usefulness was considered as a concern with an immediate impact instead of an enabling impact. In this case, the practitioner did not notice that the achievement of some pragmatic goals (i.e., in the project at hand, optimizing the routes planning) could require a longer time and also additional resources (e.g., the actual number of kilometers used by each truck with a minimum weight). These deficiencies found in the created DM were also confirmed by the practitioner, who indicated: “It was difficult to decide whether an attribute belongs to immediate, enabling or systemic scope of impact”. He also suggested adding some criteria that could be considered to help to frame the right type of expected impact.
- Identifying the type of effects: We observed that only positive effects were considered when some negative effects could have been also identified. Missing other types of effects could be a consequence of the lack of inclusion of other sustainability design concerns in the scope. Identifying a richer set of design concerns can potentially help to identify negative or undecided effects, too. For example, driver tracking is one of these sustainability design concerns that could have harmed the data privacy concern (quality attribute).
6.2. SAF Evaluation Based on Practitioner Perceptions
6.3. Improvements in the Decision Maps
- Recommendation support for navigating the SQ model.
- The practitioner needs support in navigating the SQ model and, when a QA is considered/selected, be proposed with possible related QAs (characteristics and/or sub-characteristics) for further consideration, as well as the ability to compare and contrast similar QAs. A simple recommender could provide the needed support.Besides, so far the SQ model still carries a single definition per QA. Naturally, when a QA is classified in different sustainability dimensions, it is also defined differently. A simple example is resource efficiency, which in the technical dimension can be defined as the efficient consumption of, e.g., CPU time, while in the environmental dimension can be defined as the efficient consumption of energy. Accordingly, we plan to populate the SQ model with the definitions emerging of our study. In this respect, action research will help us refining extending the definitions emerging over time.
- Guidance to correctly position the design concerns in the expected impact.
- As suggested during our study, we plan to make explicit the criteria for a QA to belong to the immediate-, enabling- or systemic-level of impact. Framing such criteria in a checklist, for example, will make it simple for the practitioner to check that each identified concern belongs to the impact level it is expected to have. Also, framing the criteria in an equivalent decision graph could support the identification of the most appropriate level of impact should the practitioner need support in deciding in the first place. Figure 7 shows an example of a decision graph for guiding in the identification of the type of impact (Immediate, Enabling, Systemic—represented as nodes of the decision graph). In this example, the decision graph was built by considering three main criteria—represented as questions in the corresponding diamond-shaped nodes. Answers to these questions are represented as decision edges of the graph.
- Guidance for design space exploration:
- The use of the SQ model seems to steer the attention of the practitioner away from design concerns other than QAs. We need to complement the SAF with an instrument that helps practitioners explore the design space more thoroughly, or systematically. To this aim, we plan to identify triggers for problem space (the design space is defined as the combination of the problem space including the design elements framing the problem, like design concerns, and the solution space including the elements of the design solution, like design decisions.) exploration similar to Tang et al.  (using decision patterns as triggers) and Razavian et al.  (using reflective questions). Our triggers (maybe in the form of a checklist) should stimulate the identification of the dependencies between concerns with positive or negative effects.Lastly, an open research question is how we can support the practitioner in making explicit the link between DMs (e.g., a design concern) and the software architecture elements (e.g., addressing such concern). Future research will focus on defining an architecture viewpoint addressing this important new SAF instrument, which factually will support navigating between problem space and solution space.
- New QAs not been considered in the model were added. From Table 5, it can be seen that many of these new QAs contribute to the social sustainability dimension (i.e., data privacy, timeliness, regulation compliance, tailorability). However, only one QA (tailorability) was identified as a new contributor to the technical sustainability dimension. Overall, this result shows the importance of making explicit which sustainability dimension is relevant for which QA, so that significant metrics can be identified and monitored.
- New direct dependency relations (a direct dependency is defined as a finite set of ordered pairs of QAs, which is reflexive, symmetric and transitive ) were added as a consequence of identifying new contributions to different sustainability dimensions. Table 2 shows the QAs that were considered as part of this type of dependencies (cells are marked with a “+”). For example, the direct dependency between environmental and economic dimensions consists of four ordered pairs, whose QAs are efficiency, availability, modifiability, and reusability. This is because modifiability and reusability were discovered as new contributors to the economic dimension (+).
8. Threats to Validity
- two participants, who are also part of the software company and they could not have provided an objective view of the situation.
- researchers as the designers of the action, who could have interpreted the results positively (selective bias) when reporting the results.
Conflicts of Interest
- Training material. It consists of slides that introduce the basic concepts and elements of the Decision Map (DM) notation. Some examples were also included as part of the training material. This material can be downloaded from the following link: https://bit.ly/2tAtugy.
- Questionnaire. It aims to gather practitioner’s perceptions on ease of use and usefulness of the DM and the SQ model. Our questionnaire focuses on exploring these last two constructs (PEOU, PU), which were instantiated through two types of questions: five closed questions in a 5-points Likert scale (5 strongly agree, 4 agree, 3 neutral, 2 disagree, 1 strongly disagree), and four open-ended questions. Table A1 shows the items that were formulated for understanding the practitioner’s perceptions of ease of use and usefulness of the SAF instruments (SQM and DM).
- DM template. It was created with the Draw.io editor tool, which was used by the practitioner in the first session of the experiment (see Figure A1).
- Reference DM. It was created by the researchers, and used in the second session by the practitioner. Figure A2 shows the reference DM used in the study.
|I1||I found the notation of the decision maps as clear and understandable||x||Ease of use|
|I2||I believe that the decision maps are well-suited to be understood|
by people not involved in the process
|I3||I think the SQ model provides a good support|
for mapping the most important concerns
|I4||I have the feeling that the time expended in creating the decision map was too long||x||Ease of use|
|I5||I think that the sustainability model and the decision maps|
help analyzing the sustainability concerns of the software architecture
- Avgeriou, P.; Stal, M.; Hilliard, R. Architecture Sustainability [Guest editors’ introduction]. IEEE Softw. 2013, 30, 40–44. [Google Scholar] [CrossRef]
- De-Arteaga, M.; Herlands, W.; Neill, D.B.; Dubrawski, A. Machine Learning for the Developing World. ACM Trans. Manag. Inf. Syst. 2018, 9, 9:1–9:14. [Google Scholar] [CrossRef]
- Bates, O.; Thomas, V.; Remy, C.; Nathan, L.; Mann, S.; Friday, A. The future of HCI and sustainability: Championing environmental and social justice. In Proceedings of the CHI EA ’18 Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems, Montreal, QC, Canada, 21–26 April 2018; Volume 2018. [Google Scholar] [CrossRef]
- Penzenstadler, B.; Duboc, L.; Venters, C.C.; Betz, S.; Seyff, N.; Wnuk, K.; Chitchyan, R.; Easterbrook, S.M.; Becker, C. Software Engineering for Sustainability: Find the Leverage Points! IEEE Softw. 2018, 35, 22–33. [Google Scholar] [CrossRef]
- Chitchyan, R.; Becker, C.; Betz, S.; Duboc, L.; Penzenstadler, B.; Seyff, N.; Venters, C. Sustainability Design in Requirements Engineering: State of Practice. In Proceedings of the 38th International Conference on Software Engineering Companion, Austin, TX, USA, 14–22 May 2016; pp. 533–542. [Google Scholar] [CrossRef][Green Version]
- Moghaddam, F.A.; Lago, P.; Grosso, P. Energy-Efficient Networking Solutions in Cloud-Based Environments: A Systematic Literature Review. ACM Comput. Surv. 2015, 47, 64–96. [Google Scholar] [CrossRef]
- Nidumolu, R.; Prahalad, C.; Rangaswami, M. Why sustainability is now the key driver of innovation. IEEE Eng. Manag. Rev. 2015, 43, 85–91. [Google Scholar] [CrossRef]
- Calero, C.; Moraga, M.Á.; Bertoa, M.F. Towards a Software Product Sustainability Model. arXiv 2013, arXiv:1309.1640. [Google Scholar]
- Venters, C.; Lau, L.; Griffiths, M.; Holmes, V.; Ward, R.; Jay, C.; Dibsdale, C.; Xu, J. The Blind Men and the Elephant: Towards an Empirical Evaluation Framework for Software Sustainability. J. Open Res. Softw. 2014, 2. [Google Scholar] [CrossRef][Green Version]
- Lago, P.; Koçak, S.A.; Crnkovic, I.; Penzenstadler, B. Framing sustainability as a property of software quality. Commun. ACM 2015, 58, 70–78. [Google Scholar] [CrossRef]
- Condori-Fernandez, N.; Lago, P. Characterizing the Contribution of Quality Requirements to Software Sustainability. J. Syst. Softw. 2018, 137, 289–305. [Google Scholar] [CrossRef][Green Version]
- Penzenstadler, B. Infusing Green: Requirements Engineering for Green in and through Software Systems. In Proceedings of the Third International Workshop on Requirements Engineering for Sustainable Systems, RE4SuSy 2014, Co-Located with 22nd International Conference on Requirements Engineering, Karlskrona, Sweden, 26 August 2014; pp. 44–53. [Google Scholar]
- Condori-Fernández, N.; Lago, P. The Influence of Green Strategies Design onto Quality Requirements Prioritization. In Proceedings of the International Working Conference on Requirements Engineering: Foundation for Software Quality, Utrecht, The Netherlands, 19–22 March 2018; pp. 189–205. [Google Scholar]
- Chitchyan, R.; Noppen, J.; Groher, I. What Can Software Engineering Do for Sustainability: Case of Software Product Lines. In Proceedings of the International Workshop on Product Line Approaches in Software Engineering, Florence, Italy, 19 May 2015; pp. 11–14. [Google Scholar] [CrossRef]
- Lago, P. Architecture Design Decision Maps for Software Sustainability. In Proceedings of the International Conference on Software Engineering, Software Engineering in Society Track (ICSE SEIS), Montreal, QC, Canada, 29–31 May 2019. [Google Scholar]
- García-Mireles, G.A.; Moraga, M.Á.; García, F.; Calero, C.; Piattini, M. Interactions between environmental sustainability goals and software product quality: A mapping study. Inf. Softw. Technol. 2018, 95, 108–129. [Google Scholar] [CrossRef]
- Bagnato, A.; Rocheteau, J. Towards green metrics integration in the MEASURE platform. In Proceedings of the 4th International Workshop on Measurement and Metrics for Green and Sustainable Software Systems Co-Located with Empirical Software Engineering International Week (ESEIW 2018), Oulu, Finland, 9 October 2018; p. 39. [Google Scholar]
- Rodríguez, A.; Penzenstadler, B. An Assessment Technique for Sustainability: Applying the IMAGINE Approach to Software Systems. In Proceedings of the 2nd International Workshop on Requirements Engineering for Sustainable Systems, Rio, Brazil, 15 July 2013; Volume 995. [Google Scholar]
- Condori-Fernandez, N.; Procaccianti, G.; Ali, N. Metrics for Green and Sustainable Software: MeGSuS 2014. In Proceedings of the 2014 Joint Conference of the International Workshop on Software Measurement and the International Conference on Software Process and Product Measurement, Rotterdam, The Netherlands, 6–8 October 2014; pp. 62–63. [Google Scholar] [CrossRef]
- Condori-Fernández, N.; Lago, P.; Luaces, M.R.; Saavedra, A.; González, L. Using Participatory Technical-action-research to Validate a Software Sustainability Model. In Proceedings of the 6th International Conference on ICT for Sustainability (ICT4S), Lappeenranta, Finland, 10–14 June 2019. [Google Scholar]
- Condori Fernandez, N.; Lago, P. A Sustainability-Quality Model: (Version 1.0); VU Technical Report: Amsterdam, The Netherlands, 2018. [Google Scholar]
- Oyedeji, S.; Penzenstadler, B.; Adisa, M.O.; Wolff, A. Validation Study of a Framework for Sustainable Software System Design and Development. In Proceedings of the 6th International Conference on ICT for Sustainability (ICT4S), Lappeenranta, Finland, 10–14 June 2019. [Google Scholar]
- Condori-Fernández, N.; Bagnato, A.; Kern, E. A focus group for operationalizing software sustainability with the MEASURE platform. In Proceedings of the 4th International Workshop on Measurement and Metrics for Green and Sustainable Software Systems (MeGSuS), Co-Located at ESEM, Oulu, Finland, 9 October 2018. [Google Scholar]
- Wieringa, R.; Moralı, A. Technical Action Research as a Validation Method in Information Systems Design Science. In Design Science Research in Information Systems; Advances in Theory and Practice; Springer: Berlin/Heidelberg, Germany, 2012; pp. 220–238. [Google Scholar]
- MacDonald, C. Understanding Participatory Action Research: A Qualitative Research Methodology Option. Can. J. Action Res. 2012, 13, 34–50. [Google Scholar]
- Brundtland, G.; Khalid, M.; Agnelli, S.; Al-Athel, S.; Chidzero, B.; Fadika, L.; Hauff, V.; Lang, I.; Shijun, M.; Morino de Botero, M.; et al. Our Common Future (Brundtland Report); Technical Report; World Commission on Environment and Development: Cape Town, South Africa, 1987. [Google Scholar]
- Venters, C.; Jay, C.; Lau, L.; Griffiths, M.K.; Holmes, V.; Ward, R.; Austin, J.; Dibsdale, C.E.; Xu, J. Software Sustainability: The Modern Tower of Babel. In Proceedings of the RE4SuSy: Third International Workshop on Requirements Engineering for Sustainable Systems, Karlskrona, Sweden, 26 August 2014. [Google Scholar]
- Yan, M.; Chan, C.; Gygax, A.; Yan, J.; Campbell, L.; Nirmalathas, A.; Leckie, C. Modeling the Total Energy Consumption of Mobile Network Services and Applications. Energies 2019, 12, 184. [Google Scholar] [CrossRef][Green Version]
- Pihkola, H.; Hongisto, M.; Apilo, O.; Lasanen, M. Evaluating the Energy Consumption of Mobile Data Transfer—From Technology Development to Consumer Behaviour and Life Cycle Thinking. Sustainability 2018, 10, 2494. [Google Scholar] [CrossRef][Green Version]
- Hankel, A.; Oud, L.; Saan, M.; Lago, P. A Maturity Model for Green ICT: The Case of the SURF Green ICT Maturity Model. In Proceedings of the 28th International Conference on Informatics for Environmental Protection (EnviroInfo), Oldenburg, Germany, 10–12 September 2014; pp. 33–40. [Google Scholar]
- Becker, C.; Betz, S.; Chitchyan, R.; Duboc, L.; Easterbrook, S.; Penzenstadler, B.; Seyff, N.; Venters, C. Requirements: The Key to Sustainability. IEEE Softw. 2016, 33, 56–65. [Google Scholar] [CrossRef][Green Version]
- Petersen, K.; Gencel, Ç.; Asghari, N.; Baca, D.; Betz, S. Action research as a model for industry-academia collaboration in the software engineering context. In Proceedings of the ACM International Workshop on Long-Term Industrial collaboration on Software Engineering (WISE), Vasteras, Sweden, 15–19 September 2014; pp. 55–62. [Google Scholar] [CrossRef][Green Version]
- Tolvanen, J.P. Incremental Method Engineering with Modeling Tools; Springer: Berlin/Heidelberg, Germany, 1998. [Google Scholar]
- Kitchenham, B.; Pickard, L.; Pfleeger, S.L. Case studies for method and tool evaluation. IEEE Softw. 1995, 12, 52–62. [Google Scholar] [CrossRef][Green Version]
- Davis, F.D. Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology. MIS Q. 1989, 13, 319–340. [Google Scholar] [CrossRef][Green Version]
- Krueger, R. Focus Groups: A Practical Guide for Applied Research; Sage Publications: Thousand Oaks, CA, USA, 1994. [Google Scholar]
- Wixom, B.H.; Todd, P.A. A Theoretical Integration of User Satisfaction and Technology Acceptance. Inf. Syst. Res. 2005, 16, 85–102. [Google Scholar] [CrossRef]
- Assefa, G.; Frostell, B. Social sustainability and social acceptance in technology assessment: A case study of energy technologies. Technol. Soc. 2007, 29. [Google Scholar] [CrossRef]
- Ormeno, Y.I.; Panach, J.I.; Condori-Fernndez, N.; Pastor, O. Towards a proposal to capture usability requirements through guidelines. In Proceedings of the IEEE 7th International Conference on Research Challenges in Information Science (RCIS), Paris, France, 29–31 May 2013; pp. 1–12. [Google Scholar] [CrossRef]
- Tang, A.; Antony, T.; Lau, M.F. Software architecture review by association. J. Syst. Softw. 2014, 88, 87–101. [Google Scholar] [CrossRef]
- Razavian, M.; Tang, A.; Capilla, R.; Lago, P. In two minds: How reflections influence software design thinking. J. Softw. Evol. Process 2016, 28, 394–426. [Google Scholar] [CrossRef][Green Version]
- Becker, C.; Chitchyan, R.; Duboc, L.; Easterbrook, S.; Penzenstadler, B.; Seyff, N.; Venters, C.C. Sustainability Design and Software: The Karlskrona Manifesto. In Proceedings of the 37th International Conference on Software Engineering, Florence, Italy, 16–24 May 2015; Volume 2, pp. 467–476. [Google Scholar]
- ISO/IEC. ISO/IEC 25010—Systems and Software Engineering—Systems and Software Quality Requirements and Evaluation (SQuaRE)—System and Software Quality Models; Technical Report; ISO: Geneva, Switzerland, 2010. [Google Scholar]
|Characteristics||Attributes||Definition According to ||TECH||SOC||ENV||ECON|
|Compatibility||Co-existence||product can perform its functions efficiently while sharing environment and resources with other products.|
|Interoperability||a system can exchange information with other systems and use the information that has been exchanged.|
|Context coverage||Context completeness||system can be used in all the specified contexts of use|
|Flexibility||system can be used in contexts beyond those initially specified in the requirements.|
|Effectiveness||Effectiveness||accuracy and completeness with which users achieve specified goals.|
|Efficiency||Efficiency||resources expended in relation to the accuracy and completeness with which users achieve goals.|
|Freedom from risk||Economic risk mitigation||system mitigates the potential risk to financial status in the intended contexts of use.|
|Environmental risk mitigation||system mitigates the potential risk to property or the environment in the intended contexts of use.|
|Health and safety risk mitigation||system mitigates the potential risk to people in the intended contexts of use.|
|Functional suitability||Functional appropriateness||the functions facilitate the accomplishment of specified tasks and objectives.|
|Functional correctness||system provides the correct results with the needed degree of precision.|
|Functional completeness||degree to which the set of functions covers all the specified tasks and user objectives.|
|Maintainability||Modifiability||system can be effectively and efficiently modified without introducing defects or degrading existing product quality||+|
|Modularity||system is composed of components such that a change to one component has minimal impact on other components.|
|Reusability||an asset can be used in more than one system, or in building other assets||+|
|Testability||effectiveness and efficiency with which test criteria can be established for a system.|
|Performance efficiency||Capacity||the maximum limits of a product or system parameter meet requirements.|
|Resource utilization||the amounts and types of resources used by a system, when performing its functions, meet requirements.|
|Time behaviour||response, processing times and throughput rates of a system, when performing its functions, meet requirements.|
|Portability||Adaptability||system can effectively and efficiently be adapted for different or evolving hardware, software or usage environments.|
|Replaceability||product can be replaced by another specified software product for the same purpose in the same environment.|
|Reliability||Availability||system is operational and accessible when required for use.||+|
|Fault tolerance||system operates as intended despite the presence of hardware or software faults.|
|Maturity||system meets needs for reliability under normal operation.|
|Recoverability||system can recover data affected and re-establish the desired state of the system is case of an interruption or a failure.|
|Satisfaction||Trust||stakeholders has confidence that a product or system will behave as intended.|
|Usefulness||user is satisfied with their perceived achievement of pragmatic goals.|
|Security||Accountability||actions of an entity can be traced uniquely to the entity.|
|Authenticity||the identity of a subject or resource can be proved to be the one claimed.|
|Confidentiality||system ensures that data are accessible only to those authorized to have access.|
|Integrity||system prevents unauthorized access to, or modification of, computer programs or data.|
|Usability||Appropriateness recognizability||users can recognize whether a system is appropriate for their needs, even before it is implemented.|
|Learnability||system can be used to achieve specified goals of learning to use the system.||+|
|Operability||system has attributes that make it easy to operate and control.|
|User error protection||system protects users against making errors.||+||+|
|Accessibility||Accessibility||system can be used by people with the widest range of characteristics and capabilities.|
|Robustness||Robustness||Refers to the capability of the sytem to behave in an acceptable way in unexpected situations|
|Survivability||Survivability||The degree to which a system continues to fulfil its mission by providing essential services in a timely manner in spite of the presence of attacks|
|Session 1||Session 2|
|PEOU1||I found the notation of the decision maps as clear and understandable.||x||x|
|PEOU2||I have the feeling that the time expended in creating|
the decision map was too long.
|Session 1||Session 2|
|PU1||I believe that the decision maps are well-suited to be|
understood by people not involved in the process.
|PU2||I think the sustainability–quality model provides a|
good support for mapping the most important concerns.
|PU3||I think that the sustainability model and the decision maps|
help analyzing the sustainability concerns of the software architecture.
|Data Privacy||Data Privacy||privacy concerns arise wherever personally identifiable information is collected, stored, or used.|
|Timeliness||Timeliness||the fact or quality of being done or occurring at a favourable or useful time.|
|Regulation compliance||Regulation compliance||allows to draw conclusions about how well software adheres to application related regulations in laws.|
|Scalability||Scalability||the ability of a computing process to be used or produced in a range of capabilities|
|Tailorability||Tailorability||system’s capability to allow users to create or enable new configuration of functionality as well as control information provision.|
© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
Condori-Fernandez, N.; Lago, P.; Luaces, M.R.; Places, Á.S. An Action Research for Improving the Sustainability Assessment Framework Instruments. Sustainability 2020, 12, 1682. https://doi.org/10.3390/su12041682
Condori-Fernandez N, Lago P, Luaces MR, Places ÁS. An Action Research for Improving the Sustainability Assessment Framework Instruments. Sustainability. 2020; 12(4):1682. https://doi.org/10.3390/su12041682Chicago/Turabian Style
Condori-Fernandez, Nelly, Patricia Lago, Miguel R. Luaces, and Ángeles S. Places. 2020. "An Action Research for Improving the Sustainability Assessment Framework Instruments" Sustainability 12, no. 4: 1682. https://doi.org/10.3390/su12041682