Assessing the Sustainability Performance of Sustainability Management Software

: Companies have made considerable progress in assessing the sustainability of their processes and products, including the information and communication technology (ICT) sector. However, it is surprising that little attention has been given to the sustainability performance of software products. For this article, we chose a case study approach to explore the extent, to which software manufacturers have considered sustainability criteria for their products. We selected a manufacturer of sustainability management software on the assumption that they would be more likely to integrate elements of sustainability performance in their products. In the case study, we applied a previously developed set of criteria for sustainable software (SCSS) using a questionnaire and experiments, to assess a web-based sustainability management software product regarding its sustainability performance. The assessment ﬁnds that despite a sustainability conscious manufacturer, a systematic assessment of sustainability regarding software products is missing in the case study. This implies that sustainability assessment for software products is still novel, corresponding knowledge is missing and suitable tools are not yet being widely applied in the industry. The SCSS presents a suitable approach to close this gap, but it does require further reﬁnement, for example regarding its applicability to web-based software on external servers.


Introduction and Motivation
Companies in the information and communication technology (ICT) sector increasingly consider sustainability issues in the way they conduct business [1][2][3]. Several companies jointly work towards sustainable developments in industry initiatives, such as the Global e-sustainability initiative [4]. Relevant topics include the effects of ICT throughout the life-cycle and supply chain of products and services [1,3,5], such as health and safety as well as energy issues.
Several analyses have investigated the effects that ICT has on society and the environment. The results were ambiguous, found to be positive, negative, or even to cancel themselves out [6,7]. In general, the effects of ICT on the environment were categorized into three effects [8][9][10]: First order (direct effects) derive from the physical existence of ICT and the services they provide. In the life-cycle phases of manufacturing or recycling, ICT hardware poses a risk by exposing humans and the environment to toxic materials, e.g., e-waste [1,7,11]. Second order (indirect effects) are the result of the constant use of ICT services, including transparency and increase of speed. Third order (also indirect) effects are aggregated long-term outcomes of using ICT. Yi and Thomas [12] summarized For software, a sub-section of ICT, a core difference in terms of sustainability is whether the software product itself can be called "sustainable" rather than providing the opportunity for more sustainable behavior and processes ("sustainable by software"). The latter refers to software that is sustainable due to its ability to create positive contributions to sustainable development, based on its use and application. For example, software for the sustainability management of organizations belongs to the aspect "sustainable by software". The focus of this inquiry is whether approaches for sustainability by software (e.g., sustainability management software) is in itself a "sustainable" software. To explore this, the paper uses a case study approach focusing on one manufacturer of sustainability management software for enterprises. This paper analyzes, evaluates, and categorizes the different aspects of sustainable software by applying a previously published framework for assessing sustainability software [16], which is introduced next (Section 2). A summary of the case study methodology follows (Section 3) and leads to the results (Section 4). The results and subsequent implication for research and practice for the sustainability assessment of software are critically discussed (Section 5). The manuscript concludes with a summary and potential future research questions (Section 6).

Sustainability Assessment of Software
The sustainability assessment of the software, remains, despite several calls for research, a niche topic [17][18][19][20]. Indeed, based on the abilities and characteristics of software, the hardware capacities are determined, affecting the resource consumption during its usage phase (e.g., of end-user devices, networks and data centers [21]). In order to assess sustainability of software, the first step is to define and characterize sustainable software products [19,[22][23][24][25][26][27][28][29]. This paper builds on the definition of sustainable software by Dick et al. [30] that sustainable software positively contributes to sustainable development including its economic, societal and environmental effects and is sustainable itself. Furthermore, research has identified a multitude of possible criteria to assess sustainable software products. Existing approaches include quality models for software, e.g., ISO 25010, or built on the life cycle assessment (LCA) approach. Calero et al. [26,31] propose an extended version of the ISO model, called ISO 25010+S that includes sustainability issues by defining an additional quality aspect "greenability". They propose criteria such as energy efficiency, resource optimization, capacity optimization, and perdurability. Naumann et al. [5] present a life cycle of software products and map effects regarding sustainable development such as working conditions, download size, accessibility, hardware requirements, and backup size across the life cycle. The causal model published by Kern et al. [16] relates observable software properties (e.g., resource management functionality, user autonomy) to hardware aspects, issues of behavior and organization, and natural resources. In line with the causal model, a set of criteria for sustainable software (SCSS) by Kern et al. [16] is proposed, building on an extensive review of the academic literature on relevant criteria for green and sustainable ICT. The SCSS follows a hierarchical order, presenting indicators to assess the impacts of software on sustainable development, focusing on the usage phase of software [5]. It encompasses three main aspects to assess the impact of software on natural resources from a life-cycle perspective: Resource efficiency, potential hardware operating life, and user autonomy. Each of them is further differentiated with corresponding sub-criteria (see Table 2). Table 2. Set of criteria for sustainable software products (SCSS): main criteria and corresponding sub-criteria [16]. The SCSS is available online at: [32]. Transparency and interoperability: Can users understand resource-relevant aspects of the software product with a reasonable amount of time and effort? Are they free to re-use data they produced with this software product with other software products?
3.2 Uninstall ability: Can the software product be uninstalled easily, without leaving traces, and without avoidable disadvantages?

3.3
Maintenance functions: Does the software product provide easy-to-use functions permitting users to repair damage to data and programs?
3.4 Independence of outside resources: Can the software product be operated as independently as possible of resources not subject to the users' control?
3.5 Quality of product information: Does the information provided about the software product support its resource-efficient use?
For a better understanding if and how sustainability-conscious companies consider sustainability for their own product, this paper applies the SCSS to the case of a manufacturer of sustainability management software. The paper focuses on a web-based software, including experiments to assess the sustainability of the software (e.g., network traffic) and additionally a questionnaire and interview data. The analysis ensures reliability by reviewing verified product information and user manuals. With this approach, we applied the SCSS in a new context and, therefore, also critically reviewed it.

Methods
To answer the research question of the article, a single case study design [33] was applied. Based on the lack of theories and analyses of sustainability in software, a case study is a suitable approach for exploring this area [33,34]. Following the six-step process by Ying [33], the research question "How can sustainability management software itself be assessed in terms of sustainability" was developed (step 1). Regarding the design (step 2 [33]) a single case study approach was chosen, to explore the sustainability of a web-based software for sustainability management, which is the unit of analysis. For the exploration, the previously introduced SCSS [16] was chosen to investigate the selected sustainability management software: assessment of the sustainability software, based on the SCSS and thus focusing on the usage phase of software. To do so, we critically reviewed the SCSS, published in [16], regarding its application to web-based software. Table 3 shows the selected criteria from the SCSS applied to assess the WeSustain Enterprise Sustainability Management (ESM) software. For this paper, we chose to focus on qualitative criteria as previous publications predominantly focused on measuring energy consumption of software [16,[35][36][37]. Table 3. Selection of the criteria that are included in the sustainability assessment of the WeSustain Enterprise Sustainability Management (ESM) software against the SCSS ( √ = applied; (a)-(d) = named indicator applied; -= criterion not applied). For the preparation of the sustainability assessment (step 3 [33]) one web-based software was carefully selected. Rather than focusing on large scale software solutions for corporations, a focus was set on software suitable for small and medium-sized enterprises (SME) as the majority, 99.8 percent of companies in the EU, are SMEs [38]. The software products were selected from a list of web-based sustainability software for SMEs presented by Johnson et al. [39], with a focus on integrated sustainability aspects rather than a sole selection of ecological aspects, narrowing the list from ten to five. Furthermore, web-based sustainability management software products from the category "assessment and reporting" were chosen and the manufacturers were asked for collaboration, finally resulting in the selection of CR Kompass. After reviewing the CR Kompass with the manufacturer, WeSustain GmbH, the decision to rather focus on their software product WeSustain ESM, (ESM stands for Enterprise Sustainability Management) was made due to its higher flexibility and validity. To collect the data (step 4 [33]), the selected web-based sustainability management software was assessed, based on the SCSS [16]. With a focus on relevant criteria for web-based tools and to consider at least one criterion from each main level criteria (level 1,

Transparency and interoperability
The case study furthermore included an experiment of 30 measurements for network traffic, following a typical use case (standard usage scenario; necessary for application of the SCSS by Kern et al. [16]: A detailed version of the usage scenario is included in the replication package of this article: [40] The scenario followed the measurement method presented in [35]. The KPIs were selected randomly from the GRI G4 indicators as these are mostly frequently used by the users of WeSustain EMS, according to WeSustain GmbH. The results of the experiment were complemented with interviews and questionnaires with representatives of the manufacturer. To analyze the data (step 5 [33]), we reviewed the case in depth regarding strengths, limitations and further research for assessing sustainability in web-based software, in particular with the application of the SCSS. Finally, the last step [33], is to share the data, e.g., in this publication. The results were also shared with WeSustain GmbH, the manufacturer of the web-based software product that had been assessed.

Results
The application of the SCSS to the case of WeSustain GmbH and its ESM software reveals that the sustainability management software itself has not been systematically assessed with regards to its sustainability performance. Rather, selected sustainability initiatives, e.g., sustainable servers, demonstrate the relevance of sustainability in software for this case. This section presents key results of the case study of applying the SCSS to the web-based sustainability management software by WeSustain GmbH.
The detailed results of the case study, including experiment and questionnaire, are attached in the Supplementary Material of the article and online [40].

Hardware and Energy Efficiency: Measuring Network Traffic
To assess network traffic, the SCSS suggests reviewing the recommended minimum (indicator 1.1.1 (e) and 1.1.2 (e)) network bandwidth, as well as measuring the average bandwidth utilization for network access in idle mode (indicator 1.1.3 (d)) and when running the usage scenario (indicator 1.1.4 (d)).
According to the manual and the answers of the manufacturer in the questionnaire, "there are no technical requirements" beyond those gathered in criterion 2.1 backward compatibility [41]. Hence, indicator 1.1.1 (e) and 1.1.2 (e) cannot be assessed. The average bandwidth utilization in idle mode is too small to be distinguished from the baseline measurements (indicator 1.1.3 (d)).
To assess the indicator 1.1.4 (d) Measurement of average bandwidth utilization for network access when running the standard usage scenario under the standard configuration, the average network traffic of a system under test (SUT) was measured, running the standard usage scenario under the standard configuration as described by Kern et al. [16] and Guldner et al. [35]. The measurement results are depicted in Figure 1. The detailed results of the case study, including experiment and questionnaire, are attached in the Supplementary Material of the article and online [40].

Hardware and Energy Efficiency: Measuring Network Traffic
To assess network traffic, the SCSS suggests reviewing the recommended minimum (indicator 1.1.1 (e) and 1.1.2 (e)) network bandwidth, as well as measuring the average bandwidth utilization for network access in idle mode (indicator 1.1.3 (d)) and when running the usage scenario (indicator 1.1.4 (d)).
According to the manual and the answers of the manufacturer in the questionnaire, "there are no technical requirements" beyond those gathered in criterion 2.1 backward compatibility [41]. Hence, indicator 1.1.1 (e) and 1.1.2 (e) cannot be assessed. The average bandwidth utilization in idle mode is too small to be distinguished from the baseline measurements (indicator 1.1.3 (d)).
To assess the indicator 1.1.4 (d) Measurement of average bandwidth utilization for network access when running the standard usage scenario under the standard configuration, the average network traffic of a system under test (SUT) was measured, running the standard usage scenario under the standard configuration as described by Kern et al. [16] and Guldner et al. [35]. The measurement results are depicted in Figure 1.    sustainability management software. Generally, the data traffic induced by the software could possibly be reduced by using application level cache, optimized images and compression, as shown by Dick et al. [43].

Resource Management
The criterion resource management addresses the question "To what extent does the software product contribute to efficient management of the resources it uses during operation?" Table 3 summarizes the four sub criteria and corresponding indicators.
The analyzed software product does not allow for the adaption of hardware capacities to the current supply (criterion 1.3.2). As the assessed software does not provide default settings, the corresponding indicator of criterion 1.3.3 default settings supporting resource conservation cannot be analyzed. This criterion reviews if the default settings of the software product take the goal of resource conservation into account [16]. The corresponding indicator 1.1.3 (a) is reviewer's assessment whether the default settings of the software product are selected in such a way that they also take the goal of resource conservation into account. Furthermore, the software does not support energy-friendly modes by providing corresponding settings. Additionally, the user does not receive feedback on the usage of hardware capacities and energy (indicator 1.3.4 (a)). Thus, so far, the analyzed software product does not take resource management issues into account.
This offers great potential for improving the software product regarding its resource management. These results are common and resemble the assessment results of other software products, e.g., internet browsers, word processing, content management systems and databases [16,35]: Apart from a sleep mode that is started after a maximum of about 60 s, settings for resource management are not included in the previously tested software products, either. Web browsers provide feedback on use of hardware capacities and energy. For databases, feedback on the use of hardware capacities and energy, is, in two-out-of-three products, to some extent, presented [16,35]. Using energy management functions does not influence the usability of the analyzed web browsers, content management systems and databases.
These previous results arose from the assessment of non-sustainability-oriented software. Therefore, a different outcome was expected based on a more sustainability-oriented software manufacturer. Similar results across sustainability and non-sustainability-oriented manufacturers, however, show the novelty of this field, both in research and practice.

Potential Hardware Operating Life
The second criterion addresses the question "To what extent are hardware replacement cycles decoupled from software replacement cycles?", encompassing four sub-criteria and corresponding indicators.
To analyze the backward compatibility (criterion 2.1), we asked the manufacturer which hardware-old(er) operating systems and old(er) frameworks-are supported by the software product. The software manufacturer states that 95% of the customers use the software product in the cloud. Hence, using WeSustain ESM requires, according to the manual and the manufacturer, a current web browser, i.e., Internet Explorer version 8.0 or later, Firefox version 7.0 or later, Chrome version 15.0 or later, or Safari version 5.0 or later. For a comparison of the software product, the SCSS [16] suggests testing the product on different standard configurations: on a current version and an earlier one. Since this is the first application of the SCSS on this software product, this additional indicator was neglected.
As specified by the manufacturer, the software can be executed on the operating systems Linux and Windows. The product is a web-based tool and was thus not installed nor executed on the server of the testing environment during the assessment. Therefore, the standard usage scenario was not run on different server systems. However, regarding the client side, WeSustain ESM was successfully run on Windows using the internet browsers Google Chrome Based on the results of the questionnaire, the manufacturer does not compare the amount of hardware capacity used when updating and extending the software product. Thus, hardware of sufficiency (criterion 2.3) was not rated during the case study. Similarly, the criterion 2.3 had to be excluded from the assessment [16,35] since it was the first analysis.
Summarizing, the software product is, since it is web-based, highly platform independent and portable. Indeed, so far, the manufacturer was not aware of issues regarding backward compatibility and hardware sufficiency. Compared to previously assessed desktop software [16,35], it is more complex to analyze the amount of used hardware for web-based software, which is in line with previous results, e.g., Kreten et al. [44].

Transparency and Interoperability
The criterion transparency and interoperability addresses the questions "Can users understand resource-relevant aspects the software product with a reasonable amount of time and effort? Are they free to re-use data they produced with this software product with other software products?" It encompasses four sub-criteria and corresponding indicators. The criteria, their corresponding questions and indicators as well as the results of the assessment of the WeSustain ESM software are presented in Table 4 in detail. Table 4. Results of the assessment of the criterion "3.1 Transparency and interoperability" to the sustainability-oriented web-based software product "WeSustain ESM" (* information found in manuals and suchlike, added to the answers of the WeSustain GmbH on the questionnaire). Not included in the analysis (d) Is it possible to receive differential updates only?

Criterion and Corresponding Question
Not included in the analysis Table 4. Cont.

Transparency of Task Management
Does the software product inform users that it is automatically launching or running tasks in the background that are possibly not being used?
(a) On the basis of the installation and the execution of standard usage patterns, test which processes are automatically launched by the software product and whether it informs users of this (Scale: informs users of all such processes/informs users of some such processes/does not inform users) Not included in the analysis (b) If the software product is automatically launched at system start ("autostart"): does it inform users that this is the case?
Not included in the analysis (c) If the user carries out an action that can be understood as ending the program, but at least one of the tasks remains active: does the software product inform the user that this is the case?
Automatic logout when closing the browser; software informs user if there is unsaved data

Discussion
Sustainability management software can be seen as a facilitator to more sustainable behavior, but it can also be seen as an example for sustainability efforts by being sustainable itself. The application of the SCSS to the web-based software "WeSustain ESM" offers one approach for how sustainability management software itself can be assessed in terms of sustainability (research question). The questionnaire used to assess the sustainability of the software identified information on how the software manufacturer addresses sustainability in their products. Additionally, the article presents a critical reflection of the SCSS regarding web-based software executed on an external server. Both aspects are discussed in the following section.

Consideration of "Sustainable Software" by the Software Manufacturer
The case study confirms results from previous studies, that the sustainability assessment of sustainable software seems to still be in its infancy [45], because even software manufacturers with products that focus on sustainability, seem to lack the abilities to include these considerations in the software development process. A sensitivity of the manufacturer in our case study for sustainability is demonstrated by the consideration of data centers that use renewable energy (see Section 4.1). Nevertheless, other sustainability consideration for the software product itself seem to be neglected. Therefore, the sustainability assessment of sustainable software seems to be a novel approach and future research should investigate how these considerations can broadly be made available.
Generally, software developers are challenged to more strongly integrate the ideas of the SCSS, e.g., options to reduce the resource and energy consumption, into their software development processes.
Future research should complement the case study with broader empirical results. A focus could be on the market for sustainability management software, interrogating the question "How are aspects of sustainable software considered by software manufacturers providing sustainability management software?" This case study indicates a low focus and lack of appropriate tools such as the SCSS in practice. The research is not only beneficial in terms of understanding the status in practice, but could, with the help of accompanying workshops, show possible sustainability assessment approaches and demonstrate the relevance and benefits in practice, as well. However, the market for sustainability management software is just one possibility to extend the research regarding sustainability issues of software products. Further examples include public authorities, software companies offering tools for earth observation or for analyzing environmental data.
Practical implications from this research include ideas for the optimization of the software product, complimentary to the technical recommendations published by Dick et al. [43] (see Section 4.1). In order to provide specific indications for improving the sustainability of web-based tools, it is necessary to extend the research by accessing the server on which the software is hosted.

Application of SCSS to Web-Based Software
A novel focus of this case study is the application of SCSS to a web-based software. Therefore, this section critically reviews and discusses the applicability of the SCSS to web-based software, suggesting implications for further research.
In general, web-based software indicates a reduction of environmental impacts. Dick et al. [46] highlight that "Green Web Engineering describes the art of developing, designing, maintaining, administrating, and using a website in such a manner, that direct and indirect energy consumption within the complete life cycle of a website is reduced". The SCSS includes green web aspects by addressing network traffic in different criteria and corresponding indicators (see section 0 of this article). While assessing these aspects in the case study, the SCSS was critically reviewed regarding the application to web-based software that is hosted on external servers (black box assessment) and, thus, direct measurements of the server side is impossible.
This results in the elimination of the following criteria from the assessment, because they have no or limited applicability to web-based software:

Conclusions and Outlook
This paper extends the research by assessing sustainability management software by means of the set of criteria for sustainable software (SCSS), thereby applying the SCSS to web-based software. Thus, it provides innovative results by connecting the aspects of "sustainable by software" and "sustainable software". This case study confirms previous publications that sustainability aspects of software are rarely considered. For this case of a sustainability management software manufacturer, first approaches to sustainable software, e.g., by using sustainability-oriented data centers, are indicated, pointing to the willingness to include sustainability considerations in their software products but missing the necessary tools for assessment and potential for improvement.
The SCSS presents such a tool. By applying the SCSS to a novel environment, a web-based software run on external servers, the SCSS was critically evaluated regarding its applicability to this context. This pointed to several areas for improvement, including more precise definitions in order to more easily apply the SCSS to web-based software, e.g., by providing information about which criteria is suitable for what kind of software products: local software application, software application with external data storage, software application with external execution, or server application.