Testing and Verification of a New Corporate Sustainability Assessment Method for Manufacturing: A Multiple Case Research Study

Abstract

This paper aims at answering the question of how to design a theoretically sound corporate sustainability assessment (CSA) method that can strengthen the ability of manufacturing companies to contribute to global sustainable development (SD). In our effort to answer this question, we conduct a case study to test a newly designed CSA method to develop assessment tools for each of four different case companies. The method combines criteria-based indicators development with qualitative system dynamics modeling based on mental models of decision-makers. This strategy ensures a holistic approach to what corporate sustainability and sustainable manufacturing are. The paper is intended to serve, first of all, as a practical guide to the development of CSA tools. The scientific value of the method is discussed in terms of how it assists in the development of a tool that provides the potential to overcome shortcomings of existing approaches to sustainability assessment and to embrace the complexity of the sustainability concept. From the case study results, we demonstrate the capability of the new method by showing how it satisfies scientific requirements to sustainability assessment and fulfills functions of CSA. We also show how it may overcome some of the observed limitations of existing CSA tools.

1. Introduction

In the last three decades, we have seen increased attention to sustainable development (SD) from governments and businesses, and myriads of attempts to conceptualize sustainable development and put it into practice [1,2,3,4,5,6]. Both successful and unsuccessful attempts have motivated researchers and practitioners to search new pathways to accelerate the transition to sustainability. Extensive work has been done around the role of sustainability assessment (SA) for improving the evidence-base and the quality of decision-making. Sustainability assessment is currently widely recognized as a decision support tool and a governance mechanism for sustainable development [7,8,9]. The research community and international institutions are looking for methods to support the Sustainable Development Goals (SDGs) transition and to measure business contribution to global sustainable development progress [10,11,12,13,14,15,16]. Similarly to PwC [17], which advices business to measure and report the total impact on the environment and society, rather than just outputs, S&P Dow Jones Indices [18] see corporate sustainability assessment (CSA) as one of the enablers to move from awareness to meaningful actions on SDG-aligned business. However, Topple et al. [19] argue that the voluntary nature of conducting sustainability assessments by the private sector can potentially lead to selective interpretation and reporting of sustainability performance outcomes, increasing the danger of greenwashing. There is also a necessity to extend the scope of such assessments beyond a company’s internal performance indicators toward external data on sustainable development trends, risks, and opportunities [20]. Despite these recommendations, the success of the existing efforts is argued to be very limited [21] due to the fact that research on how to translate social and environmental boundaries into contextually-based measures of sustainable development is still in an embryonic stage (ibid). Thus, the translation of societal-level indicators to a corporate context remains one of the most significant challenges in the operationalization of SDGs in the business world. It has been argued that strategically mindful firms with a solid corporate philosophy that values environmentalism are more willing to practice corporate sustainability [22], which is driven by the corporate sustainability strategy [23].

Research about corporate sustainability assessment is still dominated by neoclassical thinking that implies the use of the so-called paradigm of rational and objective knowledge [24]. Therefore, many of the currently prevailing assessment tools are quite technocratic and deterministic by nature, with characteristics of their techno-rational roots and a high level of engineering. Such tools are often considered to be ‘truth machines’, with the ability to deterministically predict the behaviour of systems in the future. The technocratic approach has resulted in significant limitations of CSA as an instrument for better governance of sustainability. This lack of a more holistic view on sustainability and sustainable manufacturing, along with the need to overcome the limitations and strengthen the capacity of the assessment to accelerate the transition to sustainability, calls for a systems-thinking approach to CSA. Such a new, improved approach has to eliminate the observed limitations and systematically address scientific requirements to SA. This requires a holistic view and attention to the dynamics and complexity of the company as a socio-environmental-economic-technological system. The issues of dynamics and complexity are usually approached by different systems-oriented methods; e.g., system dynamics, systems engineering, system of systems, and interpretative systemology [25].

In this paper, we seek to answer the research question of how to design a theoretically sound CSA method that can strengthen the ability of manufacturing companies to contribute to global sustainable development. Thus, the aim of this paper is twofold: first, to apply the recently designed CSA method to the development of CSA tools (for four case companies); second, to evaluate the feasibility of the proposed CSA method to fulfil purposes and functions of CSA, satisfy scientific requirements to SA, and overcome the observed shortcomings of existing CSA tools and challenges of SA in general. Third, based on the result of the test and verification of the new CSA method, we outline recommendations to the developers of CSA methods and tools.

The remainder of the paper is organized, as follows. First, we present a theoretical underpinning for this research, including the scientific requirements to SA and CSA. Second, in the methodology section, we describe a new CSA method (an architecture and application guideline), multiple case study for testing the method, and verification process. Third, the result section includes four CSA tools and presents verification of the method. Finally, we outline recommendations to the developers of CSA and discuss the lessons that were learned from the case study, followed by conclusion and outline for future work.

2. Theoretical Background

2.1. Corporate Sustainability Assessment

Despite being a relatively new research discipline, sustainability assessment already has a coherent body of research that explores a variety of topics. The major topics are SA definitions [26,27], the purpose of assessments [28,29,30,31], requirements to SA [29,30,32,33,34,35,36,37,38], types of SA methods [39,40,41], and criteria to categorize the methods [42,43], shortcomings and challenges of SA [44,45,46,47,48], and applications of SA [49,50,51,52,53,54]. As stated by Pope et al. [55], the term sustainability assessment is used to refer to both ex-post and ex-ante evaluative techniques or processes. However, there is currently little or no consensus regarding the terminology used to refer to sustainability assessment methods. One of the latest categorization frameworks is proposing to classify SA methods according to sustainability context and decision-making context [55]. Sustainability context represents the normativity principle of sustainable development; i.e., underpinning sustainability discourse and representation of sustainability. Decision-making context represents the subject of the assessment (policy, project, company, product., etc.) and a question to be answered by the assessment (alternative selection, contribution to sustainability, etc.).

In the scientific and professional literature, CSA (or denoted SA at company level) as a branch of sustainability assessment has been embraced as an instrument to evaluate organizational performance to assist decision-makers in determining which actions should or should not be taken in an attempt to contribute to sustainable development. CSA, as a type of SA, is focused on the environmental, social, and economic considerations at company level [56]. Donovan et al. [57] define CSA as “a planning tool for the private sector on how to identify, assess, and manage the impacts of their business operations across environmental, social and economic issues”. Schneider and Meins [58] define it as “an approach to measure the contribution of firms to sustainable development”, focusing on contribution rather than impact. Sometimes, CSA is argued to be a replacement for financial performance as the sole measure of corporate success [44]. Maas et al. [59] define the two main purposes of CSA; i.e., to improve organizational performance, and to inform stakeholders about a company’s impact. The former requires detailed, disaggregated information for the exact evaluation of potential internal improvements—in contrast to the latter, which requires aggregated figures of the whole company. Maas et al. also state that sustainability indicators selected by an outside-in approach and for reporting purposes are not necessarily useful for internal decision-making. Therefore, CSA developed based on the outside-in perspective (e.g., taking SDGs as a starting point) may fail to provide internal decision-makers with sufficiently detailed information for the identification of improvement initiatives.

Profound research on the development of CSA for different types of organizations—e.g., SMEs, large companies, manufacturing, and public—has been carried out and the literature on CSA demonstrates that researchers and practitioners have already gained valuable insights into CSA [50,51,60,61,62,63,64,65,66,67,68]. Montiel and Delgado-Ceballos [56] conclude that the corporate sustainability (CS) field is still evolving and different approaches to define and theorize corporate sustainability exist, whereas a standardized method to measure it remains to be proposed. A variety of CSA methods exists in the literature, e.g., [58,69,70], including several ones for manufacturing companies, e.g., [56,71,72]. Some examples of CSA tools in manufacturing are [50,54,60,63,72,73,74,75,76,77,78,79,80,81]. The approaches to the development of tools include fuzzy methods, AHP, system dynamics modeling, indices, and principles-based assessments.

Although extensive research has been done, the scientific community is still inconclusive about many of the shortcomings and limitations of CSA. Some of the potential shortcomings are: (1) CSA often fails to address system performance [41]; (2) CSA tends to mix sustainability performance of the company and sustainability-oriented practices [44]; (3) CSA includes an uncomprehensive and unsystematic list of indicators [46]; (4) an observed inability of many CSAs to provide a practical approach for the companies to identify improvements and possible sustainability-oriented practices [48,82]; (5) the inability of some CSA to capture the complexity of sustainable manufacturing (SM) due to a widely used reductionist approach to CSA [83]; (6) the use of a set of unrelated indicators [84]; and, (7) companies measure what is measurable, rather than what is necessary concerning the given subject or phenomenon [47]. These observed shortcomings affect the ability of CSA to provide a holistic view on the sustainability performance of a company and identify the associated improvement potential. Furthermore, shortcomings and challenges of SA are in general also present in CSA tools, as a type of SA. These shortcomings were previously presented in [85], based on the systematic literature review [26,28,29,36,39,50,51,52,60,62,63,73,77,84,86,87,88,89,90,91,92,93,94,95,96,97].

One of the challenges of CSA is a selection of the ‘right’ indicators since these have a direct effect on the decision-making and the ability of tools to address a company’s context. Companies often select sustainability indicators that are based on their strategy or type of business. Thus, different companies aim to achieve different values and sustainability performance; for example, “even when a company use experts to select the right set of indicators, this step is subjective” [62]. To address this challenge, a branch of criteria-based tools has been researched. Gibson [98] advocates the use of the SA criteria but states that it is a challenge to identify and structure the case- and context-specific considerations that should be integrated with the generic requirements into the overall set of criteria. Similarly, Hallstedt [46] discusses a criteria-based approach to SA as “sustainability criteria can be developed to define the sustainability design space and thereby make more use of the detailed metrics such as indicators”.

The challenge to choose the right set of indicators for CSA in manufacturing is related to the normativity principle of sustainable development, which states that sustainability is a matter of social definition and can change with time and values [49]. When CSA is developed for a manufacturing company, the tool has to represent what defines sustainable manufacturing. However, an extensive list of sustainable manufacturing definitions [99] and hundreds of sustainability indicators for manufacturing [62,90,100] indicate that the question of what to include in the assessment is not yet answered. This issue was addressed by Moldavska and Martinsen [101], who proposed to define sustainable manufacturing as “a complex behavior pattern to which any manufacturing organization should tend to evolve” and which is defined by the criteria for SM. This approach, i.e., to define sustainability through the criteria, is similar to the one proposed by Pope et al. [26], who suggested that sustainability can be represented within SA by means of sustainability criteria.

To address the shortcomings and challenges, the need to shift a worldview from reductionism to complexity has already been stressed by different researchers, e.g., [102]. Halog and Manik [103] concur that SD is a complex phenomenon that cannot be fully covered by the reductionism-oriented tools. Researchers advocate for a more holistic approach as a more viable alternative [41], ensuring that the assessment indicates ‘system performance’ instead of an aggregation of a number of unrelated individual indicators. Veleva and Ellenbecker [60] state, “while the number of sustainability indicators in the literature is growing, none of them advances the understanding of corporate sustainability”. Similarly, Gasparatos et al. [83] emphasize that none of the existing popular assessment methods can encompass consideration of a plethora of social, economic, and environmental issues simultaneously due to the use of the reductionist approach. In addition, little consensus exists as to how to define and measure sustainable manufacturing [90], as every tool represents an individual view on how to measure, meaning that each assessment presents different sustainability capabilities [95].

Many challenges that were related to the development of CSA are associated with the complexity of SD and SM concepts. What part of the company to assess and what sustainability aspects to address are the central questions that must be carefully addressed by developers. Although Bond and Morrison-Saunders [104] stressed the need to redefine SA as a facilitator of deliberation, i.e., to move away from an embedded pragmatist discourse to a new deliberative sustainability discourse, Sala et al. [105] demonstrate that most case studies still adopt the common triple bottom line (TBL) approach (example of the reductionist approach). Morrison-Saunders et al. [106] point out that a traditional sustainability assessment based upon linear cause-effect thinking is inadequate. Their research demonstrates that beyond the largely conceptual contributions to recognize the potential of a system approach to sustainability assessment, relatively little has been done to incorporate systems thinking into assessment—and even less practical examples have been demonstrated.

2.2. Scientific Requirements to Corporate Sustainability Assessment

SA is a part of sustainability science. Therefore, researchers define a range of principles and requirements for SA in order to ensure its compliance with the essence of sustainability science. Gibson and Hassan [37] defined the seven core principles as socio-ecological system integrity, livelihood sufficiency and opportunity, intra- and inter-generational equity, resource maintenance and efficiency, socio-ecological civility and democratic governance, precaution and adaptation, and immediate and long-term integration. Bellagio STAMP principles for SA, proposed in 1996 [107] and revised in 2012 [38], and characteristics of an ideal-typical sustainability assessment [30]. However, Sala et al. [105] state that principles and characteristics, although acknowledged by many practitioners, are rarely found in the available empirical examples of SA. Scientific requirements to sustainability assessment are summarized in

Table 1. Researchers’ requirements [29,30,32,33,34,35,36,37,38].

What to Consider during the Assessment:

Address intergenerational equity

Address intragenerational equity

Address geographical equity

Address interspecies equity

Address procedural equity

Assess the system as a whole, including its parts and their interactions

Assess the system considering the different sustainability objectives together (integration)

Assess dynamics and interactions between trends and drivers of change

Adopt appropriate time horizon (short-, medium-, and long-term) and (geographical) scope

Consider the normative nature of sustainability

Assessment of sustainability impacts and alternatives for decision-making, including synergies and trade-offs

Assessment is based on a conceptual sustainability framework and its indicators

Adapted to and integrated into the institutional context

Relations to global sustainability

Address socio-ecological system integrity

Address livelihood sufficiency and opportunity

Address resource maintenance and efficiency

Address socio-ecological civility and democratic governance

Address precaution and adaptation

Address immediate and long-term integration

Address resources consumption vs. value creation

Triple bottom line consideration

Consider each product lifecycle stage

How to do the assessment:

Establish formal and transparent synergy/trade-off rules

Be responsive to change, including uncertainties and risks (dynamism)

Avoid irreversible risks and favors a precautionary approach

Mix of qualitative and quantitative metrics

Broad participation of stakeholders, including experts, while providing active leadership to the process

Develop and maintain adequate capacity

Continuous learning and improvement

Transparency regarding data (sources, methods), indicators, results, choices, assumptions, uncertainties, funding bodies and potential conflicts of interest

Ensure effective communications (clear language, fair and objective, visualization tools and graphics, make data appropriately available)

An iterative assessment process, starting at the onset of the decision-making process

A mix of leading and lagging indicators

A mix of absolute and relative metrics

Trade-offs between external and internal performance measures

Integration of indicators across different policy arena

Reference values for indicators

Avoid “data drivenness”

Address interlinkages (socio-economic, socio-environmental, environmental-economic)

.

Many comprehensive SA methods have been presented in the literature. Still, the application of SA as a governance mechanism for sustainability is limited. To strengthen the application of SA, Pope and Petrova [9] propose to incorporate system analysis and deliberative approach into assessment. The purpose of the former is to conceptualize and analyze sustainability in terms of interconnected and dynamic socio-environmental system, rather than as static separate social, environmental, and economic factors. The latter should ensure incorporation of multiple perspectives and forms of knowledge into assessments addressing the normative principle of sustainable development. Similarly, Sala et al. [108] stress the need for SA methods to demonstrate the following characteristics:

• analytical-descriptive: SA has to address the key features of sustainability problems, the systemic nature of the problem, the potential systemic changes, and consequences;
• solution-oriented: SA has to provide goal-oriented and actionable knowledge, and provide direction for the solution strategy, as well as assist the decision-making in assessing sustainability at systemic level, moving from predictive to exploratory analysis;
• participative: SA has to be designed and developed in a participatory, interactive (non-extractive) collaborative way; through participative processes of scientists and stakeholders;
• suitable for scalability, transferability and comparability: SA has to allow for scaling and transferability of the generated solution; and,
• capable to manage uncertainties of information.

Rotmans [24] criticizes the current paradigm that underlies SA, since previous attempts to develop adequate tools had more or less failed. A prevailing approach is rooted in neo-classical economics, using the rational actor paradigm and equilibrium approximations to describe the behavior of actors. However, the neo-classical approach cannot address the complexity of SD since non-linear dynamics cannot be described in terms of equilibrium, efficient resource allocation, or price-driven actor behavior. Furthermore, Rotmans argues that new methods and tools are needed for SA. These should provide modeling capabilities that can semi-quantitatively (i.e., relative analysis instead of an absolute one) assess multiple dimensions of sustainability. Rotmans advocates for a new form of planning that aims at sustainable innovation rather than optimization—one that takes complexity and uncertainty as a starting point, using experimenting and learning as a guiding principle. According to Rotmans and Loorbach [109], in order to combat system failures one has to restructure a system; i.e., transition. Using the ideas of Rotmans [24], Sala et al. [108], and Haxeltine et al. [110], the characteristics of CSA methods to support decision-makers within the transition discourse were developed in [85], see Figure 1. The pyramid shape structure presents three types of characteristics: ones for underlying principles for CSA, ones for a design of the assessment method, and finally, ones for outcomes of the assessment.

3. Methodology

The work presented in this paper focuses on the second and third stages (see Figure 2) of the Design and Development Research [111]. The deliverable from the first research stage (‘Development stage’) is the CSA method for manufacturing (architecture and application procedure). The architecture, which defines the general structure of the assessment tool, is used to develop CSA tools for four case companies in the second stage, denoted ‘Testing stage’ in Figure 2. In the third research stage, i.e., ‘Verification stage’, we evaluate the feasibility of the method that is based on the results of the case study.

3.1. CSA Method

3.1.1. CSA Architecture

The CSA method (a CSA architecture and an application guideline) was newly developed in order to overcome the shortcoming of existing CSA methods and satisfy the requirements to the CSA. The development process presented in [31] included the analysis of scientific requirements to SA, such as [30,38,105], and addressed the limitations of CSA discussed previously in the literature. A CSA architecture was developed using a complexity-based model of SM presented in [101,112]. Moldavska and Martinsen [101] define SM as a complex behavioral pattern to which any manufacturing company should tend to evolve. This behavioral pattern is defined by the criteria for SM, such as to improve operational effectiveness, improve professional knowledge and competence of employees, increase the wealth of the society, and reduce discrimination. An organization can be seen as one that contributes to SM if it is continuously changing and its change is defined by the criteria for SM.

The CSA architecture (Figure 3) includes three modules: (1) organizational model, which has to be developed for each company, ensuring contextualization of the assessment; (2) a set of criteria for sustainable manufacturing; and, (3) a set of criteria for a sustainable world. This architecture is made to ensure that the organizational context is addressed, and that the assessment takes a holistic view on sustainable manufacturing. Employing an organizational model (Module 1) as a part of CSA ensures that the result of the assessment can be used internally, either to identify reasons of a low sustainability performance or to identify potential for improvements. The intention behind a holistic systems representation of a company is to address the dynamics of the company and interrelationships between its components, integrating different business functions. The need to embrace the dynamics and causal relationships within the assessment has already been discussed by researchers, see e.g., [113]. Since the knowledge of an individual is too limited to identify a consistent set of elements of the system, a participatory approach is suggested as the appropriate procedure to build the model of the company. System mapping can be obtained through participatory modeling with different employees, e.g., one from each functional unit of the company. A combination of different mental models can contribute to a more holistic and adequate representation of the system. Videira et al. [114] discuss the need for integrating participatory model building with system dynamics to support CSA. Participation of employees in organizational modeling can help create a shared view of the company, enabling non-linear knowledge generation and including subjective knowledge in the model. Moreover, it ensures the participation of potential users in the development stage of the CSA.

The proposed architecture conceptualizes sustainability through the criteria, which are then used to develop sustainability indicators. The use of a holistic set of criteria as a base for indicators development ensures that CSA measures what is relevant for sustainable development instead of what is measurable. The architecture enables a context-based assessment of the company while covering the span of issues associated with sustainability. Module 2 and Module 3 provide a comprehensive list of criteria that can guide the selection of sustainability indicators for different types of companies. Module 2 ensures that companies measure not only what matters most to them, but what is relevant to SM according to prior art [99,101]. Unified criteria for SM can, therefore, be a common ground for developers of CSA, whereas the choice of indicators will depend on the specific company context. Such an approach proposes a combination of a science-led process for defining criteria for SM and a business-led process for developing sustainability indicators. Moreover, to ensure that the company is seen as a part of the larger system, Module 3 secures that the company assesses its contribution to a global SD progress. Modules 2 and 3 provide the flexibility to change as the general understanding of SM and SD evolves over time. A set of criteria for SM has been developed by Moldavska and Martinsen as a part of the definition of SM [101]. 76 criteria have been proposed based on content analysis of SM definitions and a literature review. A comprehensive list can provide a complete picture of sustainability as compared to a selective use of common or most known criteria. Criteria for SW are developed based on the SDGs [115] and they are presented in [85]. Moreover, work on SDGs as a network has already been published [116], arguing that relationships between goals are important for a transition to SD. Thus, there are several opportunities to present criteria for SW as a network in the future, as presented in Figure 3.

Module 1 and 2, and Module 1 and 3 are linked by indicators. For each criterion in Module 2 and Module 3, sustainability indicators are developed and connected to the elements of the organizational model. Therefore, indicators are linking organizational model and sustainability criteria, forming the ‘Criterion—Indicator—Element of organizational model’ numerous links. Each indicator may have three main states: (1) green–indicating the positive contribution of an organization to SM and SW; (2) red–indicating the negative contribution of an organization to SM and SW; and, (3) grey–when data are not collected.

The key characteristics of the new CSA method are:

• SM is defined through the model of SM, using complexity theory;
• the model of SM is a frame for CSA architecture;
• CSA architecture has a modular structure: Module 1–dynamic organizational model, developed for each company, Module 2–sustainability criteria for manufacturing, Module 3–sustainability criteria for the world;
• the organizational model is developed based on mental models of managers (how managers define performance, what they see as important for desired performance, and what how they see cause-effect relationships related to the performance of the company), using qualitative system dynamics;
• criteria for SW are developed from SDGs;
• criteria for SM are developed based on the current state of the art;
• criteria are flexible, satisfying adaptability requirement and dynamism principle of sustainable development;
• criteria-based indicators development, integrating scientific-led development of criteria and business-led development of indicators;
• indicators are not aggregated;
• for each indicator, the desired direction is defined; and,
• indicators are links between criteria and organizational model.

Since the CSA architecture was developed based on the complexity-based model of SM, Figure 4 illustrates the transition from the model of SM (top block) to CSA architecture (middle block), and, finally, to CSA tool for a case company (bottom block). The model of SM represents a ‘sustainable’ company as ‘a system that contributes to the sustainability of the larger system while maintains its own sustainability’, i.e., company should tend to contribute to the global sustainability, i.e., world (A1), as well as maintain its own sustainability (A2) [101]. Manufacturing company is a sub-system with its own sustainability values (A2)–SM, and global sustainability values (A1)–SW. The former represents the performance of the company; the latter represents the contribution of the company to sustainable development of the world. A1 and A2 can be seen as complex behavioral patterns to which the company is attracted. Behavioral pattern A1 is defined by sustainability criteria for the global system (sustainable world), whereas A2 is defined by sustainability criteria for manufacturing. Thus, a company can be seen as sustainable if it is continuously changing, and this change is defined by the criteria for SW (A1) and SM (A2).

3.1.2. Application Procedure

Development of CSA tool–Customization of the CSA architecture.

Stage 1. Develop an organizational model.

A model of the company is to be developed, representing dynamics and complexity. Involvement of employees is strongly recommended. Approaches as agent-based modeling, social network analysis, and system dynamics (both qualitative and quantitative) can be used for modeling. The overall goal is to leverage the ability to identify any potential improvements in the company.

Stage 2. Choose relevant sustainability criteria.

A holistic set of sustainability criteria should be chosen from the proposed list [85]. This list can be modified as the research community’s understanding and conceptualization of SD and SM emerge. The choice of criteria has to be done by the specialist in sustainability and can be done either without company’s participation or with limited participation. The company should not influence the choice too much, since the list should include issues relevant for SD rather than what is more desirable or comfortable for the company, avoiding leaning the sustainability concept in the direction of what the company wants.

Stage 3. Develop indicator(s).

Indicator(s) should be developed for each criterion, ensuring a mix of lagging and leading indicators. This ensures that both actual performance and actions toward sustainability are tracked. Lagging indicators will reveal performance patterns (e.g., a percentage of products assessed with life cycle assessment method), while leading indicators can be used to track whether measures toward sustainability do change the performance of the company (e.g., reduction of natural resources used). The development should be guided by the list of criteria and should avoid data-driveness, i.e., when indicators are chosen based on what data are already available. Moreover, depending on the company’s experience with sustainability assessment and formal measurement systems, employees’ participation in indicators development can create a stronger feeling of ownership, which in turn will increase the acceptance of the resulting indicator list.

Stage 4. Identify a desired direction for indicators.

For each indicator, a desired direction of changes should be defined, i.e., ‘increase’ or ‘decrease’. For some of the indicators, instead, the desired value/outcome can be identified, for example, for indicator ‘Child labor’.

Stage 5. Link criteria and organizational model.

Identify corresponding variable(s) from the organizational model for each indicator, whenever possible. If there is no corresponding variable, then either leave the indicator without it or add relevant variable(s) to the organizational model.

Assessment of sustainability performance of the company.

Stage 6. Data collection.

Establish the frequency of data collection for each indicator and continuously collect data. The frequency can be different for different indicators (daily, weekly, monthly, yearly). For example, the OEE (overall equipment efficiency) can be measured daily, use of energy–weekly, training of employees–monthly, and employee turnover–yearly.

Stage 7. Data analysis.

The data should be regularly reviewed according to the chosen frequency of the indicators (Stage 6). Each indicator is assessed against the desired direction and color-coding is used. Then, changes in an indicator’s value should be compared with the desired direction of change. If the change in a value is matching the desired direction, then choose green color for the indicator (criterion is met), otherwise choose red color (criterion is not met). If data are unavailable for a specific indicator, then choose grey (information is lacking).

Identification of the improvement potentials.

Stage 8. Analyze red colored indicators

For an indicator colored in red, find corresponding variable(s) in the organizational model. Study and discuss variables that influence the corresponding variable. During the analysis and discussion, the variables in the model can be changed or added, and new links between model and indicators can be created. Linking indicators to variables in the organizational model is what creates the arena for discussion on how to improve organizational performance, including optimization and innovation. Elements of the model should be continuously discussed and analyzed by the decision-makers during the root cause analysis of sustainability performance and the identification of actions and potential improvements toward SD. At this stage, possible actions to improve the value of the indicator should be identified. In addition, the list of indicators is dynamic and can be updated.

Stages 6–8 should be continuously repeated.

3.2. Testing the CSA Method

Hyett et al. [117] distinguish between two approaches to case study methods, one proposed by Stake [118] and Merriam and Tisdell [119] and one by Yin [120] and Flyvbjerg [121]. The first approach, grounded in constructivism paradigm, supports a transactional method of inquiry, where the researcher has personal interaction with the case, which is developed in a relationship between the researcher and his/her informants. In this work, a multiple case research study that was situated in a social constructivist paradigm was performed. A multiple research strategy has the potential to provide more robust and compelling evidence on the phenomenon under study. A multiple case study was performed to test a new CSA method, by developing a customized tool for each case company. It is important to emphasize that the purpose of the case study was not to assess the sustainability performance of the case companies, but to develop an assessment tool for each of them.

Four Norwegian manufacturing SMEs from different industry sectors were deliberately chosen to introduce diversity into the sample. A multi-sector study has been done in order to cover a broader scope of the manufacturing industry and avoid focus on a particular sector, thus reducing possible bias related to specific sectoral characteristics. Moreover, this allows evaluating the applicability of the architecture to different sectors. The companies either participated in research projects related to sustainability before or proclaimed through the media that they work with sustainability. All case companies are B2B type businesses.

Table 2. Description of case companies.

Company	A	B	C	D
Number of Employees	40	169	47	77
Turnover (Million Euro)	7	45	12	27
Type of Industry	Supplier of plastic products	Supplier of automobile parts	Supplier of equipment for fishery	Supplier of parts for the public utility industry
Interviewed Functions	8 managers, covering functions: Production, Maintenance, R&D, Industrialization, Improvement, Logistics, Purchasing, Economy, HMS, Quality, CEO	11 managers, covering functions: Tooling, Plant manager, R&D, Industrialization, Production, Maintenance, HMS, Quality, Logistics, Purchasing, Prototyping	8 managers, covering functions: CEO, Supply chain, HR, Marketing, R&D, Purchasing, Service, Sales, Production	8 managers, covering functions: CEO, Marketing, Sales, Maintenance, Finance, Production, R&D, Logistics, Quality, Purchasing

presents a brief profile of each case company.

Semi-structured interviews were the main source of data for the development of the organizational model (Module 1). The purpose of the interview is to enter another person’s perspective [122], therefore the focus of interviews was on the participants’ ways to construct the meaning about the same phenomenon, i.e., the structure and behavior of the company. The semi-structured interviews were chosen within the hermeneutic-dialectic perspective in order to elicit the constructions and interpretations of managers. The managers’ constructions are the result of the context, information, other individuals, experience, and values. Although constructions are never perfect, more informed and sophisticated construction can be expected through reaching consensus among the individuals that are most competent to form such constructions [123]. Therefore, the multiple semi-structured interviews are seen as a suitable approach to collect the mental models of the managers for modeling the company. The choice of individual interviews instead of group interviews was motivated from the need to interact with the interviewees: to reduce the randomness of topics and increase the willingness to talk openly about their views on how the company operates. The mental models of the managers are important because managers in this type of SMEs are the ones with the greater knowledge about the company and their decisions and actions define the performance of the company, i.e., systems behavior. Interviewing representatives from different departments allows for addressing different interviewees’ views and opinions based on his/her experience. Moreover, it ensures giving consideration to the whole organization instead of the view of a single employee.

Interviews were conducted in each company with the goal to collect mental models of managers to develop an organizational model of each company. First, four interviews with the contact persons from each company were conducted, focusing on the background information about the company, including the revision of the documentation as strategy, products portfolio, yearly plans, indicators, metrics, code of conduct, IT systems, etc. Second, semi-structured interviews were conducted with different managers (see Table 2 for the interviewed business functions). 35 face-to-face semi-structured interviews were conducted with eight to eleven managers in each company during the period from September 2016 to June 2017. On average, each interview lasted for 1.5 h. All managers were ensured and agreed with anonymization in order to suppress their identity. The main focus was on how managers perceive/view the company: (1) what managers see as the most important issues (e.g., for economy manager it can be cost, and for maintenance manager–motivation of his workers and availability of reserved parts), and what influences these issues and what these issues can influence; (2) what can describe the performance of his/her business unit (e.g., delivery time for logistics manager, safety incidents for quality manager); and, (3) what problems and challenges the department has. These three questions were used to guide the conversation while maintaining flexibility to permit inquiry about new topics that emerged during the course of the conversation. The obtained data were further used to build the organizational model. Also, a plant tour was conducted in each company to observe and see the organization at work.

A holistic representation of a manufacturing company, i.e., organizational model, includes technical and human elements, and their interactions. Since the company has many decision-makers with their multiple realities, an organizational model that is based on mental models of different managers ensures the incorporation of knowledge and insights from different viewpoints. An integration of different mental models can contribute to a more holistic and adequate representation of the company. The purpose of an ‘organizational model’ is to customize the tool to the context of the company, i.e., type of functions and departments, B2B/B2C, make-to-order/make-to-stock, type of problems and challenges, work environment, organization of work, etc. Customization ensures that the result of the assessment is useful for identifying improvement opportunities by decision-makers in the company. Although multiple approaches can be used for creating an organizational model for CSA, we choose the qualitative system dynamics modeling, since several aspects of manufacturing may be difficult to express in mathematical equations. Qualitative modeling using causal loop diagrams (CLDs) does not require mathematical formalism and can represent causal relationships within the company by visualizing mental models and capturing the dynamics implicitly contained in the interviewees’ answers. CLDs are used to overcome the limitation of linear thinking, i.e., action X leads to result Y, because system dynamics reflects the fact that action X leads to various consequences with a cascading effect. Such an approach—i.e., building CLDs based on data from interviews—has previously proven to be effective in capturing and integrating mental models [124].

3.3. Verification of the CSA Method

Verification is a process of checking, confirming, making sure, and being certain, and it is performed to ensure the reliability and validity of a study [125]. The verification phase aims at evaluating the validity of the new CSA method along with the credibility of the development approach, which was designed and followed in this research study. The validity of the new method was seen in terms of persuasiveness and utility rather than proof, and quality of the method was evaluated against the previously defined set of desirable properties the method should satisfy [126]. Therefore, the verification phase evaluated whether the new method, as a method of inquiry, is appropriate (feasible) for the phenomenon of interest—i.e., a theoretically sound CSA that can assist in identification of specific problems and potential improvements toward sustainable development. The verification was performed by analytical reasoning using the results of the case study; i.e., based on the resulting four assessment tools. As a result of the design and development research, the quality of the new CSA method was verified. Verification is aimed at examining whether the developed method advances the state of the art in CSA, i.e., how the new method (1) fulfills the functions of CSA and satisfies the scientific requirements to SA in general and CSA in particular, and (2) overcomes shortcomings of existing CSA methods presented in state of the art. The former, namely verification against requirements, supports theoretical soundness of the developed method. The latter, verification against existing shortcomings, demonstrates the added value as compared to state of the art. The verification was performed by analytical reasoning using the results of the case study; i.e., based on the resulting four assessment tools.

4. Results

4.1. CSA Tools

The organizational models were developed based on the mental models of managers, which were collected through semi-structured interviews. It is assumed that managers may see the company differently, causing various interpretations and perceptions of the same situation. Although these interpretations might be similar in some cases, the difference can be a barrier to consensus in decision-making. Therefore, for the purpose of CSA, the company is modeled through the study and inclusion of multiple viewpoints to uncover decision-makers’ mental models. Despite the fact that a myriad of variables and relationships could be depicted when modeling a socio-technical system, the resulting models are considered relevant as a basis to discuss the performance of the company, since each one is built upon the mental models of the managers who are the ones making decisions in these SMEs. When the company is modeled as a part of the CSA, the purpose is not to perfectly model a system in order to accurately and deterministically predict its future behavior; i.e., “a model of X” [127]. Instead, the goal is to develop an accurate representation of mental models, shared by decision-makers, in order to support decision-making process with ‘what-if’ questions, i.e., “a model relevant to debate about X” [127,128].

Each interview was audio-recorded and transcripted, followed by a thorough analysis. The knowledge that the managers conveyed about the structure and behavior of the company was expressed in a causal loop diagram. Here, arrows describe the causal influences among the variables of the system [129]. Each arrow has a polarity that indicates the relation between variables. A positive (+) polarity means that the effect changes toward the same direction as the cause (reinforcing feedback loop). A negative (−) polarity means that the effect changes toward the opposite direction of the cause (balancing feedback loop). For each case company, the organizational model was built iteratively. First, one interview was analyzed and variables and links with positive and negative polarity were drawn. Then, the next interview transcript was read, and variables and links were added to the previous diagram, thus building the model interview by interview. Figure 5 shows the causal loop diagram depicting one interview with a logistics manager. Since organizational models are based on the managers’ mental models, the model verification was performed to depict the elements more accurately, by adding intermediate relationships and variables. One of the reasons for managers to skip intermediate links/variables is that they do not tend to think in terms of causal loop diagrams and some issues are just too apparent to them, i.e., implicit knowledge.

The resulting models for each case company show both similarities and differences. Variables and relationships, which depict objective issues such as OEE and total cost, are similarly modeled for each case company. On the other side, variables and relationships that were related to subjective issues as motivation, productivity, customer satisfaction, and cooperation are modeled differently, depicting opinions of the managers. Appendix A presents the organizational model for Company B, where variables in blue color are issues named by managers as to what matters most to them, challenges they have, or what they think can describe the performance of their business function. Although a myriad of variables and relationships could be depicted when conceptualizing the socio-technical system, the resulting model is considered to be relevant as a basis to discuss the performance of the company, since it is built based on the mental models of the managers who are the ones making decisions. Each resulting causal loop diagram describes the company as a system in terms of significant variables and relationships, as defined by managers. To illustrate the difference between organizational models, Figure 6 presents the parts related to customer satisfaction. None of the representations is more correct or accurate, because each one is the result of the integration of mental models within each particular company. In other words, the four representations depict the managers’ perception of how the company operates. As a part of CSA, a dynamic modeling of the manufacturing company can address the problems of reductionism and linear knowledge, and structures complexity of sustainable manufacturing. The model can help to understand elements of sustainable manufacturing and interconnections, and become a basis for redesigning companies. Moreover, development of CLDs is a learning process in itself, which can be used to engage stakeholders and decision-makers.

After the causal loop diagram for each company was built based on the managers’ perception of the company’s structure and behavior, they were additionally verified. This was done by first consulting the literature, see for example [130], and then conducting a workshop with experts in operations management and organizational management, with the purpose to standardize the terminology in the model and more accurately depict the links. The workshop was held with five specialists with experience in both industry and academia. Due to time constraints for interviews, and taking into account that at the time of the interview managers might forget some issues or not consider them to be relevant, the workshop with experts helped enrich the model with variables and links overlooked during the interviews. After the models had been presented to the participants, the variables and relationships were reviewed and discussed. Intermediate relationships and variables were added. For example, during the interview, one manager said, “when we have time to apply new thinking and new research, then we have more happy customers” (Figure 7). However, this formulation skips intermediate links existing in reality (Figure 8). Another example of the formulations, which were enriched by the workshop, is “If I have to use a lot of time to find relevant documents, I do not have time to discuss a better price for purchased components”. One of the reasons for managers to skip intermediate links is that they do not tend to think in terms of CLDs and some things are just too obvious to them, i.e., implicit knowledge.

First, relevant criteria were selected for each company from the list of criteria for SW. The company did not influence the choice of the criteria, since the list should include issues that are relevant for SD rather than what is more desirable or comfortable for the company, thereby avoiding leaning the sustainability concept in the direction of what the company wants. All criteria for SM were chosen. The result demonstrates the difference between criteria sets due to the type of industry. For example, criteria related to SDG 14 ‘Life belong water’ are mostly relevant for Company C, a supplier for the fishery industry. Criteria related to SDG 6 ’Clean water and sanitation’ are mostly relevant for Company D, a supplier for water utility sector. The resulting list of criteria for SW for Company A includes 74 criteria, Company B–74, Company C–78, and Company D–76. The list of criteria for SM is the same for each company and includes 75 criteria. Second, indicator(s) were developed for each criterion, and the desired direction for each indicator was chosen (see the example of criteria and indicators for Company B in

Table 3. Sustainability criteria and indicators, and link to the organizational model, excerpt from the result for Company B.

Criteria	Indicators	Desired Direction	Element of the Organizational Model
Sustainable World
2.1. Reduce extreme poverty	Minimal wage for workers per year.	↑	Cost of labor.
	Total tax paid per year.	↑	Income.
	Range of benefits for workers.	↑	Salary/Benefits.
	Supplier price/Market price.	↑	Price for raw materials.
2.2. Improve nationally social protection systems and measures	Social security benefits for workers.	↑	Salary/Benefits.
	Number of employees with insurance, per type of insurance.	↑	Salary/Benefits.
2.3. Ensure equal rights to basic services (property, inheritance, natural resources, new technology, and financial services)	Number of new solutions/techniques/technologies implemented.	↑	Effective solutions. Improvements implemented.
2.4. Reduce exposure and vulnerability to climate-related extreme events	Risk management related to climate-related events.	Y/N
	Investments in resilience to environmental hazards and resource scarcity, NOK & % of all investments.	↑	Investment potential.
2.22. Reduce mortality from non-communicable diseases	Type of injury and rates of injury, occupational diseases, lost days, and absenteeism, and total number of work-related fatalities, by region and by gender	↓	Injuries. Incidents.
	Total hours of overwork.	↓	Re-work.
	Hours of overwork per employee.	↓	Excessive workload per employee.
	Type of benefits for parents.	-	Salary/Benefits.
	Total air emissions (GHGs, ODS, NOx, SOx, POP, VOC, HAP, PM).	↓
	Total number and volume of significant spills.	↓
	Total water discharge by quality and destination.	↓
	Human health impact (from LCA e.g., ionizing radiation, human toxicity, respiratory inorganics).	↓	Product quality.
2.26. Increase health coverage	Life insurance, disability, and invalidity coverage.	↑	Salary/Benefits.
2.27. Reduce deaths and illness from hazardous chemical	Hazardous chemicals used in production.	↓
	Hazardous chemicals in product.	↓
2.28. Reduce deaths and illness from air, water, and soil pollution and contamination	Total air emissions (GHGs, ODS, NOx, SOx, POP, VOC, HAP, PM).	↓
	Water quality within the company.	↑
	Total water discharge by quality and destination.	↓
	Total weight of waste by type and disposal method.	↓
2.35. Ensure equal access to affordable and of good quality technical, vocational and tertiary education	Employees training and education.	↑	Competence.
	Number of courses/conferences/workshops attended.	↑
Sustainable Manufacturing
1.1. Reduce cost of product during the whole LC of the product/service	Total product cost.	↓	Production cost. Packaging cost. Transport cost.
1.2. Reduce noise from all processes	Noise level in the factory.	↓
1.3. Improve safety of technologies	Total safety incidents.	↓	Injuries. Incidents.
1.4. Reduce pollution to air during the whole LC of the product/service	Total air emissions (GHGs, ODS, NOx, SOx, POP, VOC, HAP, PM) during material extrusion.	↓
	Total air emissions (GHGs, ODS, NOx, SOx, POP, VOC, HAP, PM) at the suppliers.	↓
	Total air emissions (GHGs, ODS, NOx, SOx, POP, VOC, HAP, PM) during the production.	↓
	Total air emissions (GHGs, ODS, NOx, SOx, POP, VOC, HAP, PM) during the distribution.	↓
	Total air emissions (GHGs, ODS, NOx, SOx, POP, VOC, HAP, PM) during the use.	↓
1.5. Improve customers satisfaction	Customer satisfaction.	↑	Customer satisfaction.
1.8. Minimize the use of non-renewable energy during the LC of the product	Ratio of non-renewable energy compared to renewable during the production.	↓
	Ratio of non-renewable energy compared to renewable during the material extrusion.	↓
	Ratio of non-renewable energy compared to renewable during the transportation.	↓
	Ratio of non-renewable energy compared to renewable at the suppliers.	↓
1.12. Ensure competitiveness of the product	Cost of product compared to similar products.	↓
	Quality of product compared to competitors.	↑	Product quality.
1.13. Ensure competitiveness of the organization	Organizational income.	↑	Income.
	Organizational image.	↑
	Technological advancement.	↑
1.27. Improve quality of the process	Yield for process.	↑	Yield.
1.29. Improve reliability of the product	Failure rate of product in use.	↓
1.34. Improve safety of processes	Safety training, hours per employee.	↑	Injuries. Incidents.
	Safety incidents per process.	0	Following routines. Quality of routines.

). An example of such an approach, i.e., minimization or maximization as a preferable value, has previously been used in measuring corporate sustainability [131]. In this work, the indicators were developed without the participation of managers, because during the interviews most of the managers had difficulties in suggesting possible indicators that could provide them with useful information. The correlation was observed between the formalization of the measurement system, the experience of managers with formal measurements, and the ability of the interviewee to suggest specific indicators. During the course of indicators development, attention was paid to leverage a mix of lagging and leading indicators. For example, for criterion “1.34, Improve safety of processes” (see Table 3) one leading and one lagging indicators are employed. ‘Safety training per employee’ indicates ex-ante performance (leading), while ‘Safety incidents per process’ indicates ex-post performance (lagging). Also, absolute and relative (e.g., ‘Total hours of overwork’, ‘Hours of overwork per employee’), and qualitative and quantitative indicators were included. The resulting list of indicators for SW for Company A consists of 132 indicators, Company B–132, Company C–136, and Company D–135. For each developed indicator, a desired direction was chosen. However, for some of the indicators the desired value is defined instead of the direction, such as for ‘Disaster risk management’—‘Yes’/’No’, and for ‘Safety incidents per process’—‘0’. The list of indicators for SM for all companies includes 125 indicators. Occasionally, the same indicator (both SW an SM) is assigned to different criteria.

Once the indicators had been developed, they were associated with one or more variables from the organizational model (‘Element of organizational model’ in Table 3). For each indicator developed for sustainability criteria, a corresponding variable from the organizational model was identified. For example, for the indicator ‘Total number of safety incidents’, the corresponding variable is ‘Incidents’. In the case when no corresponding variables are available for an indicator, the indicator was left without a variable. Table 3 shows an excerpt from the two tables: ‘Criteria for SM—Indicators—Desired direction—Elements of organizational model’ and ‘Criteria for SW—Indicators—Desired direction—Elements of organizational model’ (links between Modules 1 and 2, and Modules 1 and 3). The complete CSA tools for case companies, i.e., the organizational model and two tables, can be found in Supplementary Materials.

4.2. Verification of the New Method

Table 4. Verification of the new CSA method.

Part I. Scientific requirements
1. Requirements to CSA functions:
CSA truly leads to the identification of actions that will contribute to global sustainable development.	This is addressed by combining organizational model and sustainability criteria in the new method. When the value of an indicator is not desirable (red), the variable from the organizational model linked to the indicator should be analyzed and causes behind the unsustainable trend identified and discussed. Thereafter, actions can be proposed.
Generate information to identify potential improvements toward global sustainability.	This is addressed by including Module 3 with sustainability criteria developed from SDGs, which describe global sustainability. Potential improvements toward global sustainability can be discussed in relation to the variables of organizational model linked to the criteria wished to be met.
Structure complexity of organizational dynamics and sustainability objectives.	This is addressed by separating sustainability criteria (as a substitute for sustainability objectives since the ‘transition’ sustainability discourse was adapted) and dynamic model of the company.
Foster attitude shifts in decision-makers.	This is addressed by developing an organizational model based on current mental models of decision-makers and by linking organizational model to indicators; thus, it allows to perform a root-cause analysis and group discussion to understand better the reasons behind the indicators values.
2. Requirements to what to consider during the assessment:
Address intergenerational equity	Intergenerational equity is addressed by using 17 SDGs as a base for the development of sustainability criteria, especially those related to environmental preservation and education as well as those related to Goal 10 such as ‘ensure equal access to affordable and quality technical, vocational and tertiary education’, ‘reduce exposure and vulnerability to climate-related extreme events’, ‘strength capacity for adaptation to climate change and extreme events’, ‘preserve world’s cultural and natural heritage’, ‘reduce air pollution’, ‘reduce GHG emission’, ‘eliminate genders disparities in education’, ‘increase number of CC mitigation actions’, ‘increase support for developing countries for CC-related planning and management’, ‘reduce marine pollution’.
Address intragenerational equity	Intragenerational equity is addressed by using 17 SDGs as a base for the development of sustainability criteria such as ‘ensure equal rights to basic services’, ‘ensure equal access to affordable and quality technical, vocational and tertiary education’, ‘ensure equal opportunities for women for leadership’, ‘promote gender equality’, ‘reduce inequalities and discrimination’, ‘adopt policies to decrease inequality’. Moreover, criteria for SM also include ‘ensure social equity’, ‘provide equitable opportunities for all employees’.
Address geographical equity	Geographical equity is addressed by the criteria related to developing countries such as ‘increase resource flows for development to developing countries’, ‘support developing countries in scientific and technological capacity’, ‘increase support for developing countries for CC-related planning and management’, ‘expand scholarships for developing countries’, ‘increase the export of developing countries’, ‘increase research and innovation in developing countries’.
Address interspecies equity	Through the inclusion of sustainability criteria related to biodiversity and animals protection, especially those related to SDG 14 and 15, such as ‘increase genetic diversity (plants, animals)’, ‘reduce degradation of natural habitats reduce loss of biodiversity’, ‘decrease illegal trafficking of species’, ‘increase financial resources to conserve and sustainable use of ecosystems and biodiversity’, ‘reduce trafficking of protected species’.
Address procedural equity	To develop a CSA tool using a new method, there is a need in a cooperation between professionals in sustainability assessment (expert knowledge) and internal decision-makers (managers). Moreover, the organizational model has to be built using the input from different managers to ensure that attention is paid to different parts of the system.
Assess the system as a whole, including its parts and their interactions	Organizational model aims at presenting the dynamic model of the company as a system based on the perspectives of different managers, who make decisions and have to see the causes and effects in the whole company. Also, by linking sustainability indicators to the organizational model, a whole system can be assessed.
Assess the system considering the different sustainability objectives together (integration)	The method includes a holistic set of sustainability criteria for manufacturing, based on SDGs and state of the art in sustainable manufacturing; thus, integrating different sustainability objectives through the criteria.
Assess dynamics and interactions between trends and drivers of change	In the new method, indicators represent the trends and criteria represent drivers of change. Therefore, by assigning indicators to each criterion, the trends and drivers are connected.
Adopt appropriate time horizon (short-, medium-, and long-term) and (geographical) scope	The new method represents sustainability through the criteria, not targets or desired values, for which the consideration of the time horizon is important. Time horizon is, therefore, more important for impact assessments, while the new method represents the performance assessment instead. Moreover, the new method is developed for a continuous decision-support in a company. Time horizon is considered at the stage of data collection when the frequency of data collection for each indicator is established. The frequency can be different for different indicators since indicators can have different time horizon such as ‘safety incidents’, ‘organizational image’, and ‘hours of safety training per employee’.
Consider the normative nature of sustainability	The new method recognizes the normative nature of sustainable development (i.e., what sustainability means depends on the views regarding the kind of world we want to live in). Therefore, sustainability criteria are developed from SDGs, which are the latest view of the global society on what kind of future we want, and state of the art in SM, as the values researchers currently associate with sustainable manufacturing. Also, in the new method, the possibility to modify the sustainability criteria (as the society’s norms and values will change) is ensured by the modular structure, which allows updating sustainability criteria as an understanding of sustainability evolves.
Assessment of sustainability impacts and alternatives for decision-making, including synergies and trade-offs	The new tool evaluates the performance of the company, not impact assessment of the alternatives.
Assessment is based on a conceptual sustainability framework and its indicators	The complexity-based definition of sustainable manufacturing provides a conceptual sustainability framework. Manufacturing company is a sub-system with its own sustainability values SM, and global sustainability values SW. These values are seen as complex behavioral patterns to which a company is attracted. Thus, a company can be seen as sustainable if it is continuously changing and this change is defined by the sustainability criteria. Indicators are developed for sustainability criteria; thus, they indicate how much development of the company is defined by the sustainability criteria.
Adapted to and integrated into the institutional context	The tool is adapted to the context by developing an organizational model for a company based on the mental models of internal decision-makers (managers). Also, sustainability criteria are chosen with the consideration of the context, such as for a supplier of equipment for fishery, criteria related to marine are prioritized.
Relations to global sustainability	In the new method, manufacturing company is seen as a sub-system of the larger system, with its own sustainability values (SM), and global sustainability values (SW). The module ‘Criteria for SW’, developed from 17 SDGs, addresses global sustainability.
Address socio-ecological system integrity	In the new method, sustainability is not separated into the pillars. A company is represented as a sub-system of the global socio-ecological system, and the organizational model is built based on the variety of company’s elements discussed by managers.
Address livelihood sufficiency and opportunity	Through the inclusion of criteria such as ‘increase access to safe, adequate and affordable housing’, ‘increase access to safe, adequate and affordable transport’, ‘provide access to affordable and reliable energy services’, ‘increase investments in energy infrastructure’, ‘increase decent job creation’, ‘improve safe and secure working environment’.
Address resource maintenance and efficiency	Through the inclusion of criteria related to natural resources such as ‘minimize the use of non-renewable natural resources during the whole LC of the product/service’, ‘reduce the use of natural resources’, ‘improve water resources management, ‘improve resource efficiency in consumption’, ‘increase resource-use efficiency’.
Address socio-ecological civility and democratic governance	Through the inclusion of criteria related to Goals 16 and 17 such as ‘reduce corruption and bribery’, ‘improving public institutions’, ‘improve decision-making at all levels (responsive, inclusive, participatory, representative)’, ‘increase a participation developing countries in the institutions of global governance’, ‘increase legal identity for all’, ‘improve public access to information’, ‘improve global partnership for SD’, ‘improve public, public-private and civil society partnership’, ‘increase capacity-building support for increasing data availability for SD’.
Address precaution and adaptation	Through the inclusion of sustainability criteria related to both mitigation of a negative impact (e.g., water pollution) and adaptation to negative consequences (e.g., risk management related to climate change).
Address immediate and long-term integration	All the principles of sustainability where attempted to be addressed simultaneously.
Address resources consumption versus value creation	Through the inclusion of sustainability criteria related to both value for the society (e.g., increase decent job creation, increase youth employment) and resources use (e.g., reduce the use of natural resources, decrease deforestation).
Triple bottom line consideration	The new method does not separate sustainability into pillars. By using SDGs and state of the art in SM, sustainability criteria cover social, environmental, economic, cultural, institutional, etc. issues.
Consider each product lifecycle stage	Through the criteria for SM related to product LC such as ‘reduce product cost during the whole LC of the product’, ‘reduce cost of product during the whole LC of the product/service’, ‘reduce pollution to soil during the whole LC of the product/service, ‘minimize the use of toxic materials during the whole LC of the product/service’, ‘minimize the use of water during the whole LC of the product/service’.
3. Requirements to how to do the assessment:
Establish formal and transparent synergy/trade-off rules	In the new method sustainability indicators are not aggregated and not weighted, therefore trade-off rules are not proposed.
Be responsive to change, including uncertainties and risks (dynamism)	In the new method, the possibility to change the sustainability criteria (as the society’s norms and values will change) is ensured by the modular structure, which allows updating criteria as an understanding of sustainability evolves. Moreover, the choice to avoid aggregation and weighting of indicators is made due to the uncertainties as an inherent characteristic of sustainable development (i.e., the change of raw materials can have a different effect in the long term on different aspects like price, availability, job creation, pollution).
Avoid irreversible risks and favors a precautionary approach	This issue should be ensured at the stage of assessment. However, to address this issue in the new method, the variety of criteria are included to safeguard from the unintended consequences.
A mix of qualitative and quantitative metrics	Both qualitative and quantitative indicators are developed.
Broad participation of stakeholders, including experts, while providing active leadership to the process	In the new method, professionals in sustainability assessment (expert knowledge) should cooperate with users/decision-makers (managers) to develop the model of the company and indicators.
Develop and maintain adequate capacity	In the new method, both sustainability criteria and model of the company are responsive to change. The organizational model can be (and should be) modified to ensure that it represents the structure and dynamics of the company. Also, sustainability criteria can be modified when the knowledge about sustainability evolves. The tool is developed for a continuous assessment and improvements.
Continuous learning and improvement	The tool is designed to be used for continuous improvement and change of mental models of managers. Since the organizational model is built from mental models of managers and indicators are linked to organizational models, the unsatisfactory values of indicators can be analyzed in relation to the current mental models (Stage 7. Data analysis. Stage 8. Analyze red colored indicators).
Transparency regarding data (sources, methods), indicators, results, choices, assumptions, uncertainties, funding bodies and potential conflicts of interest	This should be ensured during the assessment process, which happens after the tool is developed; therefore, this aspect cannot be evaluated now.
Ensure effective communications (clear language, fair and objective, visualization tools and graphics, make data appropriately available)	This should be ensured during the assessment process, which happens after the tool is developed; therefore, this aspect cannot be evaluated now.
An iterative assessment process, starting at the onset of the decision-making process	The new method is developed for a continuous assessment, i.e., data collection, links to organizational model, making decisions, and updating the model.
A mix of leading and lagging indicators	This is addressed during the indicators development process for each relevant criteria.
A mix of absolute and relative metrics	This is addressed during the indicators development process for each relevant criteria.
Tradeoffs between external and internal performance measures	In the new method, indicators are not aggregated; therefor trade-off rules are not applicable. However, since sustainability criteria are developed from both SDGs and state of the art in SM, criteria address both internal performance (e.g., ‘ensure acceptable working hours’, ‘improve quality of the process’, ‘enhance learning of employees’) and external performance (e.g., ‘increase value creation for society’, ‘reduce extreme poverty’).
Integration of indicators across different policy arena	In the new method, perspectives of different managers are included as a part of organizational model ensuring that interests of different arenas are represented.
Reference values for indicators	In the new method, instead of reference values for indicators, desired direction is defined, i.e., increase, decrease, and for some indicators–Yes/No and ‘0’. This is done due to the choice of the ‘transition’ sustainability discourse within the assessment.
Avoid “data driveness”	In the new method, first, relevant criteria are chosen without the involvement of a company, second, indicators are developed. Such an approach should safeguard against data-driveness.
Address interlinkages (socio-economic, socio-environmental, environmental-economic)	In the new method, sustainability is not separated into pillars. Interlinkages are addressed through the dynamic modeling of a company (relationships between organizational elements) and connections between organizational model and sustainability criteria.
4. Requirements to CSA within the transition discourse:
4.1. Outcomes:
Foster system innovation rather than optimization	Comparing to the optimization, which is the result of the gradual change, innovation requires a radical change in the company. In the new method, this is addressed by the introduction of a wide range of sustainability criteria related to internal and the external performance of the company. Criteria, therefore, can work as a ‘pull’ factors that stimulate the company to innovate in order to contribute to different sustainability aspects (e.g., increase afforestation, foster climate resilience, improve knowledge and skills to promote SD). Analysis of only organizational model can help to optimize the current company’s performance. Instead, analysis of indicators that link organizational model and sustainability criteria can stimulate innovation in order to find ways to meet the range of criteria.
Analyze unsustainable symptoms at the system level	This is addressed by modeling a company using the perspectives of different management, thus ensuring that attention is paid to all parts of the company. Analysis of indicators, which link the organizational model and sustainability criteria, can be a valuable input for analysis of unsustainable symptoms at the system level.
Recognize non-sustainable patterns in the system	Modeling company as a system and linking organizational model to sustainability criteria can indicate the unsustainable patterns in the company.
Foster social learning	This is addressed by providing a possibility to change mental models of managers by discussing indicators in relation to the organizational model, which was built based on current mental models.
Provide exploratory value rather than predictive	Due to the fact that any model and has limitations and uncertainties caused by the complexity of real systems (such as manufacturing company), the predictive capability of the assessment tools is succeeded by exploratory capabilities. The new method aims at supporting a dialogue among managers (internal decision-makers), who are trying to understand unsustainable patterns and how to improve the sustainability performance of the company. This is done by the choice not to aggregate indicators and allowing the exploration of the reasons (root-cause analysis) for any undesirable value of the indicator.
4.2. Design:
Modular structure	The new method includes three modules: two modules include sustainability criteria and one includes the organizational model.
Include key sustainability criteria for the system being assessed	Sustainability criteria for manufacturing company are included by Module 2 and Module 3.
Models contain both subjective and objective knowledge	Subjective knowledge is included in the organizational model (Module 1) by using managers’ mental models to develop a model. Objective knowledge is included in Module 2 and Module 3 by using an analytical approach to defining SM and sustainability criteria for manufacturing.
Capture non-linear dynamics	This issue is addressed by using qualitative system dynamics modeling (causal loop diagrams) for developing a model of the company.
Semi-quantitatively assess sustainability dimensions	This is addressed by the development of different types of indicators such as qualitative, quantitative, and semi-quantitative (i.e., relative scale such as ‘low’, ‘medium’, ‘high’ for, for example, ‘customer satisfaction’, ‘organizational image’).
Demand-driven (i.e., involve users in the development phase)	Managers as internal decision-makers are involved in organizational modeling to ensure that the result of the assessment is relevant for managers’ context. Moreover, the new method suggests that employees can be consulted regarding the indicators development when possible (i.e., when employees are able to suggest indicators).
4.3. Underlying principles:
Interdisciplinary approach	This is addressed by developing criteria from SDGs (which were developed by an interdisciplinary team) and participation of managers from all functions for organizational modeling.
Heuristic nature of assessment tools	The result aims to gain more insight into and achieve a better understanding of the problem areas (red indicators), rather than provide a deterministic value of the company’s sustainability performance. Since managers in a company perceive a problem from different perspectives, and therefore act differently, this has to be reflected in the CSA tool if the heuristic value is the goal. The new method addresses it by involving managers in modeling the company, thus enabling all-around insight into problems.
Complex systems theory approach	SM was defined using the complexity theory, and a complexity-based model of SM was used as a frame for the CSA method. The new method was developed using a combination of systems approaches.
Unavoidability of uncertainty that is a symptom of complexity and not an artifact that can be reduced	The aim of the proposed CSA method is not to avoid the uncertainty of the dynamic interconnections in a company, but to model it, to the fullest extent possible, for a continuous discussion. Moreover, since the uncertainty is an integral part of sustainable development, indicators are not aggregated or weighed.
Non-linear knowledge generation (interaction between knowledge producer and knowledge consumer)	In the new method, development of the organizational model is done by the interaction between specialist (sustainability researcher/professional) and users (managers). Also, the purpose of the organizational model is to stimulate a better understanding of the company’s performance through the cooperation between managers and sustainability professional, who is responsible for assessment.
Part II. Shortcomings and limitations of existing methods
1. Shortcomings of existing CSA methods:
CSAs fail to address system performance [41];	In the new method, manufacturing company is modeled as a part of the global socio-ecological system using dynamic modeling. Moreover, organizational model, representing manufacturing company as a system, is linked to the holistic set of sustainability criteria.
Use of incomprehensive and unsystematic lists of indicators for CSA in manufacturing [46];	In the new method, a comprehensive framework of sustainability criteria for manufacturing companies was developed from SDGs and state of the art on sustainable manufacturing. Since indicators are developed for each sustainability criterion, the resulting list of indicators is systematic and comprehensive.
CSAs tend to mix sustainability performance of the company and sustainability-oriented practices (most of them do not distinguish between the extent of implementation of sustainability-related practices and actual sustainability performance of the organization) [44];	The purpose of CSA is to evaluate the contribution, for each criterion indicators of actual performance and sustainability-oriented practices are included (leading and lagging), but not aggregated into an index.
Organizations measure what is measurable rather than what is important concerning the SM, due to the challenge to address simultaneously organizational context and SM phenomenon (context-based & global SD) [47];	What to measure is defined by the comprehensive list of criteria, selected by the specialist. The company’s context is addressed by developing an organizational model for each company and choosing relevant criteria systematically, involving the company. The method combines a science-led process for defining sustainability criteria and a business-led process for developing sustainability indicators; thus addressing organizational context and SM phenomenon.
Inability of many CSAs to provide a practical approach for companies to identify improvements and possible sustainability-oriented practices [48,82];	Improvements can be identified by analyzing the organizational elements linked to the indicators that should be improved (those having a red code).
The use of a set of unrelated indicators, since an assessment of separate entities without considering relationships between them neglects the dynamics of the company [84];	In the new method, indicators are linked to the elements of an organizational model, which in turn are connected by cause-effect relationships. Thus, changes in one part of the company as a system (element of the organizational model) will affect the values of different indicators.
Inability of some CSA to capture the complexity of SM—relationships between sustainability issues and interlinkages between elements of the organization—is due to a widely used reductionist approach to CSA [83].	Complexity of a manufacturing company and sustainable development is addressed by separating them into two blocks: organizational model and sustainability criteria (SM/SW). The organizational model focuses on the interlinkages between elements of the company, while sustainability criteria are connected to different elements of the company (organizational model).
2. Shortcomings of existing SA methods:
None of the existing indicator schemes is adequate for the purpose of providing all essential information about a system and its rate of change [86].	This issue is addressed by connecting the model of the company with a comprehensive set of sustainability criteria, and by proposing a criteria-based development of indicators instead of the selection of indicators from the existing indicator schemes.
The aggregation of indicators may hide serious deficits in some parts of the assessed system [86].	The new method does not aggregate the indicators in order to avoid hiding the deficits in parts of the company. Therefore, the result of the assessment provides the exploratory value and more holistic information for decision-makers instead of mechanistically determining the most serious problem or more important problem.
Most indicator frameworks are still under development and no framework is applicable as a whole to evaluate sustainable production [60].	For the new method, the priority was placed on the holistic set of sustainability criteria that should be a base for indicators development. Thus, the new method does not provide indicator framework, instead advocating for a combination of a science-led process for defining sustainability criteria and a business-led process for developing sustainability indicators.
Limited consensus exists on a reasonable taxonomy of sustainability metrics [87].	The choice was made towards the holism when developing sustainability criteria and indicators, although some researchers argue for a reasonable number to be around 10–20.
Very few examples of effective assessment processes can be found in the literature [26].	Since the new CSA tools have not been used to assess the sustainability performance of the company, the effectiveness of the assessment process cannot be evaluated.
Determination of weights of indicators may not always be straightforward and accurate, reflecting opinions of decision-makers and may, therefore, suffer from a high degree of subjectivity [51].	The new method does not weight indicators in order to avoid subjectivity or biases of decision-makers.
There is still no useful method for integrated sustainability assessment on the company level available [51].	The new method focuses on the company level and attempts to aggregate different parts of the company (modeling the company from managers’ mental models) and different sustainability aspects (sustainability criteria from SDGs and SM categories).
There is a lack of a comprehensive framework of sustainability criteria for sustainability assessment of manufacturing companies [51].	A comprehensive framework of sustainability criteria for manufacturing companies was developed from SDGs and state of the art on sustainable manufacturing.
Normalization and weighting of indicators are the source of subjectivity and reveal a high degree of arbitrariness, scientific rules for aggregation are often not taken into account [28].	The new method does not normalize, aggregate, or weight indicators in order to avoid subjectivity related to these procedures.
Most of the existing tools miss a holistic approach to sustainability [89].	In the new method, the holistic approach to sustainability is achieved by a complexity-based definition of sustainable manufacturing, utilizing a comprehensive list of sustainability criteria developed from the SDGs and content analysis of SM definitions.
Some of the existing frameworks for indicators are aimed at external reporting, rather than providing valuable information for internal decision-makers [88].	To ensure the value for the internal decision-makers, the model of the company includes issues relevant for internal decision-makers (managers), obtained through the interviews. Since indicators are linked with the organizational model, managers get the possibility to analyze how their work influences the value of sustainability indicators.
SA tools can be too technical and complicated for manufacturing companies [50,73,89].	The new method provides a conceptual architecture and a detailed step-by-step guideline for the development and use of the CSA tool. The tool does not require specific knowledge except the dynamic qualitative modeling.
Three spheres of sustainability have not received equal attention during sustainability assessment, and how to measure the social dimension remains a major problem in recent research [90].	In this researchers, the choice was made not to separate sustainability into pillars or spheres. Sustainability is represented through the criteria, developed from SDGs and categories describing SM concept. Therefore, the new method includes all sustainability aspects defined as important by the international organizations and researchers, i.e., SDGs were defined as a holistic description of sustainability by international institutions and definitions of SM were proposed by the wide range of researchers.
Comparison and aggregation of indicators can be difficult because different types of indicators use different reference units (work cycle, yearly production volume, days, product, etc.) [73].	In the new method, indicators are not aggregated and not compared. This decision was made due to the focus on the exploratory value rather than predictive, following the complexity approach instead of technocratic. Such an approach to SA was previously advocated by researchers.
There is still a lack of comprehensive assessment models and tools covering all three aspects of sustainability in a holistic approach [91,92].	The new method aimed at a holistic approach by defining manufacturing company as a sub-system of the world, with the criteria that define the development of the company defined by the comprehensive list generated from the SDGs and SM definitions.
The lack of a clear framework for measures and metrics at strategic, tactical, and operational levels in sustainable business development [93].	In the new method, strategic, tactical, and operational levels are represented in the organizational model. Since the model is developed based on what is important for different managers or what are the issues of the concern for managers, different organizational levels are represented. For example, organizational models for case companies include OEE, changeover time for equipment, cooperation with research institutions, competitive advantage.
It is difficult “to identify current measurable indicators that point to sustainability” since sustainability is associated with the future while indicators measure the present [94].	To address this challenge, the new method incorporates the criteria-based development of indicators, when criteria define the future and indicators measure the present. The use of the desired direction for each indicator demonstrates how the present performance (indicators) corresponds with the future (criteria).
Different assessment tools can present different sustainability performance due to the choice sustainability issues covered [95]	Since the new method, like any other assessment method, covers a different set of sustainability issues, the aim was to include a holistic set of issues as well as ensure the transparency of the choice. Definition of the resulting set of sustainability issues, i.e., criteria, are presented by the detailed description of the performed content analysis of the SM definition, and description of the inclusion of SDGs.
Contradictory strategies on how to improve sustainability performance can be established due to the compositions and interpretations of the indicators [62].	In the new method, indicators are not aggregated in order to compare the changes in values of different indicators during the same period. Disaggregated indicators can safeguard from the development of contradictory strategies, by seeing how each decision influences different indicators.
Most available assessment tools focus on environmental aspects of manufacturing system sustainability [52].	Since the sustainability criteria were developed from SDGs and a state of the art on sustainable manufacturing, the new method ensures that the wide range of sustainability aspects is included. The new method does not distinguish between environmental, social, economic, technological, institutional, cultural, etc.
The existence of many indicator sets has created confusion when a manufacturing company attempts to select a suitable tool [62].	The new method proposes a criteria-based development of indicators instead of the selection of indicators from the existing indicators sets.
Sustainability assessment tools may appear too theoretical and abstract [63,96].	The detailed step-by-step guideline for the development and use of the CSA tool is provided.
Manufacturing companies have had difficulty identifying assessment tools that are relevant to their desires to assess and improve the sustainability of their plants [132].	The new method incorporates SDGs in order to ensure a holistic approach to sustainability improvements by demonstrating to the companies a range of sustainability aspects to be improved. Also, since the sustainability criteria are defined, the companies get access to the nearly full list of issues associated with sustainable manufacturing. Therefore, the companies can choose which issues to address first.
There is a lack of easily applicable tools that assess the status of sustainability based on key performance indicators and that derive priorities for systematic improvement [77].	The new method can be argued not to provide key performance indicators due to the extensive number of indicators. This is caused by prioritizing the possibility of systematic improvements. Manufacturing company is a too complex system to be able to define few key indicators to assess a system’s performance.
While theory moves toward a constructivist approach, the practitioners in SA still utilize technical-rationalist models and suggest that it is caused by a resistance to change of practitioners and challenges created by inevitable complex systems [97].	The new method attempts to utilize a constructivism approach, allowing different managers to participate in the modeling of the company, thus incorporating in the assessment the issues that are important and relevant to each internal decision-maker. The method does not utilize any procedures for normalization, aggregation, weighting, which are common for the technical-rationalist approach. The result of the assessment, therefore, is providing more comprehensive information about a complex system (manufacturing company), instead of attempting to objectively assess the subjective aspects of sustainability.
A traditional sustainability assessment based on linear cause-and-effect thinking is inadequate [29].	The method addresses these issues by including the dynamic qualitative model of the company, using mental models of managers. By combining different mental models and different perspectives on the same company, it is possible to make non-linear relationships within the company more visible for decision-makers; thus, assist in root-cause analysis and cooperation between different sub-systems within the company.
A comprehensive assessment that spans over all pillars of sustainability, fully connected in terms of the covered themes and techniques used, and forward-looking does not exist at present [29].	In the new method, sustainability criteria were developed from SDGs and a state of the art on sustainable manufacturing, attempting to cover a wide range of sustainability aspects. Since the method does not distinguish between environmental, social, economic, technological, institutional, cultural, etc. pillars, the assessment might have benefits in terms of connectedness between the sustainability themes. Also, the ‘criteria–indicators–organizational model’ links aim at connecting different parts of the company and different sustainability aspects. Comparing to the CSA tools that model company and sustainability aspects together by system dynamics models, the new method presents sustainability criteria and organizational model separate. This was done because in some companies, especially SMEs that do not have experience with sustainability, managers think in terms of matters relevant for their everyday work such as equipment breakdown, sick leave, and cost of transportation, and not always link the matters relevant for their everyday work with sustainability aspects as reduction of energy use and pollution, community well-being, and infrastructure development. Therefore, the choice was made to separate organizational model, that represents the matters relevant for decision-makers, and sustainability criteria, that represent sustainability concept in a holistic way.
Sustainability indicators, which are calculated by gauging, comparing, correlating these quantities during a specified period of time, are blind to the dynamics of the manufacturing processes in that period of time [84].	In the new method, indicators are not aggregated in order to compare the changes in values of different indicators during the same period. For example, hiring new employees affects the indicators related to the cost negatively, but can affect indicators related to the quality of work and wealth of the society positively.
SA has to deal with different sources of complexity, i.e., different assessment levels (product, process, company, etc.), different sustainability dimensions (social, economic, and environmental), different perspectives, and different time references [84].	This challenge was addressed by allowing the complexity of different levels fall out of the needs and priorities of decision-makers (they will name issues that are important or relevant for them whether it is related to process, product, or other). Different sustainability dimensions were covered by translating SDGs and SM categories into sustainability criteria. Different perspectives are included by combining the mental models of managers, and assuming that SDGs were defined by the global society that includes the wide variety of perspectives.

includes a detailed feasibility evaluation of the CSA method. Verification is aimed at examining whether the developed method advances the state of the art in CSA, i.e., how the new method (1) fulfills the functions of CSA and satisfies the scientific requirements to SA in general and CSA in particular, and (2) overcomes shortcomings of existing SA and CSA methods that are presented in the state of the art. The scientific requirements and shortcomings of existing CSA and SA methods were previously identified through a systematic literature review, see Section 2.2 and [85]. Theoretical soundness is analyzed in Part I in Table 4, whereas added value is analyzed in Part II.

The evaluation of the method, both against the scientific requirements and existing shortcomings, demonstrates the feasibility of the proposed method to produce a tool that has a potential to assist manufacturing companies in a transition to more sustainable practices. The verification herein shows that some of the scientific requirements are not satisfied. For example, requirements that are related to trade-offs (‘assessment of sustainability impacts and alternatives for decision-making, including synergies and trade-offs’, ‘establish formal and transparent synergy/trade-off rules’, ‘tradeoffs between external and internal performance measures’) are not met because of the choice not to aggregate indicators and due to the objective of the CSA tool to assess sustainability performance instead of the impact of alternatives or actions. ‘Triple bottom line consideration’ requirement (Table 4) is not fully satisfied since the new method (1) implies the sustainability discourse of ‘transition or directed change’ instead of ‘pragmatic integration of development and environmental objectives’, and (2) represents sustainability within the assessment through ‘systems representation’ instead of ‘triple bottom line’ (see [55] for more details). ‘Reference values for indicators’ requirement is not satisfied as well, since the new method is built upon the discourse of ‘transition’, which called for the definition of ‘desired direction’ for indicators instead of ‘reference values’.

5. Recommendations to the Developers of CSA

Several researchers had discussed the need for a new approach to SA as well as the limited number of the CSA methods that can assist with the identification of actions toward sustainable development. Developers face the variety of questions such as: How to design the CSA method? What sustainability issues to include into assessment? What is sustainable manufacturing? How to develop indicators? The results of this research allow for outlining the central recommendations for the developers of CSA methods that are aimed at assisting with the contribution to SD.

• To ensure that CSA can assist a manufacturing company with the identification of actions that contribute to the global sustainable development progress, CSA should be based on the exploratory approach to assessment, holistic view on sustainability, systems approach to SM, the integration of SDGs into assessment, and the integration of context-specific and SOTA-based approaches.
• To enable the identification of improvements toward SD (stimulate optimization and innovation), the design of CSA should be modular, flexible, and include sustainability criteria and a dynamic model of a company.
• To ensure both context-specific and State of The Art-based indicators set, science-led criteria development and industry-led, criteria-based indicators development are recommended. The key challenge is not to design new indicators, since a sufficient number of them are presented in the literature. Instead, the challenge lies in understanding sustainability criteria for manufacturing companies.
• To reduce the risk of greenwashing, CSA method should combine the science-led development of criteria and industry-led indicators development. It should also avoid the aggregation of indicators to ensure transparency of the causes behind the performance.
• To address different aspects of SD, CSA should include both qualitative and quantitative indicators.
• To reduce the risk of sub-optimization, the CSA method should incorporate a holistic, dynamic model of a company. Interconnections between specific aspects of a company and a comprehensive list of criteria should enable a context-specific, still holistic assessment.
• To deal with the uncertainties of the desired and acceptable values for some indicators, it is recommended to use the desired direction for indicators.
• To align CSA with sustainability science, it is recommended to think about SD as (1) a process, not a destination, (2) an attribute of the system, and (3) a search process that never ends.
• To ensure the effectiveness of CSA, it is recommended to think about CSA as a system.
• To include site-specific consideration into CSA, the input from the future users/decision-makers should be included, such as their needs and requirements.
• To develop theoretically sound CSA method, different systems approaches should be combined to satisfy the requirements for assessment and cope with observed challenges of SA.
• The constraints regarding the system’s modeling and availability of data should be acknowledged, but if the goal is to assess the entire company, a more holistic scope and a more prominent analysis of system’s interactions (where manufacturing company is considered as a system) are crucial.
• To ensure transparency, developers need to describe how and what purposes of CSA are fulfilled, how requirements for SA and CSA are satisfied, and what observed shortcomings of existing methods were avoided and how. This should be done in a systematic way.

6. Discussion

Sustainable development, which is currently conceptualized by 17 SDGs defined by United Nations (UN), is becoming increasingly important for manufacturing companies. This creates a clear expectation for corporate sustainability assessment to operationalize the concept of sustainability for manufacturing, guiding companies toward sustainable development. Therefore, the development of CSA tools has to ensure that the resulting tool can assist a manufacturing company in the identification of actions that can contribute to global sustainable development progress. To achieve this, the proposed approach to the development of CSA is based on the exploratory approach to sustainability assessment, holistic view on sustainability, systems approach to sustainable manufacturing, the integration of SDGs into assessment, the combination of dynamic modeling and sustainability criteria, and the integration of context-based and state of the art-based approaches. The motivation behind the exploratory approach is that the complexity of any sustainability concept makes it nearly impossible to expect a tool to predict deterministically the future impacts and changes. Hence, the resulting CSA tools include both qualitative and quantitative indicators, but do not involve the aggregation of indicators into an index. The sole focus on the quantitative indicators is seen as a shortcoming of the existing indicator frameworks for manufacturing [60]. Our strategy to include a mix of qualitative and quantitative indicators is motivated form the focus on the both subjective (e.g., satisfaction, equality, knowledge) and objective aspects of sustainability (e.g., cost, wastes, investments).

The holistic view on sustainability resulted in an extensive list of sustainability criteria and indicators that collectively conceptualize global sustainable development and sustainable manufacturing. Although most of the existing tools tend to include a manageable list of indicators (10–20) [60], this results in the reductionism of the sustainability concept, which in turn fails to cover the whole range of relevant issues. Similarly, the systems approach that is applied to a manufacturing company has resulted in a relatively complex organizational model. On the one hand, this can limit the use of the assessment tool due to the need for knowledge about dynamic modeling. On the other hand, this enables a context-based assessment (representing the issues relevant for a company) and the identification of potential improvements, which is currently one of the limitations of the existing CSA tools.

A common approach to the indicators development uses the so-called TBL perspective that can be considered to be a reductionist approach to sustainability. By dividing a holistic concept into pillars, the further process of indicators’ integration becomes challenging. Usually, indicators are used to conceptualize sustainability concept for SA, but Pope et al. [26] proposed an alternative approach to the conceptualization, which defines sustainability, as a state to which society aspires, in terms of sustainability criteria. We took a similar approach but chose to define sustainability as a process of directed change, or transition, which is defined by the sustainability criteria. The difference is that while Pope et al. suggest using the criteria to define the state, we argue for the use of criteria to define/navigate the process of transition. Since the criteria for defining the transition are grounded on the 17 SDGs by UN, this enables a SOTA-based assessment, meaning that the range of issues covered by the assessment is relevant for the current state of knowledge about sustainable development. The generation of criteria from SDGs also means that the list of criteria is flexible and it can and should be modified as the knowledge about sustainable development evolves among scientists and practitioners. An example of such an evolution in the past was the transition from Millennium Development Goals to Sustainable Development Goals [133].

Although the list of criteria for SM can be made applicable for any manufacturing company, criteria for SW should be chosen for each case company addressing the type of industry, i.e., criteria related to Goal 2 are more relevant for the food industry, Goal 3–for healthcare manufacturing companies, Goal 6–for water and sanitation industry, Goal 7–for energy industry, and Goal 14–for marine industry and fishing industry. On the one hand, the need to choose criteria every time for a company can be seen as a limitation of the new CSA method in terms of time that is used to customize CSA for the company. On the other hand, this approach makes the CSA architecture flexible and adaptable for any type of industry, being an example of the holistic and context-based approach to sustainability assessment.

User involvement is often recommended for indicators development, since it can create the feeling of additional ownership for indicators. However, in our study managers were asked if possible indicators could be helpful for them, and most of them could not say what would be useful for their decision-making. This might be explained by the type of case companies engaged in this study; i.e., SMEs that have limited experience with sustainability assessment or formal performance management systems. The correlation was observed between the ability of managers to propose possible indicators and the size of the company and the formalization of the measurement system. A combination of the selection of criteria without the involvement of the company and criteria-based development of indicators with the involvement of the company can demonstrate the advantage in terms of equally addressing all the aspects of sustainability, avoiding data-driveness, ‘greenwashing’, and sub-optimization. Greenwashing happens when a company chooses indicators based on data availability or focuses only on its viability, ignoring the contribution to the sustainability of the larger system in which the company operates.

The development of CSA for four case companies reveals important issues to be considered during the development process. Participatory modeling is often presented as an effective approach to capture mental models and collect information about the modeled system; however, it might be less beneficial for the sake of sustainability assessment. One of the reasons for this, observed during the interviews, is the openness of managers to discuss their view on the company’s organization of work and performance. Another reason for modeling done by the specialist rather than during the group interview is the complexity of the model. The process of interviews and workshop showed that for the individuals not experienced with system dynamics modeling, it could be challenging to focus only on cause-effect relationships, ignoring the correlations, and circumstantial and conditional effects. The ability to depict dynamic relationships as a part of causal loop diagrams can also be challenging if the participants are more used to linear modeling techniques.

The resulting organizational models have boundaries that do not coincide with the organizational boundaries of the existing departments. The reason is that, in reality, the cause-and-effect variables are in different departments, and some activities in the model are actually missing in the real situation. The model in which the boundary cuts across the organizational boundaries of the actual departments provides the advantage of stimulating the discussion and learning within the company when the model is used to analyze the reasons behind sustainability performance. Moreover, an organizational model that is based on the mental models of the managers in combination with the choice of relevant criteria for industry ensures a context-based assessment. As a part of the CSA, the organizational model does not have to be ‘right’; it rather has to represent the views of the decision-makers. If the mental models of decision-makers are incorrect, then it will be addressed during the analysis of the indicators. Change of mental models is one of the purposes of sustainability assessment. Therefore, the ‘organizational model’ should be continuously modified and updated, representing the changes in the company and decision-makers’ mental models.

One of the main shortcomings of causal loop diagrams is the inability to re-use models for other problem situations. However, the development of the organizational models for four manufacturing companies demonstrates that some ‘blocks’ can be re-used for most of the manufacturing companies. This can simplify the modeling process for other companies. The case study shows that ‘objective’ blocks, related to, e.g., OEE and cost, can be re-used, as these do not depend on the managers’ perception or mental models. However, more ‘subjective’ blocks related to motivation, quality of design, customer satisfaction, etc. are perceived differently by individual managers in different companies (as shown in Figure 6). These parts cannot be re-used as they are context-specific and represent views of managers in their specific organizational context. The value in developing CLDs as a part of sustainability assessment is to help understand the behavior of complex systems by analyzing time delays and feedback loops that affect the behavior. CLDs can be developed based on historical data or ‘anecdotal’ information from stakeholders through a process of engagement [128]. A sustainability assessment that is based on the CLDs can be an alternative approach to the assessment methods that use a disaggregated list of sustainability indicators. Although the application of system dynamics and CLDs for the purposes of SA and CSA, including for manufacturing domain, has been reported and discussed [72,134,135,136], this work emphasizes the benefits of the CLDs built upon the mental models of internal decision-makers.

7. Conclusions

This paper seeks to answer the research question of how to design a theoretically sound CSA method that can strengthen the ability of manufacturing companies to contribute to global sustainable development. To answer this question, we applied a new CSA method to develop assessment tools for four manufacturing case companies, and, using the result of the case study, we evaluated the theoretical soundness of the method and added value as compared with the state of the art. This paper presents the new CSA method (the architecture and the application guideline), four CSA tools, and feasibility analysis of the method. Sala et al. [105] previously stated that although many practitioners acknowledge requirements to SA, these are rarely found in the available empirical examples of SA. Therefore, we conducted a detailed verification that thoroughly demonstrates how the new method addresses the scientific requirements to SA and CSA. Finally, based on the results of this research and lessons learned, we outline recommendations to the developers of CSA.

Overall, to assist companies with the identification of potential improvements, the necessary attributes of the design of CSA are the dynamic modeling, a comprehensive set of sustainability criteria, indicators linked to SDGs, and incorporation of needs of decision-makers in a company. This should as a minimum safeguard an assessment from greenwashing and sub-optimization as well as integrate a company’s context-specific aspects and state-of-the-art knowledge about SM. The distinguishing feature of the new method when comparing to the ones that are presented in the literature is the integration of dynamic modeling of the company using mental models of decision-makers and sustainability criteria—developed from SDGs and state of the art in SM, and serving as a base for indicators development. Consequently, the claimed advantage of the proposed method is its ability to develop an assessment tool that can, among others, (1) structure the complexity of sustainability and organizational dynamics, (2) address system’s performance, (3) consider a normative nature of sustainability, (4) incorporate a comprehensive list of sustainability indicators, (5) avoid greenwashing and data-driveness, (6) asses a company as a whole, including its parts and interactions, and (7) enable continuous learning and improvements.

Future studies should focus on the use of the developed assessment tool to assess the sustainability performance of companies while identifying potential improvements. Also, the possibility to reduce the number of indicators without compromising on the underlying holism of the proposed approach should be studied. A study of SDGs as a network has been reported already [116]. Therefore, the new method can be enriched by representing criteria for SW and criteria for SM as networks, in addition to establishing links between criteria for SM and between criteria for SW.