Developing Capacities to Lead Change for Sustainability: A Quasi-Experimental Study of Simulation-Based Learning

Abstract

Education for sustainability should be capable of developing citizens with the mindset and skills to meet the complex sustainability challenges faced in the 21st century. Although educators have advocated for the use of simulations and serious games in education for sustainability, there is need for experimental research that tests the effects of these active learning methods on learner knowledge, skills, and attitudes. This empirical study aimed to fill this gap by examining the effectiveness of the Leading Change for Sustainability—Business, an online computer simulation that challenges learners to transform a company’s capacity to achieve a triple bottom-line of sustainability goals. This article reports the results of a quasi-experimental study that was designed to assess change in knowledge and attitudes of 87 students studying in a Master of Management program in Thailand. The research found that learning with the Leading Change for Sustainability—Business simulation resulted in large, significant change in knowledge of sustainability and change management concepts, as well as in the sustainability mindset of students. The successful use of this simulation in a “fully online instructional environment” not only offers support for the use of this simulation in educating for sustainability, but also for the broader potential of simulations in online learning during the post-COVID-19 era.

1. Introduction

Companies throughout the world face the challenge of reorienting their goals beyond the singular quest for shareholder value and towards the achievement of global sustainable development goals. This was recently recognized by leading corporations in the United States.

Nearly 200 chief executives, including the leaders of Apple, Pepsi, and Walmart, tried to redefine the role of business in society—and how companies are perceived by an increasingly skeptical public. Breaking with decades of long-held corporate orthodoxy, the Business Roundtable issued a statement on “the purpose of a corporation,” arguing that companies should no longer advance only the interests of shareholders. Instead, the group said they must also invest in their employees, protect the environment, and deal fairly and ethically with their suppliers.

[1]

The values that guide corporate leaders are developed through workplace socialization, as well as during their education in graduate schools of business [2,3]. Business schools, either implicitly or explicitly, communicate normative expectations to their students concerning the values-orientation of corporate leadership [4,5,6]. Given the changing mindset suggested by the opening quotation, business schools bear a special responsibility for developing leaders who demonstrate positive attitudes towards global sustainability goals. In addition, future leaders will need knowledge and skills that can be applied to the transformation of companies with the capacity to create a positive impact on the economic, social, and environmental sustainability of the planet [7,8,9].

Once educators accept this responsibility, the question becomes, ‘What learning methods and experiences will yield graduates who demonstrate the motivation, knowledge, and skills needed to support the change towards sustainability in their organizations?’ While there is no consensus on the answer to this question [10,11], a growing number of scholars have proposed that simulations and serious games are particularly well-suited to meeting the challenges of educating for sustainability [4,12,13,14,15,16]. This is due to empirical findings that support the efficacy of simulations and serious games in enhancing the kind of systems thinking and knowledge application that is required to address real-world sustainability challenges [17,18,19,20]. That said, there remains inadequate research on the efficacy of business simulations used in management education for sustainability [4,17,21].

This represents the gap that was addressed in the study reported in this article. More specifically, this research describes the use of a new simulation developed for business education, Leading Change for Sustainability—Business (LCS-B), and evaluates its effects on student knowledge and attitudes. The research questions addressed in this study are as follows.

• RQ1: To what extent were students engaged by learning with the LCS-B simulation module?
• RQ2: Does student knowledge of sustainability and change management concepts improve significantly as a result of learning with the LCS-B simulation?
• RQ3: Do students develop more positive attitudes towards sustainability as a result of learning with the LCS-B simulation?

This article reports on the use of the LCS-B simulation with 87 graduate students studying in a Master of Management degree program in Thailand. A quasi-experimental research design was employed to assess the impact of learning with the LCS-B simulation on student engagement, knowledge, and attitudes. Data were collected through pre/post-tests of student knowledge of change management and sustainability concepts, as well as their attitudes towards sustainability. Paired sample t-tests were used to assess change in student knowledge and attitudes following the completion of a nine-hour, fully online learning module built around the simulation.

The research offers several potential contributions to the literature. First, the study offers an empirical evaluation that extends prior research on the efficacy of the LCS-B simulation based on student perceptions [22]. Second, the research contributes to a small but growing body of experimental research on the use of simulations and serious games in the domain of managing for sustainability [4,17,23]. Third, the research expands research on the use of simulations in management education for sustainability in the context of an Asian developing society [24]. This is significant in that some educators have asserted that Asian learners are difficult to engage in active learning [25,26]. Finally, the fact that this research was conducted during the COVID-19 pandemic meant that the LCS-B simulation module was delivered in a fully online synchronous/asynchronous instructional environment. Thus, the findings may also offer implications with respect to the potential efficacy of simulations as a pedagogical framework for use in online learning during the post COVID-19 era [27,28].

2. Conceptual Background

2.1. Leading Change for Sustainability in Business Simulation

The LCS-B simulation is a web-based simulation with a user-friendly interface that facilitates learner access anywhere in the world where there is Internet connectivity. The learning objectives of the simulation include the following.

• To understand key concepts related to economic, environmental, and social sustainability and their implications for management practice.
• To apply theories of individual and organizational change to the challenge of organizational transformation.
• To plan and execute a simulated long-term strategy for leading a company to become more sustainable.
• To develop positive attitudes towards sustainability challenges facing corporations and society.

The LCS-B simulation employs a problem-based approach to learning. This means that students are presented with the problem first. Only after they have begun working towards solution of “the problem” is theoretical content gradually introduced [29]. Thus, at the outset, learners are presented with the challenge of change. The Managing Director of the Best Company launches the project with a speech which concludes as follows.

Starting today, ‘business as usual’ is no longer acceptable at the Best Company. One Future will become the engine to spark new investments in the development of our employees, launch actions to protect and improve the environment, and make dealing fairly and ethically with staff, customers, suppliers, and the community a core value of our corporate culture. Moreover, as other companies have proven, ‘sustainability’ can also become a driver for broader innovation leading to improved long-term productivity. Simply stated: within three years, it is my goal to see the Best Company become a model of sustainable business against which other companies seek to benchmark their practices.

The learners are informed that they have been appointed to the One Future project team, charged with making the Best Company more sustainable. Al summarizes the team’s task.

During this three-year assignment, you will be responsible for working with corporate management to guide the company’s transformation into an organization that models sustainable business values and practices. This transformation should become evident in stakeholder practices as well as in our company’s economic, environmental and social results. I know that meaningful change takes time, but it is my hope that in three years, we will see a transformed corporate culture at the Best Company.

You will lead the pilot implementation of One Future in three business units: the Head Office, Eastern Branch, and Western Branch. In each of the three years, your team will have an annual budget to spend on the project. Use the budget to fund activities aimed at informing, engaging, empowering, and supporting stakeholders during our company’s transformation into One Future. Use your budget wisely, think strategically, and remember that time is your most valuable resource. Good luck!

When learners log into the online simulation, they see the ‘game screen’ displayed in Figure 1. Key features of the game screen include:

• The 24 stakeholders listed along the left side of the game screen are representative of stakeholders associated with the three business units participating in pilot implementation of One Future;
• Three business units: Head Office (Red), Eastern Branch (Blue), Western Branch (Green);
• Five stages of change encompassing squares or spaces through which the stakeholders will move as they go through the process of change on their way to the ‘Sustainability stage’ (i.e., Awareness, Interest, Preparation, Practice, and Sustainability);
• Change implementation activities listed along the right side that the One Future project team will use to move staff through the stages of change;
• Bits in the upper right corner; these represent the budget that will be spent on the activities, each of which has a different cost;
• Bennies in the upper right corner, which represent triple-bottom-line benefits that are accrued when the team implements activities that have an impact on the triple bottom-line (e.g., gain 150 Bennies when you succeed with ‘Use Sustainable Practices’).
• A button at the bottom right which shows the Level achieved by the team; the team starts at the ‘Apprentice Level’;
• A Strategy Record button which will display all of the moves played by the team at any point during the simulation.

The 24 stakeholders begin the simulation off the game board, equivalent to the “unaware stage of change”. However, as the One Future project team begins to implement activities (e.g., Talk to, Talk to 2nd time, Sustainability Survey) designed to raise awareness, stakeholders will begin to move across the game board (see Figure 2). Each time the team chooses to ‘do an activity’, the cost of the activity is deducted from their budget and a feedback card pops up explaining ‘what happened’. For example, in Figure 2, the response to conducting a Sustainability Survey is positive; stakeholders move as a result of gaining information, and the project team better understands the extent of the challenge ahead.

However, success is not guaranteed. For example, assume that the project team de-cides to follow up the Sustainability Survey with Sustainability Training as their next ac-tivity. The response card would tell them: “Not enough of the people that you selected are interested in joining the workshop. Luckily you find out before you spend money hiring the trainer. Get back 3 Bits. Nobody moves.” This feedback prompts the team to analyze which activities they should do to create interest among the participation before trying to conduct the Sustainability Training activity again.

Embedded in the decision rules of the simulation are different requirements that must be fulfilled in order for the activity to be successful. One requirement concerns the ‘readiness’ of the stakeholders for the selected activity based upon their “stage of change”. Thus, in the above example, not enough of the stakeholders have reached the Preparation stage, suggesting that they are not yet ready to benefit from training. This illustrates the “hidden interdependencies” among the activities. That is, the team must consider the stakeholders’ readiness for particular activities, based upon what has already taken place.

These “hidden interdependencies” require the learners to think strategically about the change process. At any point in time, the team must consider the best sequences of activities, and learn to anticipate what may happen in the future. These goals are facilitated by other features of the simulation, such as the Strategy Record, which the team can access at any time. The Strategy Record shows the prior sequence of activities that they have implemented up to that point (see Figure 3). As with all simulations, learning unfolds through a process of trial and error. Because learners will play the simulation numerous times, they are able to learn from their mistakes and develop more effective strategies over time.

Figure 3 shows an example of a simulation session during the 3rd year of play. In this example, the One Future project team has implemented a “successful strategy”. This can be observed from the “results” whereby they have already moved most of the stakeholders into the Sustainability stage of change and gained a large number of Bennies. Note, however, that even with a successful strategy, some stakeholders have yet to enter the Preparation Stage. The patterns of stakeholder progress on this game screen suggest two lessons from the simulation. First, even with a strong change strategy, not all stakeholders will “change” at the same pace. Second, “successful” change in an organization does not require 100% of the stakeholders to reach Sustainable adoption of the new practices.

Note at the top of the game screen that this team has already gained 6000 Bennies. At the end of the third year, the simulation will evaluate their results at one of six levels based on player progress to the Sustainability Stage and the number of Bennies accumulated: Apprentice, Novice, Manager, Leader, Expert, or Change Master. This ‘feedback on results’ is a significant motivator that causes students to want to keep playing the simulation. It should be noted that the first time that students play, it can take two hours or more to complete the three years. However, after the players know the people, activities, and method of play, they can complete the three-year simulation in under 30 min.

This sub-section of the article has served as a brief introduction to the LCS-B simulation. A longer description of the design features, underlying theories, and method of play are included in a companion paper [22]. Readers who are interested in gaining a more complete description are referred to this alternate source.

2.2. Conceptual Framework

The quasi-experimental research design used in this study sought to assess the effects of learning with the LCS-B simulation on student knowledge and attitudes towards sustainability (see Figure 4). The conceptual framework guiding the study has been employed in in other experimental studies on the effects of simulation-based learning [12,30]. Demographic factors were considered as control variables that could influence student engagement with the simulation. Although learner engagement was proposed as a potential influence on the effectiveness of the simulation-based learning, the measures of student engagement were not incorporated into the quantitative test of the model.

As suggested in Figure 4, three dependent variables were included in the study reported in this article: change management concepts, sustainability concepts, and sustainability mindset. The simulation-based learning module described in this article was designed with explicit learning objectives focusing on all three of these dimensions. The rationale for the selection of these learning outcomes is described briefly below.

Organizational change for sustainability implies a major change in corporate culture that must become evident not only in corporate policies but also in stakeholder mindset and workplace practices. Thus, organizational change for sustainability can be viewed as the implementation bridge from the status quo to a new one [7,8]. Researchers have asserted that organizational transformation to sustainability requires proactive, skillful change management [9]. In the context of the LCS-B simulation, “sustainability” represents the change focus and “change management” the means to bringing it into reality.

Leaders who accept the challenge of enhancing the sustainability of their companies must, therefore, have some knowledge of sustainability concepts [9]. Based upon our reading of the literature, core knowledge of sustainability would include a basic understanding of concepts such as triple bottom-line, social sustainability, waste management, circular economy, green energy, green human resources management, sustainable supply chain management, stakeholder engagement, corporate social responsibility, and life-cycle assessment. These sustainability concepts are included in the instructional module and students’ basic knowledge of the concepts is evaluated in the pre/post-tests.

Through their engagement with the LCS-B simulation, students also gain the ability to apply several change management theories to the challenge of enhancing corporate sustainability. These theories include diffusion of innovations [31], concerns-based adoption model [32], and Kotter’s strategic change model [33]. These theories are learned through and applied during the process of playing the LCS-B simulation.

The LCS-B simulation module not only aims to enhance student understanding but also to shape their attitudes towards the “need” for sustainability in the corporate world and society at-large. In the context of the simulation, a positive sustainability mindset recognizes the importance or value of enhancing the social, economic, and environmental sustainability. The authors view a positive Sustainability Mindset as a pre-condition for leadership that is willing to take on corporate sustainability challenges.

2.3. Hypotheses

The hypotheses proposed for this study were as follows.

H₁. 

Simulation-based learning is positively associated with changes in students’ knowledge of general sustainability concepts.

H₂. 

Simulation-based learning is positively associated with changes in students’ knowledge of managing organizational change for sustainability.

H₃. 

Simulation-based learning is positively associated with students’ attitudes towards sustainability improvement.

3. Research Method

3.1. Research Design

The study presented in this article was part of a larger mixed method, quasi-experimental study designed to assess the impact of learning with the LCS-B simulation on student knowledge and attitudes [34]. In this report, the authors present the quantitative results of a pre/post-test study that examined change in learner knowledge of sustainability and change management concepts and attitudes towards sustainability after three weeks of studying with the LCS-B simulation.

3.2. The Treatment

The treatment was delivered and assessed in the context of a seven-week course on Leading Change for Sustainability. The duration of the treatment, however, was limited to three 180-min online class sessions. These ‘synchronous’ class sessions were conducted over Zoom and centered on the Leading Change for Sustainability—Business (LCS-B) online computer simulation. Asynchronous learning was used at home, where students accessed the simulation and assignments via the Moodle platform.

The full sequence followed in executing the treatment is shown in

Table 1. Summary of instructional activities in the LCS-B class over four-week period.

Week	Activities in Class	Individual Homework
1	Course introduction Inform students about participation in the research Students submit research consent forms Pre-test (knowledge and attitude) Mini-lecture:  Introduction to sustainability and change concepts  Introduction to playing the LCS-B simulation Teams of 3 students play LCS-B in breakout rooms Whole class debriefing	Play simulation at home Respond to Week 1 Online Discussion Forum Reading assignment
2	Mini-lecture:  Sustainability concepts  Stages of the change process  Diffusion of Innovations/Adopter types Teams of 3 students play LCS-B in breakout rooms Whole class debriefing	Play simulation at home Respond to Week 2 Online Discussion Forum Reading assignment
3	Mini-lecture:  Sustainability concepts  Kotter’s 8-stage model of org. change  Synthesize change models Teams of 3 students play LCS-B in breakout rooms Whole class debriefing	Play simulation at home Respond to Week 3 Online Discussion Forum Prepare for exams in week 4 Reading assignment
4	Post-test (knowledge, attitude ) Individual simulation exam in class	Respond to Week 4 Online Discussion Forum

. Each class session began with a 60-min mini-lecture on sustainability and change concepts (see Table 1). This was followed by 90 min during which three-student teams played the LCS-B simulation in breakout rooms. The instructor would ‘visit’ the rooms, asking and responding to questions while students played the simulation. Finally, each class ended with a whole class debriefing. This typically consisted of the instructor prompting students to brainstorm and analyze a change strategy while playing the LCS-B simulation as a whole class.

Independent practice was an essential element of learning in this module, without which the learning objectives could not be attained. Each week, at the conclusion of class, the instructor emphasized that gaining the benefits of simulation-based learning requires the user to play multiple times. How often students played was, however, left up to the students. Nonetheless, the necessity of playing multiple sessions each week was driven home to students by the inherent challenge of the LCS-B simulation. Indeed, it often took up to two weeks and six or more full game sessions before students began to achieve even moderate success (i.e., reach the Manager, Leader levels).

The weekly online forum discussion questions, posted on the Moodle platform, are shown in

Table 2. Weekly online discussion forum questions.

Week	Online Discussions Forum Question Posted by the Instructor
1	What was the most difficult problem that you encountered so far in implementing change for sustainability at the Best Company? How did you solve it? What 1 piece of advice would you like to share with others? What 1 problem would you like advice on from others?
2	What was the most surprising insight you have gained from playing the simulation so far? Why was it surprising? What 1 piece of advice would you like to share with others? What 1 problem would you like advice on from others?
3	What lesson that you learned from the simulation could you apply in your work? Please be specific and give at least one example. What 1 problem would you like advice on from others?

. The structure of the questions prompted students to not only post their responses, but also share solutions with their classmates. Although the instructor typically responded to each student’s post every week, the bulk of interactive responses came from other students. This collective problem-solving taking place outside of class was critical to students’ ability to progress in the simulation.

In Week 4, following three weeks of instruction and simulation play, two exams were given. The first was the knowledge and attitude survey post-test. Students were informed that these results would not count towards their grades for the full course module. Second, students played the simulation one time individually with their final level attained (i.e., Apprentice, Novice, Manager, Leader, Expert, Change Master) counting for 25% of their course grade. This was designed to ‘incentivize’ students to play the simulation outside of class, knowing that they would have to play individually for the simulation exam.

3.3. Sample and Participants

The study employed a convenience sample of learners who voluntarily chose to join the LCS-B module as part of a course offered at a graduate school of management in Thailand. The students were in their fourth or fifth (final) term of the Master degree program. The simulation module was, therefore, designed to draw upon and extend their prior learning. Students did not, however, have any prior coursework related to sustainability or change management concepts.

Classes in the college were capped at 25 students. In order to achieve a sufficient sample size, data were collected from four different classes over a period of three trimesters between May 2021 and February 2022. The same instructor taught all four modules using the same support materials and version of the simulation. The total number of participants was 87, or 95.6% of the total of 91 students. One student dropped out during the course, and three others chose not to participate in the research.

Demographic data were collected from the participants in the pre-test (see

Table 3. Demographic characteristics of the student sample.

Classes	n of Students	%	%	Average GPA	Average Age (Years)	Average Working Experience (Years)
		Male	Female
Class 1	22	18%	82%	3.53	30	6
Class 2	21	43%	57%	3.47	29.5	5
Class 3	24	33%	67%	3.55	28.3	4.6
Class 4	20	45%	55%	3.64	27.7	3.9
All classes	87	34.50%	65.50%	3.54	28.9	5

). The average age of the students was 29 years. All of the students were studying part-time for their Master of Management degree with an average of five years of work experience. Although there was a bit more variation in the gender composition across the four modules, female students represented the majority in all four groups. Analysis of these demographic data indicated that there were no significant differences in the student composition of the four classes. Therefore, for the purposes of this article, the researchers were able to treat the four classes as a single group. Notably, the demographic distribution of the modules mirrors the broader demographics of the college.

3.4. Data Collection

In order to address the three research questions, the researchers collected a variety of data. These included data on student engagement, demographic characteristics, knowledge of change management and sustainability concepts, and attitudes towards sustainability issues.

Data on student engagement were collected from two sources. Each time that a student played at home, s/he was required to log into the simulation with a unique username. This enabled the researchers to capture data on student engagement with the simulation as well as on their decisions and results for every session played. For the purposes of this article, the frequency data were used insight into student engagement. Quantitative data on student participation in the weekly online discussion forum were also extracted for this purpose.

Data were collected from the pre/post-test of student knowledge and attitudes using an online Google form. Students were informed that the results of the pre/post-test would not be used as part of their grades. The knowledge test focused on two conceptual domains: change management and sustainability, with questions organized around a mini case. The ‘problem’ described in the mini case mirrored the type of sustainability problem presented in the LCS-B simulation but in a different corporate context.

Multiple choice questions were formulated at the application and analysis levels of Bloom’s Taxonomy of Cognitive Objectives [35]. Questions covered key elements of change management strategy drawn from the change theories addressed in the module. Key concepts included including managing resistance to change, individual differences in responses to organizational change, and stages in the change process.

The knowledge pre/post-test contained a separate section focusing on sustainability concepts including triple bottom-line, sustainable production, supply chain management, circular economy waste management, and social sustainability. These questions also used a multiple-choice format.

A Sustainability Mindset scale was developed for the purposes of this research [36]. This scale consisted of 10 Likert type items measured on a 1 to 5 scale using a measure from Strongly Disagree to Strongly Agree. The Sustainability Mindset scale incorporated statements focusing on student attitudes towards social, environmental, and economic dimensions of sustainability. Sample questions include:

• Because the future is unknown, we should maximize our benefits today
• My society should actively promote equal opportunities for males and females in school and the workplace.
• What I do as an individual does not have much impact on the environment. (reversed item)

3.5. Data Analysis

Data analysis began with the assessment of student engagement with the LCS-B simulation. Descriptive statistics were used to gain insight into patterns of student engagement with the simulation, as well as with classmates in the online discussion forum.

The pre/post-test of knowledge yielded a total score and sub-scores for the two conceptual domains of change management and sustainability concepts. The Shapiro–Wilk test was used to check the normality of the distribution of scores on the knowledge test. A paired t-test was employed to measure the change in learners’ knowledge of change management and sustainability concepts before and after the treatment (i.e., the learning with LCS-B simulation). A 0.05 level was used to determine the statistical significance of the change in learners’ knowledge. Cohen’s d is the effect size for two means comparison [37]. Cohen’s d of “1” means two groups differ by 1 standard deviation, “2” means they differ by 2 standard deviations. While Cohen argued against the use of arbitrary cutoffs, researchers often interpret a Cohen’s d of 0.10 to 0.30 as a small effect, 0.30 to 0.60 as a moderate effect, and > 0.60 as a large effect [37].

Change in learner attitudes towards sustainability issues was also assessed using the pre/post tests conducted in Weeks 1 and 4. Cronbach’s Alpha test was used to check the reliability of the Sustainability Mindset scale developed for this research. This study used a paired t-test to measure the change in the mean learner attitudes before and after the treatment with a 0.05 level of significance. Cohen’s d was again used to assess the size of the effect of the LCS-B on students’ sustainability mindset.

4. Results

4.1. Analysis of Student Engagement

Preliminary data analysis revealed that students evidenced strong motivation to succeed and persist in learning with the simulation. As noted above, after the first couple of times playing the simulation, students could complete a three-year session in 30 to 45 min. However, students were seldom able to succeed beyond the ‘Manager’ until they had played six or eight times. With this in mind, the researchers tracked the frequency of student games played with the simulation outside of class.

The mean number of games played in the three weeks by the 87 students was 22.2 games, or an average of 7.4 games per week. Students played an average of four games per student in week one, five games per student in week two and nine games per student in week three. Using an estimate of 40 min per game this indicates that students averaged about five hours of time outside of class per week playing the simulation. These data suggest that students were committing substantial time to playing the simulation outside of class and engaged in improving their change mastery week by week.

To what extent did students finally succeed in the simulation? This was measured by the “simulation exam” given in the fourth class session. Each student played the simulation individually one time in class with a 30-min time limit. Results on the simulation exam indicated that 82 of the 87 students were able to achieve the highest level, Change Mastery. This means that they were able to formulate and execute a change strategy that yielded a substantial number of Bennies (i.e., triple bottom-line benefits) and ‘moved’ at least 80% of the stakeholders into the Sustainability Stage of change.

The authors caution against interpreting the high level of student mastery of the simulation as an indicator that the simulation was ‘easy’. We instead suggest that the combination of challenge, realism, and continuous feedback on results combined to motivate students to play a sufficient number of times to ‘learn’ the elements of an effective change strategy.

Reference to student participation in the online discussion forum also offered insight into patterns of student engagement. The forum questions were structured to prompt student engagement with one another, identifying problems they were facing in the simulation and sharing effective strategies. To what extent did students engage in the Weekly forum? In week one, the 87 students contributed 240 posts in response to the questions on “managing resistance to change”. These posts included both their individual responses to the forum question as well as their responses to the contributions of other students. The response patterns for week two and three were 197 and 198 respectively. These totals reflect active participation in the online discussion forum. A rubric used to assess the quality of student participation confirmed that the quality of student participation was also generally high, averaging 3.4 on a four-point scale (not tabled).

4.2. Change in Student Knowledge of Change Management and Sustainability Concepts

The following analyses are based on the sample of 87 graduate students drawn from the four LCS-B module classes.

Table 4. Summary of knowledge test score (n = 87).

			Knowledge Pre-Test				Knowledge Post-Test
Knowledge Domain	No. of Students	Class	Min.	Mean	Max.	SD	Min.	Mean	Max.	SD
Full Test	22	1	0	6.0	11.3	2.98	7	11.0	15.0	1.89
(16 points)	21	2	0	6.0	12.0	2.93	5	10.4	15.0	2.65
	24	3	1	6.2	14.0	3.20	7	11.9	15.0	2.00
	20	4	2	7.2	11.0	1.96	9	12.4	16.0	2.17
All four classes	87		0	6.3	14.0	2.77	5	11.4	16.0
Sustainability	22	1	0	2.0	5.3		1	4.8	7.1
Concepts	21	2	0	1.8	5.0		2	4.2	8.0
(8 points)	24	3	0	2.2	6.0		3	5.3	7.0
	20	4	0	2.7	6.0		2	5.9	8.0
All four classes	87		0	2.1	6.0		1	5.0	8.0
Change Management	22	1	0	4.0	8.0		4	6.3	8.0
Concepts	21	2	0	4.0	7.0		2	5.7	7.0
(8 points)	24	3	0	3.9	8.0		4	6.6	8.0
	20	4	2	4.6	7.0		4	6.5	8.0
All four classes	87		0	4.1	8.0		2	6.3	8.0

shows the descriptive analysis of pre-test and post-test at three levels: full test, change management concepts, sustainability concepts. Note first that the pre-test scores were very low for the full test (M = 6.3 from 16 questions), sustainability concepts (2.1/8), and change management concepts (4.1/8). This means that most students across the four classes evidenced weak knowledge of both sustainability and change management concepts at the outset of the module.

In contrast, students demonstrated a stronger command of the content in post-test scores for the full test (M = 11.4/16), as well as for the sub-domains of sustainability (M = 5/8) and change management (M = 6.3/8). If 80% correct responses is used as the standard for concept mastery, on average students were still below mastery (i.e., 66%) of sustainability concepts and just about at mastery of change management concepts (M = 79%).

The boxplot in Figure 5 provides a visual illustration of the data presented in Table 4. The mean post-test scores for all four classes were substantially and consistently higher than the mean scores for the pre-tests. Indeed, the post-test scores for the full knowledge test improved by 84.6%, 71.6%, 91.9%, and 64% for classes 1–4 respectively.

The initial conclusion drawn from the data patterns in Figure 5 and Table 4 is that the pre/post-test score distribution across these four classes were quite consistent. Thus, we were able to combine data from the four classes into a single dataset for the purposes of the inferential statistical analysis. Normality analysis of the knowledge test data found that both the pre-test (skewness −0.036, SE = 0.26; kurtosis −0.125, SE = 0.51) and post-test scores (skewness −0.23, SE = 0.26; kurtosis −0.38, SE = 0.51) were normally distributed. The reliability of the knowledge test was 0.776, indicating an acceptable level of reliability [38].

A paired sample t-test was used to test the significance of differences between pre/post-test scores for the overall knowledge test, as well as for the sub-domains of change and sustainability concepts (see

Table 5. Paired sample t-test of differences between knowledge pre-test and post-test scores (n = 87).

	Paired Differences
				95% Confidence Interval
	Mean Difference Pre/Post-Test	Std. Dev.	Std. Error Mean	Lower	Upper	t	df	Sig. (2-Tailed)
Knowledge All Items: Post-test minus Pre-test	5.106	3.145	0.337	4.435	5.776	15.14	86	0.000
Knowledge Change Management: Post-test minus Pre-test	2.195	2.062	0.221	1.755	2.634	9.93	86	0.000
Knowledge Sustainability: Post-test minus Pre-test	2.899	1.810	0.194	2.513	3.285	14.94	86	0.000

). The test found that the change in mean scores on the pre/post-tests of knowledge (i.e., 16 items) was statistically significant (p < 0.001). The Cohen’s d statistic measuring effect size was 1.96, indicating a large effect of the LCS-B simulation module on students’ overall knowledge outcomes.

Large, statistically significant changes in student knowledge were also evident on each of the sub-domains of the test: change management (d = 1.42, p < 0.001) and sustainability (d = 1.84, p < 0.001). Notably, while the mean score on the sustainability post-test (i.e., 66%) fell below the 80% standard of mastery, the effect size associated with student gains in their knowledge of sustainability concepts was particularly impressive. This means that despite failing to achieve concept mastery, student knowledge of sustainability concepts (e.g., triple-bottom-line, circular economy, social sustainability, waste management) had increased substantially.

4.3. Change in Sustainability Mindset

A reliability test of the Sustainability Mindset scale yielded a Cronbach’s alpha coefficient of 0.645 for the 10-item attitude test based. Although this level was acceptable [39], we decided to remove two items which improved the alpha reliability to 0.675.

A paired sample t-test was used to assess change in the Sustainability Mindset of students based on the pre-test and post-test scores.

Table 6. Paired sample t-test of differences between sustainability mindset pre-test and post-test scores (n = 87).

		Paired Sample Statistic and Paired Differences					t	df	Sig. (2-Tailed)
		Mean	Standard Deviation	Standard Error Mean	95% Confidence Interval of the Difference
					Lower	Upper
	Pre-test	4.0856	0.4467	0.0479
	Post-test	4.4307	0.4727	0.0506
Pair 1	Pre-test–Post-test	Mean difference −0.3450	0.4280	0.4589	−0.4363	−0.2538	−7.519	86	0.000

shows the paired sample analysis of differences between pre-test and post-test scores on the Sustainability Mindset scale. The results found statistically significant change in students’ Sustainability Mindset scores from the pre-test (M = 4.0) to the post-test (M = 4.4). Analysis of the effect size of this change in student attitudes towards sustainability yielded a Cohen’s d statistic of 0.8 which is also interpreted as a large effect. Thus, the three-week LCS-B simulation module also yielded a significant positive change in student attitudes towards sustainability.

5. Discussion

This study employed a quasi-experimental research design to assess change in student knowledge and attitudes after learning with the three-week, online LCS-B simulation module. In this section, limitations of the research are highlighted, followed by the interpretation, and implications of the findings.

5.1. Limitations

The main limitation of this study follows from the research design. The quasi-experimental design used in this research was, in fact, much stronger than the perception-based evaluations that predominate in prior research on the effectiveness of sustainability simulations and games [17,23]. Nonetheless, the authors acknowledge that the lack of a control or comparison group was a clear limitation [34].

Therefore, although the study found ‘large, significant change in student knowledge and attitudes’ in the post-test phase of the research, the study cannot answer the question, “compared to what?” That is, this research design is not capable of offering insight into the relative effectiveness of the LCS-B simulation compared with alternate learning methods such as lecture, project-based learning, service learning, or a case study. Educators will, therefore, benefit from future research that compares learning outcomes of the LCS-B simulation with other learning methods.

A second limitation follows from the decision to limit this report to the quantitative results analyzing the simulation’s impact on student knowledge and attitudes. This research design offered useful insight into “what” and “how much” students learned from studying the LCS-B simulation module. However, the absence of complementary qualitative data limits our insights into students’ “experience” of the simulation and how this impacted their motivation to learn. Complementary results that address these and other issues will be disseminated in companion articles [22].

5.2. Interpretation of the Findings

The main objectives of the quasi-experimental study reported in this article were to assess whether learning with the LCS-B simulation module produced significant gains in student understanding of sustainability and change management concepts, as well as positive change in their attitudes towards sustainability issues. Recent reviews of research have challenged scholars studying simulation-based learning to use more powerful research designs that go beyond the evaluation of student perceptions [16,21,24]. This is also the case in the domain of sustainability where scholars have asserted that the search for active learning methods for learning about sustainability challenges and solutions will benefit from the use of stronger research designs [14,17,23]. This framed the quantitative approach adopted in the current study.

The results for all three research questions were unequivocal. Students evidenced large, statistically significant gains in their knowledge of both sustainability and change management concepts. In addition, their attitudes towards sustainability, measured with a Sustainability Mindset scale, also demonstrated large, statistically significant improvement. Thus, all three hypotheses proposed for this study were supported.

These findings affirm the efficacy of simulation-based learning in achieving the objectives of enhancing learners’ knowledge as well as strengthening their ‘sustainability mindset.’ This is a positive result that should encourage educators who are exploring the use of sustainability simulations and serious games. While comparative experimental studies are still needed to gain insight into the relative efficacy of simulation-based learning, the study’s results provide a useful extension of findings from previous experimental research on the use of simulations and games in educating for sustainability [14,17,19,23].

That said, the authors would also call attention to the fact that, despite these significant learning gains, students were still below (i.e., 66%) a ‘mastery level’ of knowledge (e.g., 80%) on the post-test for sustainability concepts. The inability to achieve a mastery level of knowledge was likely due to several factors. For example, as indicated in Table 4, the students started out with an extremely low level of baseline knowledge of sustainability concepts on the pre-test (25%). Very few students were familiar—even at the comprehension level—with basic sustainability concepts such as triple bottom-line, circular economy, or sustainable consumption. This suggests that while the three-week treatment did yield a significant increase in their knowledge of sustainability concepts, there was still room for learning about sustainability concepts.

This finding suggests that the LCS-B simulation should not be used as a stand-alone tool for learning about sustainability concepts. Instead, when students begin with a low baseline level of knowledge, the simulation should be supplemented by either a longer treatment duration (e.g., four-week module) or additional learning experiences. Indeed, the LCS-B module described in this article was followed by a four-week, project-based learning project that enabled students to learn about specific corporate sustainability practices in greater depth. Anecdotally, when the same knowledge exam was given after the seventh week of the module (i.e., instead of in the fourth week), students evidenced higher exam scores on the sustainability concepts (not tabled).

The third research question concerned the impact of learning with the LCS-B simulation on student attitudes towards sustainability, as measured by a Sustainability Mindset scale. As indicated above, although almost all students began the course with extremely low baseline knowledge of sustainability concepts, they evidenced a positive pre-disposition towards sustainability (i.e., pre-test mean = 4.08 for Sustainability Mindset). The large, significant gain on the five-point Likert scale suggests that student interest in sustainability issues was further stimulated as a result of their engagement with the LCS-B simulation.

The data presented on student engagement both with the simulation and in the online discussion forum associated with the module also offer insight into these positive results. Students committed substantial time outside of class playing the simulation individually and engaging in collaborative learning with their peers. While this was not emphasized earlier, it is notable that the 87 students participating in the study were all part-time graduate students. They were all working and taking three classes at the college concurrently. The data on student engagement indicate that the students were highly motivated by learning with the simulation, and therefore, able to overcome the constraints of learning in a fully online instructional environment.

We would also like to draw attention to a potentially overlooked feature of simulation-based learning as it was incorporated into this study. This concerns the ability of the students to “practice” with the LCS-B simulation. In many instructional settings, either the nature of the simulation or time constraints limit students to a single opportunity to play with the simulation or game. While this single-use approach can be useful, the results presented in this research study highlight the additional gains that accrue when learners have multiple opportunities to play the simulation. This feature of the simulation’s use facilitates “learning by doing” [40] as students are able to learn from the strategy errors made in previous game sessions. This enables learners to “see patterns” in the enactment of their strategic decision-making and results. This type of pattern recognition is associated with the development of expertise across different professions [41]. Thus, we suggest that greater attention be paid to this feature when educators are designing and using simulations.

These findings provide an initial validation of the LCS-B simulation as a tool for use in management education for sustainability. Indeed, they provide a useful complement to a prior evaluation study of the LCS-B simulation. Using perception data, the authors found that students found the simulation to be highly effective at enhancing their practical understanding of change and sustainability concepts, as well as their ability to think strategically [22].

As educators, we tend to focus on the outcomes that we typically in a course (e.g. conceptual knowledge). Yet, students who leave the program with a strengthened sustainability mindset will be motivated to learn more about what they can do to address sustainability challenges in the future. We consider that an equally important learning outcome. This was articulated by one student in her reflection on lessons learned in the LCS-B module.

To me I think the problem of sustainability is big enough that we should have a course on it in the university. The fact that we have this class in the Master program reflects the importance of the issue. Even though right now we know just only the tip of the iceberg, it is really useful because eventually we will have to use this knowledge no matter where we are working or what we are doing.

(Student 22)

5.3. Implications of the Findings

Several implications arise from this study. First, the findings provide initial validation of the LCS-B simulation in management education courses or workshops that focus on sustainability. Depending upon the learning objectives of the program, the simulation could be used either as an introduction to sustainability concepts, or for more advanced applications aimed at the implementation of sustainability solutions (e.g., strategic implementation). This implication drew support from our anecdotal observation that pairing the three-week LCS-B module with a project-based learning unit raised the level of student mastery of sustainability concepts dramatically.

Second, the use of the LCS-B simulation need not be limited to courses that focus on sustainability. The LCS-B simulation would also be well suited for use in generic management courses that focus on organizational change. In such cases, the sustainability initiative on which the LCS-B simulation is centered would simply be used as an example of a corporate change initiative. The key learning during the module would then focus on change management. Indeed, the authors have used the simulation in this type of course context with equally positive feedback from students.

Third, results from the initial validation studies [22] suggest the potential applicability of adapting the simulation for use beyond the corporate sector. Thus, offshoots of the LCS-B simulation are under development which apply a similar sustainability challenge and learning process in other organizational contexts. Currently, simulations are being designed for use in educating stakeholders for sustainability in K-12 school and higher education contexts. When implementing change in an organizational setting, the context matters. Thus, the adapted versions of simulation are being redesigned to reflect the contextual roles, norms, and activities that predominate in these other organizational settings.

Fourth, this research also has implications for the use of simulations and games more broadly in educating for sustainability. Scholars have asserted an urgent need for empirical evidence on the efficacy of different active learning approaches [17,23]. The results of this study provide encouragement for the potential of simulation-based learning in this domain of study. Moreover, we believe that the results reported in this study take on additional significance when we consider that the four classes which employed the LCS-B simulation were delivered in a 100% online instructional environment. Thus, the positive student learning and attitudinal outcomes were achieved without any face-to-face contact between the instructor and students, or among the students in a physical classroom.

This facet of the research highlights the potential that simulation-based learning can hold for teaching and learning in the post-COVID-19 era [27,28]. Schools, at every level, will continue to use online learning, whether in a blended or fully online mode, far more than prior to the pandemic. Thus, there is an urgent need for better information concerning which pedagogical methods are most effective, under what conditions, and with which level or groups of learners. The results of this study address this need.

Building on the fourth implication, we reprise uncertainties over how Asian students respond to the challenges of active learning. Our results indicate that this group of Asian students responded to the LCS-B simulation with high levels of motivation. This was evident in their frequent engagement with the simulation both inside and outside of class. Moreover, both their success on the simulation, as well as gains in knowledge and strengthened sustainability mindset while learning in a fully online instructional environment belie such stereotypes.

Finally, consistent with the previous recommendation, we revisit an implication raised by the earlier cited limitation of this study’s research design. Continued validation studies should compare the relative effectiveness of the LCS-B simulation against alternative learning modes (e.g., face-to-face in the classroom), as well as different learning methods (e.g., case study). This further suggests that the search for effective means of learning for sustainability should continue to be expansive. Educators will benefit from experiments that test the efficacy of a wider range of active learning methods in the quest to develop learner knowledge and strengthen positive attitudes that support sustainability solutions.