Macrosystems EDDIE Teaching Modules Increase Students’ Ability to Define, Interpret, and Apply Concepts in Macrosystems Ecology

Ecologists are increasingly using macrosystems approaches to understand population, community, and ecosystem dynamics across interconnected spatial and temporal scales. Consequently, integrating macrosystems skills, including simulation modeling and sensor data analysis, into undergraduate and graduate curricula is needed to train future environmental biologists. Through the Macrosystems EDDIE (Environmental Data-Driven Inquiry and Exploration) program, we developed four teaching modules to introduce macrosystems ecology to ecology and biology students. Modules combine high-frequency sensor data from GLEON (Global Lake Ecological Observatory Network) and NEON (National Ecological Observatory Network) sites with ecosystem simulation models. Pre- and post-module assessments of 319 students across 24 classrooms indicate that hands-on, inquiry-based modules increase students’ understanding of macrosystems ecology, including complex processes that occur across multiple spatial and temporal scales. Following module use, students were more likely to correctly define macrosystems concepts, interpret complex data visualizations and apply macrosystems approaches in new contexts. In addition, there was an increase in student’s self-perceived proficiency and confidence using both long-term and high-frequency data; key macrosystems ecology techniques. Our results suggest that integrating short (1–3 h) macrosystems activities into ecology courses can improve students’ ability to interpret complex and non-linear ecological processes. In addition, our study serves as one of the first documented instances for directly incorporating concepts in macrosystems ecology into undergraduate and graduate ecology and biology curricula.


Introduction
Over the past decade, macrosystems ecology has emerged as an important subdiscipline of ecology and environmental biology that examines how populations, communities, and ecosystem processes interact across multiple spatial and temporal scales [1,2]. Conducting macrosystems ecology research requires managing, visualizing, and analyzing heterogeneous, complex datasets in addition to using advanced computational and modeling techniques e.g., [3][4][5]. As macrosystems concepts and approaches become increasingly integrated into ecological research, ecology training programs need to evolve to prepare students to conduct macrosystems-scale science [6]. However, many skills commonly used to conduct macrosystems research, including analyzing long-term and high-frequency time series data and using ecosystem simulation models, are rarely taught in ecology curricula in the U.S. [7,8]. Table 1. Module titles and descriptions for developed Macrosystems EDDIE modules. All modules are published and available in the Environmental Data Initiative repository (EnvironmentalDataInitiative.org; accessed on 23 July 2021).

Module 1: Climate Change Effects on Lake Temperatures (Carey et al., 2018)
Students set up an ecosystem simulation model for a GLEON lake, then "force" the model with a climate scenario of their own design to assess how climate change will impact lake thermal structure. Students then use a distributed computing tool to run hundreds of different climate scenarios for their lake and examine tipping points in lake responses. While this module is not directly associated with a specific macrosystems concept, it is designed to introduce students to the modeling and computational techniques used in macrosystems ecology.

Module 2: Cross-Scale Interactions (Carey and Farrell 2019)
Students set up an ecosystem simulation model for a GLEON lake, then "force" the model with climate and land-use change scenarios and evaluate how regional and local drivers, respectively, interact across spatial scales to affect lake water quality.

Module 3: Teleconnections (Farrell and Carey 2019)
Students set up an ecosystem simulation model for a GLEON or NEON lake, then "force" the model with El Niño scenarios to compare how different lakes respond to a global teleconnection. Students test their hypotheses for how global drivers influence regional weather and interact with local lake characteristics to affect water temperature and ice cover.

Module 4: Macro-scale Feedbacks (Carey et al., 2020)
Students set up an ecosystem simulation model for a GLEON or NEON lake, then "force" the model with climate scenarios to examine how greenhouse gas emissions from different lakes may either increase or decrease in the future, thereby creating a feedback that either amplifies or diminishes global climate change.
We assessed the efficacy of the Macrosystems EDDIE modules as tools for teaching macrosystems ecology concepts through the use of pre-and post-module assessments. Our focal question was: What is the effect of integrating hands-on modeling and data analysis activities in ecology courses on students' ability to define, interpret, and apply concepts in macrosystems ecology? We focused on students' ability to define, interpret, and apply macrosystems concepts across dimensions of both knowledge and cognitive processes (terms hereafter italicized) [37][38][39][40]. These three processes (define, interpret, apply) are key components of higher order thinking, as defined by different pedagogical frameworks [37][38][39][40]. While we acknowledge that define, interpret, and apply are simplifications of complex cognitive processes, they represent different scaffolded components of learning at a high level. For example, in the initial stages of comprehension, students must first be able to define key concepts in their own words. This builds toward the ability to interpret data and visualizations to understand concepts, and finally, to apply knowledge by using the newly-learned concepts to solve new problems or accomplish new tasks.

Macrosystems EDDIE Modules
Each Macrosystems EDDIE module consists of a short (1 to~3-h), self-contained suite of scaffolded activities that instructors can adapt to meet the needs of their lecture or laboratory class. Macrosystems EDDIE modules teach macrosystems ecology concepts within the four-dimensional ecology education framework (4DEE), as adopted by the Ecological Society of America [41]. While the modules use freshwater ecology examples to teach macrosystems ecology, modules were specifically designed to be broadly applied to general ecology and biology courses, as well as more specialized courses. Thus, each module includes PowerPoint slides that introduce key concepts in limnology, freshwater ecology, and aquatic biology, but can be customized by instructors with additional background knowledge that may be needed for their students. Within each module, pairs of students begin by selecting a NEON or GLEON lake from a list of sites and then complete modeling activities that build from simple (activity A) to more complex (activities B and Educ. Sci. 2021, 11, 382 4 of 16 C). Because of the modules' flexible structure, instructors can tailor the activities to match their classroom's needs. For example, instructors of introductory courses may choose to teach module activities A and B in a class and then assign activity C as homework or omit certain activities, while instructors of more advanced courses may choose to assign activity A for pre-class homework and complete activities B and C in class. Modeling activities and associated worksheets and handouts follow the 5Es of learning (engagement, exploration, explanation, expansion, and evaluation) [35,42] and are designed to foster hypothesis-driven exploration of ecosystem model outputs following [43]. Modules generally increased in complexity from Module 1 to Module 4, though none of the modules have pre-requisites and modules can be used in any order as stand-alone activities.
Each module includes: (1) Microsoft PowerPoint slides for instructors to introduce the module's focal macrosystems ecology concepts and modeling activities, as well as key concepts in freshwater ecology; (2) an editable Microsoft Word student worksheet that prompts students to make hypotheses about different model scenarios, analyze model results, and determine whether their hypotheses were supported; and (3) an annotated R script that guides students through exercises using pre-packaged ecosystem models configured for each lake ecosystem. All modules were developed assuming no prior R or modeling experience for either students or instructors and can be accessed via the Macrosystems EDDIE website (www.MacrosystemsEDDIE.org; accessed on 23 July 2021) or the Environmental Data Initiative (EDI) repository [44][45][46][47].

Module Implementation and Assessment
We assessed the effectiveness of the four Macrosystems EDDIE modules in 23 classrooms at 17 colleges and universities and one conference workshop (see Supplementary Material, Table S1). Modules were primarily taught in undergraduate classrooms in freshwater/aquatic ecology or biology courses (n = 11), with other courses ranging from introductory environmental science to environmental informatics and global change ecology. Due to the wide range in classroom size (ranging from 3 to 39 students), course type, and student and instructor experience levels (see Supplementary Material, Table S1), analyses were conducted on the overall pool of students (n = 319) and were not separated by institution or classroom. Since we include students from mixed-enrollment (undergraduate plus graduate) and graduate classrooms in our dataset and analysis and are unable to separate their anonymized responses, we focus the discussion on biology and ecology curricula broadly and do not specifically distinguish between undergraduate or graduate responses. Each classroom taught a single module (Module 1, 2, 3, or 4) during a semester, except for three classrooms that taught two modules during a semester (see Supplementary Material,  Table S1). For the three classrooms that taught two modules, we focused on data from the first module taught to be consistent with the other single-module classrooms.
Pre-and post-module assessments were administered within 2 weeks before and after module instruction, respectively, to evaluate changes in the students' ability to define, interpret, and apply macrosystems ecology concepts. Pre-and post-module assessment questions were identical to each other and had a similar structure among the four modules. The assessments included both self-assessment questions designed to evaluate students' self-perceived knowledge of macrosystems ecology and quantitative assessments designed to evaluate students' knowledge of macrosystems ecology by defining key words, interpreting complex data visualizations, and applying concepts in macrosystems ecology to new problems.
The pre-and post-module assessments included four types of questions to assess students' growth. All pre-and post-module assessments were administered online, with no time limit or restrictions on materials that could be used to answer the questions. Assessments were not graded but were used by some course instructors for extra credit if the student completed both pre-and post-module assessments. First, we used multiple-choice questions where students ranked their perceived proficiency and confidence using highfrequency and long-term data and their self-perceived knowledge of macrosystems ecology on a Likert-type scale from 1 (low, "Not at all familiar") to 5 (high, "Extremely familiar") (see Supplementary Material, Table S2). Second, free-response questions were designed to assess students' ability to define macrosystems ecology concepts based on the module they were completing (see Supplementary Material, Table S3). Third, multiple-choice questions evaluated students' ability to interpret data visualizations from model output related to the module's focal concepts (see Supplementary Material, Table S4). Finally, a free-response question evaluated students' ability to apply macrosystems ecology concepts to describe the approach they would use to predict lake ecosystem responses to climate and/or land-use change in the year 2099 (see Supplementary Material, Table S5). The latter three types of questions enabled us to move beyond self-perceptions of knowledge to quantify students' growth in response to the modules.
We analyzed the pre-and post-module assessments using standard methods [48,49]. For the define and apply questions, we used a standardized coding method [48] to classify qualitative responses and Wilcoxon signed-rank tests to compare coded pre-and postmodule responses (Appendix A). We also used Wilcoxon signed-rank tests to compare preand post-module responses for the self-assessment and interpret questions (Appendix A). Three interpret questions were excluded from analysis due to the 'ceiling effect' (correct pre-module responses >85%), which limits opportunity for evaluating growth [49,50] but are reported in the supplementary information ( Figure S1). Statistical significance was defined a priori as α = 0.05 using p-values based on a normal approximation. For each metric (proficiency, confidence) and assessment question (high-frequency data; long-term data), effect sizes were calculated as Z/ √ n. All students consented to participate in the study per our Institutional Review Board (IRB) protocol (Carleton College IRB #0002470). Additional details on module implementation and quantitative assessment are included in Appendix A.
There was a wide range in the number of student responses per question (see Supplementary Material, Table S7), and the assessment questions varied among modules. A total of 319 students who consented to participate in the study completed some portion of either the pre-or post-module assessment. For our analyses, however, a student needed to complete both the pre-and post-module response for a specific question to be included in the analysis. In addition, we asked instructors to teach modules that best aligned with their course; as such, the total number of classrooms testing each module varied (see Supplementary Material, Table S1). Thus, given differences in student response rates to individual questions and the numbers of students completing each module, the total number of students used in the analysis of each question varied (see Supplementary Material,  Table S7). Second, the types of questions included in the assessments for each module (Module 1-4) varied slightly. While we generally sought to keep assessment questions consistent among modules, some changes were made in subsequent modules as we learned from ongoing module testing. By aggregating results across modules, we feel our analyses are applicable to the overall module goals and to a range of classroom sizes, course levels, and institutions.

Increased Familiarity with Macrosystems Ecology and Greater Confidence Using Data
Based on the self-assessment questions, students' median self-reported knowledge of macrosystems ecology significantly increased after completing a Macrosystems EDDIE module (p < 0.001, effect size = 0.66; Figure 1, see Supplementary Material, Table S8). On average, students described their pre-module knowledge of macrosystems ecology as 'Slightly familiar, I have heard of macrosystems ecology, but cannot elaborate' (median Likert score of 2), and their post-module knowledge as 'Somewhat familiar, I could explain a little about macrosystems ecology' (median Likert score of 3). There was also a significant increase in students' self-reported confidence and proficiency using both high-frequency and long-term data after completing a module (p < 0.001, effect size = 0.60 and 0.58, respectively; Figure 2, see Supplementary Material, Table S8). Students' median self-reported proficiency and confidence working with long-term data was higher than for high-frequency data both before and after module use, but students exhibited greater gains in their proficiency and confidence working with high-frequency data after module completion (see Supplementary Material, Table S8). significant increase in students' self-reported confidence and proficiency using both highfrequency and long-term data after completing a module (p < 0.001, effect size = 0.60 and 0.58, respectively; Figure 2, see Supplementary Material, Table S8). Students' median selfreported proficiency and confidence working with long-term data was higher than for high-frequency data both before and after module use, but students exhibited greater gains in their proficiency and confidence working with high-frequency data after module completion (see Supplementary Material, Table S8).  Table S8 for statistical results).  Table S8 for statistical results).

Hypothesis-Driven, Hands-On Activities Promote Macrosystems Ecology Learning
In addition to comparing students' self-perceived knowledge of macrosystems ecology, as reported above, we also directly evaluated students' ability to define, interpret, and apply key concepts in macrosystems ecology before and after module use. Overall, the proportion of students able to correctly define "macrosystems ecology" significantly increased (p = 0.03, effect size = 0.22) from 7.4% to 16.8% after completing a module ( Figure 3, see Supplementary Material, Table S9). Moreover, significantly more students also correctly defined "cross-scale interactions" and "teleconnections" (p = 0.004, effect size = 0.29 and p = 0.02, effect size = 0.21, respectively) following the completion of modules on these topics. The proportion of students able to correctly define macro-scale feedbacks also increased, from 11.9 to 19.1%, however, this increase was not statistically significant (p = 0.35, effect size = 0.14). These define questions specifically sought to capture students understanding of the multiple interacting scales across which ecological processes occur (see Supplementary Material, Table S3). Specifically, for the macrosystems ecology definition question, there was a significant increase post-module in student responses noting that macrosystems ecology inherently contains processes that occur across multiple temporal and/or spatial scales (10.5% to 23.2%, Figure S2).  Table S8 for statistical results).

Hypothesis-Driven, Hands-On Activities Promote Macrosystems Ecology Learning
In addition to comparing students' self-perceived knowledge of macrosystems ecology, as reported above, we also directly evaluated students' ability to define, interpret, and apply key concepts in macrosystems ecology before and after module use. Overall, the proportion of students able to correctly define "macrosystems ecology" significantly increased (p = 0.03, effect size = 0.22) from 7.4% to 16.8% after completing a module ( Figure  3, see Supplementary Material, Table S9). Moreover, significantly more students also correctly defined "cross-scale interactions" and "teleconnections" (p = 0.004, effect size = 0.29 and p = 0.02, effect size = 0.21, respectively) following the completion of modules on these topics. The proportion of students able to correctly define macro-scale feedbacks also increased, from 11.9 to 19.1%, however, this increase was not statistically significant (p = 0.35, effect size = 0.14). These define questions specifically sought to capture students′ understanding of the multiple interacting scales across which ecological processes occur (see Supplementary Material, Table S3). Specifically, for the macrosystems ecology definition question, there was a significant increase post-module in student responses noting that macrosystems ecology inherently contains processes that occur across multiple temporal and/or spatial scales (10.5% to 23.2%, Figure S2).

Figure 2.
Students' self-reported proficiency and confidence using high-frequency and long-term data significantly increased after module use. Questions used a Likert scale from 1 (low) to 5 (high). Jittered points represent individual students; boxplots represent the median response (dark horizontal line) while the boxes represent the interquartile range (IQR; 25th and 75th percentile), the whiskers correspond to 1.5x IQR; n indicates number of students with paired pre-and post-responses (see Supplementary Material, Table S8 for statistical results).   Table S9 for statistical results).
A second skill necessary to conduct macrosystems ecology is the ability to create and interpret data visualizations [25], which is also an important component of several pedagogical frameworks used to commonly assess student comprehension e.g., [37][38][39][40]. All students completing Macrosystems EDDIE modules create multiple visualizations in R to interpret their ecosystem simulation model output and assess whether their initial hy-  A second skill necessary to conduct macrosystems ecology is the ability to create and interpret data visualizations [25], which is also an important component of several pedagogical frameworks used to commonly assess student comprehension e.g., [37][38][39][40]. All students completing Macrosystems EDDIE modules create multiple visualizations in R to interpret their ecosystem simulation model output and assess whether their initial hypotheses were supported. After completing a module, there was generally an increase in students' ability to correctly interpret data to answer questions about macrosystems ecology-related concepts (Figure 4, see Supplementary Material, Table S10, Figure S1), though only one question showed a statistically significant increase. Below, we offer several reasons potentially driving these results.
Educ. Sci. 2021, 11, x FOR PEER REVIEW 9 of 17 introductory ecology course). Thus, gains post-module could have resulted in part from an increased knowledge of concepts in freshwater ecology and not necessarily from an increased understanding of macrosystems ecology concepts (e.g., Module 1, Question 1). Finally, we recognize that gains in student comprehension following module use could be attributed to students' ability to use logical reasoning to arrive at the correct answer (i.e., process of elimination; interpretation of graphical output, etc.) rather than an increased comprehension of macrosystems ecology. While we acknowledge these caveats, we nonetheless think these questions represented an important component of our preand post-module analysis to assess whether there was evidence for increased comprehension of macrosystems ecology concepts following module use, especially given the inherent challenges of assessing student growth across a range of diverse classrooms and experience levels.  Table S10 for statistical results).
Finally, students were significantly more likely to apply macrosystems approaches to describe how they would predict changing ecosystem responses to future land use and/or climate change after completing a Macrosystems EDDIE module. The responses to this question suggest a greater understanding of the complexity of ecosystem responses and the need for more sophisticated computational and modeling techniques to predict future changes. For example, significantly more students described the use of simulations, models, and/or forecasts in their responses ('model'; p < 0.001, effect size = 0.48; Figure 5, see Supplementary Material, Table S11), while statistically fewer student responses included the use of "trend lines" ('trend'; p < 0.001, effect size = 0.25; Figure 5, see Supplementary Material, Table S11). The increase in student responses that described using ecosystem models suggests an increased understanding of the complexity associated with predicting ecosystem processes and the need to use more sophisticated techniques (e.g., simulation modeling) over more simplistic linear trends and empirical models, which can fail to account for the complexity of ecosystem responses. Similarly, there was a significant increase in the percent of student responses that described assessing multiple lakes or multiple land-use and/or climate change scenarios ('many'; p < 0.001, effect size = 0.36; Figure  5, see Supplementary Material, Table S11), versus just one ecosystem or scenario. However, there was no change in the percent of responses that included 'data' (p = 0.61, effect size = 0.03), indicating many students already recognized the importance of collecting data to answer questions in macrosystems ecology prior to module use.  Table S4); n indicates number of paired pre-and post-module student responses; stars (*) indicate statistically significant (p < 0.05) increases between pre-and post-module assessments (see Supplementary Material, Table S10 for statistical results).
First, the questions removed due to the ceiling effect (>85% correct pre-response questions) suggest that students might already have been familiar with interpreting graphical output prior to module use, indicating that students were able to appropriately reason correct responses based on a given graphical output without prior knowledge of the concepts (i.e., concepts in macrosystems ecology). We thus speculate that the questions removed due to the ceiling effect may have been testing students' ability to interpret graphical output, as opposed to being able to interpret new macrosystems ecology concepts represented. We also acknowledge that some of the interpret questions may have required knowledge of freshwater ecology concepts that some students may not have been familiar with prior to completing a Macrosystems EDDIE module (i.e., in an introductory ecology course). Thus, gains post-module could have resulted in part from an increased knowledge of concepts in freshwater ecology and not necessarily from an increased understanding of macrosystems ecology concepts (e.g., Module 1, Question 1). Finally, we recognize that gains in student comprehension following module use could be attributed to students' ability to use logical reasoning to arrive at the correct answer (i.e., process of elimination; interpretation of graphical output, etc.) rather than an increased comprehension of macrosystems ecology. While we acknowledge these caveats, we nonetheless think these questions represented an important component of our pre-and post-module analysis to assess whether there was evidence for increased comprehension of macrosystems ecology concepts following module use, especially given the inherent challenges of assessing student growth across a range of diverse classrooms and experience levels.
Finally, students were significantly more likely to apply macrosystems approaches to describe how they would predict changing ecosystem responses to future land use and/or climate change after completing a Macrosystems EDDIE module. The responses to this question suggest a greater understanding of the complexity of ecosystem responses and the need for more sophisticated computational and modeling techniques to predict future changes. For example, significantly more students described the use of simulations, models, and/or forecasts in their responses ('model'; p < 0.001, effect size = 0.48; Figure 5, see Supplementary Material, Table S11), while statistically fewer student responses included the use of "trend lines" ('trend'; p < 0.001, effect size = 0.25; Figure 5, see Supplementary Material, Table S11). The increase in student responses that described using ecosystem models suggests an increased understanding of the complexity associated with predicting ecosystem processes and the need to use more sophisticated techniques (e.g., simulation modeling) over more simplistic linear trends and empirical models, which can fail to account for the complexity of ecosystem responses. Similarly, there was a significant increase in the percent of student responses that described assessing multiple lakes or multiple land-use and/or climate change scenarios ('many'; p < 0.001, effect size = 0.36; Figure 5, see Supplementary Material, Table S11), versus just one ecosystem or scenario. However, there was no change in the percent of responses that included 'data' (p = 0.61, effect size = 0.03), indicating many students already recognized the importance of collecting data to answer questions in macrosystems ecology prior to module use.
Educ. Sci. 2021, 11, x FOR PEER REVIEW 10 of 17 Figure 5. The percentage of student responses to the apply questions that contained thematic keywords. Responses including 'many' or multiple model runs and 'model' or the use of ecosystem simulation models increased after module use, while 'trend' or the use of linear trendlines to estimate responses in the future decreased. We also report changes in preand post-module responses that included 'data' or the use to data to predict how lake ecosystems will change in the future. Student responses were aggregated across modules (modules 2, 3, and 4; n = 232 total). Stars (*) indicate statistically significant (p < 0.05) differences between pre-(grey) and post-module (blue) assessments (see Supplementary Material, Table  S11 for statistical results).

Discussion
Taken together, results from the define, interpret, and apply assessment questions demonstrate that short (1-3 h) inquiry-based, hypothesis-driven, ecosystem modeling exercises, such as those in the Macrosystems EDDIE modules, can help to introduce ecology and biology students to macrosystems ecology. In each Macrosystems EDDIE module, students are introduced to ecosystem simulation modeling, which enables students to develop and test their own hypotheses about ecological phenomena using model scenarios. By explicitly asking students if their hypotheses are supported by their modeling results Figure 5. The percentage of student responses to the apply questions that contained thematic keywords. Responses including 'many' or multiple model runs and 'model' or the use of ecosystem simulation models increased after module use, while 'trend' or the use of linear trendlines to estimate responses in the future decreased. We also report changes in pre-and post-module responses that included 'data' or the use to data to predict how lake ecosystems will change in the future. Student responses were aggregated across modules (modules 2, 3, and 4; n = 232 total). Stars (*) indicate statistically significant (p < 0.05) differences between pre-(grey) and post-module (blue) assessments (see Supplementary Material, Table S11 for statistical results).

Discussion
Taken together, results from the define, interpret, and apply assessment questions demonstrate that short (1-3 h) inquiry-based, hypothesis-driven, ecosystem modeling exercises, such as those in the Macrosystems EDDIE modules, can help to introduce ecology and biology students to macrosystems ecology. In each Macrosystems EDDIE module, students are introduced to ecosystem simulation modeling, which enables students to develop and test their own hypotheses about ecological phenomena using model scenarios. By explicitly asking students if their hypotheses are supported by their modeling results as part of the module activities, students are encouraged to critically examine their own understanding of macrosystems ecology. Considering that only a few students in our study had been introduced to macrosystems ecology prior to our study-based on discussions with instructors and the low proportion of correct definitions in the pre-module assessments (Figure 3)-the significant growth in student understanding we observed provides support for a short, self-contained, module-based approach to integrate complex macrosystems ecology concepts into undergraduate curricula.
To the best of our knowledge, Macrosystems EDDIE is one of the first formalized educational programs to incorporate macrosystems ecology concepts directly into ecology and biology curricula. Our study indicates that pre-packaged modules such as those in the Macrosystems EDDIE program may be a relatively easy and efficient way for instructors to formally incorporate new concepts into their classrooms, compared to developing their own lectures and/or lab materials. Due to the flexible nature of the modules, all three module activities can be completed within one 3-4 h lab period or three 60-min lecture periods, depending on student experience level and course structure. In addition, modules were developed assuming no prior knowledge of R or ecosystem modeling for either the instructors or students, allowing implementation in a variety of classroom settings and skill levels. While multiple modules can be incorporated within curricula, our assessment results demonstrate that even single-module use leads to significant gains in student comprehension of macrosystems ecology in addition to increases in students' self-perceived knowledge and confidence using high-frequency datasets. Our module assessments included students from mixed-enrollment (undergraduate plus graduate) or graduate classrooms, although the majority of our respondents were enrolled in undergraduate-only classes, and previous work on other modeling-based modules suggests undergraduates may exhibit greater learning gains than graduate students through such activities [51].
Multiple recent studies have highlighted the challenges of teaching concepts and processes that span multiple spatial and/or temporal scales in undergraduate science courses [26,28], a key component of the sub-discipline of macrosystems ecology. Here, we demonstrate how hands-on activities embedded into ready-to-use teaching modules can be used to successfully teach students ecological concepts that span multiple temporal and spatial scales. Similar to this study, Styers et al. (2021) observed gains in students' understanding of cross-scale biodiversity processes following the use of hands-on teaching modules that analyzed NEON data collected across multiple spatial scales [28]. Taken together, our study and Styers et al. (2021) highlight the utility of hands-on activities that analyze real environmental data to teach students complex concepts spanning multiple scales, an important component of macrosystems ecology [28].

Module Limitations and Potential Improvements
The student assessments highlight potential improvements for future module use and assessments. First, we note that there was a wide range in the number of paired preand post-assessment responses across modules due to a varying number of classrooms that completed the different modules, in addition to a wide range in student experience level and institution type, which likely impacted our results. While questions were specific to the macrosystems ecology concepts contained in each respective module (i.e., crossscale interactions, teleconnections, or macro-scale feedbacks), the complexity and structure of the questions were designed to be comparable across modules (Tables S3-S5). By aggregating responses across modules, we feel that the results from this study can be broadly applied to a wide range of institution types, classroom sizes, and student experience levels, and are broadly applicable across all modules. While this assessment design allowed for comparisons across modules, future improvements could be made to ensure similar classroom sizes and student response rates across all modules for better comparisons, as well as recruiting a more representative distribution of institutions.
Similarly, we were unable to parse out individual classroom experiences in our analysis, as each classroom varied across multiple axes including institution type, classroom level, instructor experience, classroom size, and course focus/goals. Thus, we did not have enough statistical power to compare individual classrooms. Future studies should focus on recruiting classrooms that may be similar on one metric (i.e., size) but differ on another (i.e., classroom level) to determine differences in module effectiveness among classrooms. More granular studies that parse the effectiveness of module elements based on studentand classroom-specific characteristics could highlight ways that module activities could be customized for different teaching contexts. Finally, a longer-term study is needed to quantify students' lasting growth in comprehension and skills, especially in courses that use multiple modules throughout a semester. While our results suggest short-term gains in student comprehension following single-module use, we did not include longer (>2-week) post-module assessments to assess longer-term (months to years) student comprehension.
In addition to the improvements for module assessment, student assessments and informal instructor feedback also highlighted potential module improvements. Specifically, the process-based model used in the four Macrosystems EDDIE modules can effectively demonstrate the non-linear, interacting processes that occur in lake ecosystems through the simulation of multiple ecosystem variables, represented by linked equations [21]. However, because of the model's complexity, students are unlikely to fully understand its mechanistic underpinnings during a 1-3 h module. A simpler ecosystem model that allows students to understand and manipulate individual model components would enable additional engagement and interaction, as highlighted by several instructors during informal postmodule reflections.

Conclusions
Here, we demonstrate the effectiveness of Macrosystems EDDIE modules to teach undergraduate students' concepts in macrosystems ecology while also increasing students' self-reported confidence and proficiency using high-frequency and long-term data. Importantly, the skills (i.e., high-frequency and long-term data analysis; ecosystem modeling) gained during module use can be broadly applied to fields outside of ecology and can enhance students' abilities to use large data sets in many emerging fields and career paths both within and beyond STEM. In addition, incorporating hands-on activities into undergraduate and graduate classrooms has been shown to lead to increased gains in student comprehension [52]. Moreover, an increasing number of studies have highlighted the importance of incorporating active learning into traditional, lecture-based courses to help close the achievement gap between historically underrepresented and historically overrepresented students, e.g., [53][54][55].
Importantly, our work highlights one of the first sets of teaching modules designed to integrate macrosystems ecology concepts and approaches into ecology and biology classrooms. As highlighted by recent studies [26,28], students need a greater understanding of complex and non-linear ecological interactions that occur across multiple temporal and spatial scales. While teaching concepts in ecology that span multiple scales can be challenging, students' greater understanding of the complexity of ecological systems after completing a module suggests that short-term (1-3 h) modeling activities, such as those embedded in each Macrosystems EDDIE module, can be an effective approach for helping to prepare students to tackle global environmental challenges that occur across multiple temporal and spatial scales.

Supplementary Materials:
The following are available online at https://www.mdpi.com/article/10 .3390/educsci11080382/s1, Table S1: Description of classrooms assessed; Table S2: Likert-type scales for student pre-and post-module self-assessment questions; Table S3: Define questions for pre-and post-module responses and coding criteria; Table S4: Interpret questions for pre-and post-module responses for the four modules; Table S5: Questions for the apply assessments; Table S6: Coding criteria for apply questions; Table S7: Number of paired student responses for each module and each assessment question; Table S8. Statistical results for student self-reported proficiency and confidence using high-frequency and long-term data and their self-reported knowledge of macrosystems ecology; Table S9: Statistical results for define questions; Table S10: Statistical results for interpret questions; Table S11: Statistical results for apply questions; Figure S1: Results from interpret questions including those removed from analysis due to the ceiling effect; Figure S2: Percent of students for the define questions that included elements of "ecology", "effect", and "scale" thematic bins.

Acknowledgments:
We thank all of the instructors and students who enthusiastically tested Macrosystems EDDIE modules. We thank Kristin O'Connell from the Science Education Research Center (SERC) of Carleton College for assistance with qualitative data analyses as well as Ellen Iverson, Cailin Huyck Orr, Monica Bruckner, and Sean Fox from SERC for providing Macrosystems EDDIE programmatic and web support. We also thank members of the Reservoir Group at Virginia Tech for helpful comments to improve the manuscript. In addition, we thank two anonymous reviewers and the handling editor for their feedback which greatly improve the manuscript.

Conflicts of Interest:
The authors declare no conflict of interest. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results.

Appendix A Appendix A.1. Assessment Description
First, the pre-and post-module assessments included multiple choice questions ranking students' perceived proficiency and confidence using high-frequency and long-term data on a Likert-type scale from low (1) to high (5) and self-reported knowledge of macrosystems ecology, from 'not at all familiar' (1) to 'extremely familiar' (5) (see Supplementary Material, Table S2). Such self-assessments are valuable tools for students to gauge their own understanding and how it changes over time, as well as motivate further learning [56].
Next, students completed free-response questions designed to assess their ability to define macrosystems ecology concepts based on the module they were completing; specifically, cross-scale interactions and macrosystems ecology (Module 2), teleconnections (Module 3), and macro-scale feedbacks (Module 4, see Supplementary Material, Table S3). Module 1 did not include a define question as this module did not focus on a specific concept in macrosystems ecology but instead was designed to provide a broad introduction to the modeling and computational techniques used in macrosystems ecology. In addition, we removed the define macrosystems ecology assessment question from subsequent modules (Modules 3 and 4) to limit survey fatigue and the keep the assessments <10 min.
Following the define questions, a series of multiple-choice questions evaluated students' ability to interpret data visualizations. The graphs and corresponding data interpretation questions were developed to represent a similar level of complexity across modules (see Supplementary Material, Table S4). These questions were designed to assess students' ability to interpret ecosystem simulation model output as related to the module's focal concepts. The data interpretation questions within each module were scaffolded to increase in difficulty from question 1 (least difficult) to question 3 (most difficult). We excluded questions from analysis in which >85% students answered the pre-module assessment correctly because of the limited ability of those questions to evaluate students' learning (by the presence of the 'ceiling effect,' following [49,50]).
Finally, students completed a free-response question that evaluated students' ability to apply macrosystems ecology concepts, including the use of ecological simulation models and high-frequency and long-term data, by asking them to describe how they would predict lake ecosystem responses to climate and/or land-use change in the year 2099 (see Supplementary Material, Table S5). As with the define question, we did not include an apply question for Module 1 as this module did not focus on a specific concept in macrosystems ecology.

Appendix A.2. Student Assessment Analysis
We used Wilcoxon signed-rank tests to compare the mean Likert scores among paired pre-and post-module student responses for their perceived proficiency and confidence using high-frequency and long-term data, and for self-reported knowledge of macrosystems ecology following single module use. Responses were aggregated across modules and only included paired pre-and post-module student responses for a given question. Statistical significance was defined a priori as α = 0.05 using p-values based on a normal approximation. For each metric (proficiency, confidence) and assessment question (high-frequency data; long-term data), effect sizes were calculated as Z/ √ n. For data interpretation (interpret) questions, the percentage of students who answered each question correctly was compared in the pre-and post-module assessments and analyzed for significance using Wilcoxon signed rank-tests.
The qualitative responses for the define and apply free-response questions were coded in two steps using a provisional coding method [48]. The first step identified themes in student responses that were used to develop consistent coding criteria that could be applied across responses for each individual question. The second step applied the coding criteria to all pre-and post-module student responses. This process resulted in a database of student responses that were consistently coded for the presence or absence of identified themes (details below). For the define and apply questions, pre-and post-assessment responses were aggregated and randomized prior to coding During step one of coding, two Macrosystems EDDIE coordinators preliminarily reviewed student responses to each assessment question and noted emerging themes and keywords to develop an initial codebook. After developing the initial codebook, a set of 10% of the pre-and post-module responses for each question were randomly selected. This set of responses was then coded independently by two Macrosystems EDDIE coordinators for each question. Coding responses were then compared, coding disagreements resolved, and the codebook further refined to best characterize student responses.
During step two, student responses for each individual module and question were divided among and coded by two Macrosystems EDDIE coordinators using the refined codebook. As during the codebook development, all identifying student, course, and survey timing (pre-or post-module) information was hidden and all responses were randomly ordered based on a random number generator. This two-phase process resulted in coded student responses for each question of presence or absence for each thematic bin. Question responses that were left blank or included 'I don't know' were excluded from further analysis.
For the define questions (Modules 2, 3, 4), coding was separated into thematic bins by module and question (see Supplementary Material, Table S3). 'Correct' student answers included elements from each thematic bin. All other answers were coded as 'incorrect.' Wilcoxon signed-rank tests were then used to compare the percentage of correct answers pre-and post-module. We further divided responses by each thematic bin (i.e., 'eco' (ecology), 'effect', 'scale' for macrosystems ecology definition; see Supplementary Material, Table S3) which each student answer included, allowing us to assess if students' answers increased in complexity pre-and post-module use. For the apply questions (Modules 2, 3, 4), responses were aggregated and coded across all modules using a common set of thematic bins (see Supplementary Material, Table S6). Wilcoxon signed-rank tests were used to compare the percentage of responses that included each thematic bin in pre-and postmodule responses. For the define, interpret, and apply questions, results include students who completed either one or two modules. All statistical analyses were conducted in R 4.0.3 [57].