Engaging Middle School Students in Authentic Scientific Practices Can Enhance Their Understanding of Ecosystem Response to Hurricane Disturbance

Abstract

Ecosystem response to hurricane disturbance is complex and multi-faceted. The likelihood of increased frequency of severe hurricanes creates a need for the general public to understand how ecosystems respond to hurricanes. Yet, opportunities to study disturbances to complex systems are rare in U.S. K–12 schools. Educators and researchers in the Luquillo Long-Term Ecological Research program used the results of research on ecosystem response to hurricane disturbance in the Luquillo Experimental Forest as a foundation for the development of Journey to El Yunque, a web-based, middle-school curriculum unit. The program engages students in using models as evidence to develop explanations for how particular species respond to hurricane disturbance. Prior research in education has shown that engaging students in a particular role, like that of a scientist, could have detrimental effects on students’ abilities to transfer what is learned from one context to another. In this research, we sought to understand whether having students engage in authentic scientific practices could support transfer of knowledge to the abstract context of a standards-based assessment. Students were randomly assigned to engage in the program in the role of a scientist or in the role of a student learning about an ecosystem. The dependent variables included students’ comprehension of the background readings, their predictions of population changes, and their overall learning of ecology. The results indicate that taking on a scientist role during the learning activities had an indirect effect on general ecology knowledge by increasing the quality of students’ notetaking during background reading. The results also indicate that students struggled to use their knowledge to develop a robust explanation for how species respond to hurricane disturbance. Journey to El Yunque provides a framework for engaging students in authentic investigations of hurricane disturbance. Future research will examine how to improve the quality of students’ final explanations.

1. Introduction

Hurricane Maria made landfall in Puerto Rico as a category 4 hurricane on 20 September 2017. In the aftermath of the storm, the entire island was without running water, electricity, or communications [1]. In addition to concerns about the recovery of the infrastructure, there was much concern in the media and general public about the ability of the El Yunque National Forest in Puerto Rico to recover [2]. These concerns reveal common misconceptions about the dynamics of forest ecosystems. Seeing pictures of stripped and fallen trees, where once was luscious green forest, led many to believe that the forest was damaged beyond repair.

As the research in this volume suggests, ecologists paint a different picture of the El Yunque National Forest as part of the Luquillo Long-Term Ecological Research (LTER) program in Puerto Rico [3]. Rather than view the rainforest at a particular point in time, disturbance ecologists view the rainforest as a dynamic system that is adapted to being struck by hurricanes on a regular basis. There are a variety of species whose niche is the bright sunlight and abundant debris conditions that exist after a hurricane. These species keep precious nutrients in the forest and form the foundation for recovery.

Understanding the dynamics of complex systems is essential for scientific decision making [4]. Yet, complex systems are not generally taught in school, which has negative consequences for students’ ability to understand what happens when complex systems are impacted by disturbances [5]. Models are an important source of evidence about the dynamic, long-term changes within a system. Using models as evidence to develop an argument about how a system behaves is a complex task with a wide variety of subtasks. These subtasks related to comprehending evidence in relation to ideas, evaluating ideas in relation to evidence, and producing explanations based on evidence are complex and difficult. Most students have trouble with explanation and argumentation across all subject areas [6,7,8,9], but particularly with argumentation and explanation in science [10,11,12,13,14]. For instance, student explanations often focus on a single major cause to explain the scientific phenomena in question (i.e., lack completeness), while ignoring enabling and mediating factors (i.e., lack coherence) [15,16,17]. This difficulty with scientific explanation and argumentation is especially prevalent for phenomena that extend over large temporal and spatial scales, since students cannot directly experience the magnitude of the phenomena (e.g., climate, ecosystems) and the phenomena require the use of abstract models and textual descriptions to construct explanations and arguments [18].

We define a causal explanation as stating how one or more factors lead to the to-be-explained phenomenon through one or more intermediate processes, mechanisms or states [19,20]. When learning about a new topic in the science classroom, knowledge of potential explanations will generally come from reading texts or class instruction. This is not too different from how scientists learn about scientific phenomena by reading the work of other scientists [21,22,23]. We define scientific argumentation as the process of evaluating scientific explanations, weighing evidence and justifying conclusions. Students can provide evidence for an assertion that is both appropriate and sufficient. This process of scientific explanation is frequently instantiated in classroom settings as a Claim-Evidence-Reasoning framework [24]. Scientists generally gather evidence to support explanations from four sources: direct observations, empirical output from a model or simulation, experimental findings from scientific texts, and statements from experts or texts [22,23]. Students need to know the valid types of support in a given discipline and to practice using each of these sources [11,25,26]. Indeed, it is sometimes difficult to distinguish explanation and argumentation in scientific writings, but like Osborne and Patterson [20], we believe such a distinction is essential.

A key characteristic of our approach is placing students in the role of a scientist to develop an argument about the impact of hurricane disturbance on the El Yunque rainforest. Prior research has shown that having students take on roles related to the content discipline can have positive benefits for both motivation and learning (see [27] for a review). However, interventions designed to place students in specific roles have implications for the level of abstraction that is optimal for transfer. Son and Goldstone [28] found that taking a perspective that focused individuals on features of the particular context, which were unnecessary for understanding the primary concepts to be learned, decreased transfer. For example, in a study in which participants learned about signal detection theory, some were asked to take on the role of a doctor in order to explore signal detection theory in the context of a medical test. Learners who took on this role performed more poorly than those who did not take on a particular role. Likewise, contexts that are rich in detail can provide motivational benefits for students but could also focus student attention on seductive details that are less relevant for learning the primary concepts [29]. If the concrete situation only presents superficial similarity to the important concepts to be learned, transfer is hindered [30]. These studies present a dilemma. It is possible that although authentic experiences in science might promote involvement in science, they might also undermine the more general scientific principles that are intended to be learned.

While there is an abundance of literature that provides empirical support for learning principles related to role playing and using evidence to model complex systems, such as reading to learn, modeling, data analysis, explanation, and evaluating arguments, less work has been done on bringing these principles together to guide the development of units [31]. In this paper, we describe one attempt to bring these learning principles together to develop and study the Journey to El Yunque program, a middle school curriculum unit that places students in the role of a scientist and develops their ability to use modeling to construct explanations about the process of hurricane disturbance and recovery.

In order to test whether engaging students in the role of scientist can help students learn about the dynamics of complex systems like hurricane disturbance, we manipulated the role that students were asked to take while engaging with the Journey to El Yunque program. Students were randomly assigned to take on the role of a scientist or take on an explicit role of a student. Students assigned to the scientist role were prompted to think as a scientist and to focus on explaining how the species respond to a hurricane disturbance. Those who were assigned to the control condition took on the explicit role of a student and were prompted to think about learning specific content ideas as expressed by the middle school Illinois Learning Standards. It is typical for school management approaches to recommend that teachers provide students with explicit learning goals related to the standards. Students in each condition conducted the same set of exercises in Journey to El Yunque, but their purpose for doing so was based on the role they were assigned.

1.1. Journey to El Yunque

Journey to El Yunque (http://elyunque.net) is a web-based curriculum unit that aims to improve middle school students’ understanding of the dynamics of disturbance and recovery after severe hurricanes (for a detailed description of the program, see [18]). The program exposes students to the authentic research practices of ecologists in the Luquillo LTER. The research surrounding the recovery of species from these disturbances provides a rich example of basic ecosystem processes at work.

Journey to El Yunque engages students in the same problems that researchers in Luquillo are investigating. The program consists of five modules that support the overarching goal of investigating what will happen to the rainforest if severe hurricanes strike the rainforest more frequently, as suggested by climate models (see Figure 1). In the program, students learn about the historical patterns of hurricane frequency and damage in El Yunque, investigate what happened to the producers and consumers in El Yunque after Hurricane Hugo (1988), explore the impact of a hurricane on an entire food chain, and investigate how the cycles of water, carbon, and nitrogen resources are altered by hurricane disturbance.

The background readings in this study come from the module on consumers. Prior to engaging in modeling activities, students are presented with background readings about the life history of each species. The program introduces students to six consumer species that are representative of the types of hurricane responses researchers have found in El Yunque [32]. There are two decomposers (mushrooms and snails), two primary consumers (caterpillars and walkingsticks), and two secondary consumers (anole and coquí). As part of the investigation cycle (see Figure 2), students read about the life history of their assigned consumer, make a prediction about the population dynamics after Hurricane Hugo, and explore a model of population dynamics to explain what happened to their species after Hurricane Hugo.

When investigating population dynamics after a disturbance, scientists in El Yunque typically consider changes to five primary limiting factors for a given species: access to prey, avoidance of predators, direct mortality from the hurricane, and suitability to changes in environmental conditions, in particular drought conditions and the influx of forest debris from the hurricane. Each species has five background readings focused on providing the life history of each limiting factor. Each of the readings is around 500 words in length and written at a 5th grade reading level. Given the average reading fluency of a middle school student, it is expected that students would be able to complete all five readings in about 20 min. While the students are reading, they are provided with a notetaking template that provides boxes for students to take notes on what they learn about the life history of the species related to each of the limiting factors.

After completing the background readings, students are asked to predict the graph of the population of the species over the 5-year period following Hurricane Hugo. Figure 3 shows the process of making a prediction for the anole. Students are presented with a blank graph. Using the mouse, students click and drag within the graph to draw a blue line representing their prediction about the population levels of the anole in each month for 60 months. On their worksheet, students use their reading notes to provide a justification for their prediction. After writing their justification, they click on Test Prediction to see an orange graph of the interpolated population data that was collected by scientists after Hurricane Hugo. Students are given the opportunity to generate hypotheses about the factors that may explain the graph of the anole population.

In order to support students in exploring the factors that affect population dynamics, Zimmerman developed computational models for each of the species in Journey to El Yunque using the Stella modeling software. He started each model by identifying a basic ecological model in Hannon and Ruth [33] that had similar characteristics to the targeted species. He then synthesized the relevant LTER literature about the target species to develop estimates of the parameters related to the limiting factors. For example, the loss of canopy created competition for the ground dwelling A. gundlachi as the canopy dwelling A. stratulus were forced to live on the forest floor. As the canopy recovered, the A. stratulus were able to return to the canopy thus reducing competition. In addition, the increase in ground cover from the increased exposure to direct sunlight brought an increase in the availability of insects, which are a primary food source for the anole. Figure 4 shows screenshots of the process of modeling. The first image shows a model run using the default parameters. The blue model output line remains stable and does not provide an approximation of the population dynamics after a hurricane, represented by the orange line. In the second image, the student altered the canopy parameter to simulate the complete loss of canopy and the gradual recovery. In addition, the student increased the input of prey to the system. The model output provides an approximation of the same actual data that appeared in the prediction activity. Students are expected to use their notes from the background readings and the evidence from the model to develop a scientific argument about the factors that might explain the dynamics of the anole population after a hurricane.

1.2. Placing Students in the Role of Scientist

A key characteristic of the Journey to El Yunque program is placing students in the role of a scientist. There are a variety of programs that take a similar approach e.g., [27,34,35]. These experiences are made possible because scientists in federal agencies, academic settings, corporations, and nonprofit organizations have made a vast array of scientific resources available to K–12 educators. Across the network of 28 LTER sites, of which Luquillo is a part, each site receives funding to support students to engage in schoolyard research modeled after the research conducted at each LTER site (http://schoolyard.lternet.edu/). The assumption underlying all of these scientific experiences is that the process of engaging students as scientists is exciting for students and that exposing students to these disciplinary experiences will increase student involvement in tasks [36], interest in and identity with science [37,38] and lead to higher learning outcomes [39].

In the case of Journey to El Yunque, we used the cognitive apprenticeship framework, developed by Collins, Brown, and Newman [40], to guide the development of the program. The cognitive apprenticeship model has been used across a variety of disciplines (see [41] for a review). It uses traditional forms of apprenticeship as a starting point and applies those structures to fields of work that are primarily cognitive in nature, such as scientific inquiry. The model encompasses four primary components—content, method, sociology, and sequencing. These components address what content should be taught, the manner in which students should interact with the content, how the classroom should be organized, and how that content should be organized. The design of Journey to El Yunque addresses all four components of the cognitive apprenticeship model. At the heart of the cognitive apprenticeship model is a community of practice to whom the students are apprenticing. In the case of this project, the community of practice is comprised of the Luquillo ecologists who are studying the impact of hurricane disturbance on El Yunque. For this research, we address three of the four components of the framework: content, method, and sociology. Since this study focuses on one module within Journey to El Yunque, it does not involve the sequencing dimensions of cognitive apprenticeship. In this section, we discuss which parts of these three components were addressed in this research.

1.2.1. Cognitive Apprenticeship—Content

The cognitive apprenticeship framework differentiates content into four types of knowledge and strategies that comprise expertise in a subject area: domain knowledge, heuristic strategies, control strategies, and learning strategies. These types of content serve as the range of learning goals that a curriculum can have. In this study, we focus on domain and heuristic knowledge.

Domain Knowledge

Domain knowledge consists of the facts, concepts, and procedures explicitly identified as part of the subject area. In general, educational standards documents provide a good source for explicating the domain knowledge. For example, Journey to El Yunque addresses the major middle-school ecology domain knowledge standards that were recommended at the time by U.S. national standards (In the United States, each state determines the educational standards for their students. The national standards in the United States provide a national consensus from the scientific and educational community. Many states develop their state standards based on the national standards.) [42]. Ecology domain knowledge was measured as a learning outcome in the study.

Heuristic Strategies

Heuristic strategies describe the “rules of thumb” or problem-solving techniques used by experts in solving problems. These are developed through experience in the field and are usually tacit. There are five primary limiting factors that ecologists in El Yunque typically examine in relation to population dynamics after a hurricane disturbance—(a) access to prey, (a) avoidance of predators, (c) direct mortality from the hurricane, and environmental conditions related to (d) water and (e) debris. Students use these primary limiting factors to generate explanations of changes in population after the hurricane.

1.2.2. Cognitive Apprenticeship—Method

The cognitive apprenticeship model describes six general instructional methods that taken together comprise the cognitive apprenticeship approach: scaffolding, exploration, articulation, reflection, instructional modeling and coaching. The Journey to El Yunque program includes design features to support all six methods. In this study, we focus on scaffolding, exploration, and articulation.

Scaffolding

The cognitive apprenticeship model recommends that teachers pose complex problems just beyond students’ abilities, and then provide students with scaffolding supports so that students are successful in completing the complex problems [43,44]. Gallimore and Tharp [45] discuss cognitive structuring as one of the basic forms of scaffolding. Any assistance related to structuring a task, such as task instructions and worksheets, would be considered cognitive structuring. Providing cognitive structuring during extended problem-based learning experiences helps students maintain interest and keeps their focus on content goals rather than project management [46]. In this study, we provided cognitive structuring scaffolds to support students in learning about the effects of hurricane disturbance on population dynamics.

Exploration

The cognitive apprenticeship framework recommends that scaffolding be balanced with opportunities for exploration. There are two types of exploration that students engaged in during the study. First is free choice reading. Students were able to freely explore readings about the species in Luquillo. Second is scientific modeling. Students were able to explore the relationship between the heuristic factors and population dynamics.

Articulation

The cognitive apprenticeship framework states that it is important for students to articulate their understanding throughout the problem-solving process [47,48]. Student articulation supports learning since students need to reorganize their thoughts to present them to someone else [49]. In addition, when students make their thinking visible through articulation, teachers can provide students with feedback. Throughout the reading and modeling activities, student articulation of ideas was meant to help them learn from the experience as well as serve as assessment data.

1.2.3. Cognitive Apprenticeship—Sociology

The sociology dimension serves an important role in the framework by bringing together a cohesive perspective that underlies the collection of methods and sequencing. There are four characteristics in the sociology dimension: situated learning, intrinsic motivation, classroom community of practice, and exploit cooperation. In this study, we focused upon situated learning and intrinsic motivation. As will be discussed below, the study design necessitated that students complete the tasks individually. In normal classroom implementations, teachers take advantage of the classroom community to support cooperation and learning.

Situated Learning

The theoretical notion of situated learning suggests that students should engage in investigations that reflect the contexts in which they use their knowledge [50]. Journey to El Yunque engages students in investigations that are set in the context of El Yunque research, therefore, students’ participation reflects situated learning.

Intrinsic motivation

Intrinsic motivation occurs when students perform tasks because they are related to their own goals [51]. Journey to El Yunque is designed to present the phenomenon of disturbance as puzzling for students. In particular, the strategy of engaging students in prediction and providing them the results leads to natural curiosity when the results do not match the prediction [52].

2. Materials and Methods

2.1. Design

The study used a within-classroom, blocked randomized experimental design in which students within a class were randomly assigned to one of two levels of task role (either junior scientist or student). The students were blocked based on gender, race/ethnicity, ecology knowledge and interest prior to the random assignment. The study was conducted across six consecutive days during students’ science classes. The first and last days were reserved for collecting pre- and post-measures, leaving four days for the intervention itself. The dependent variables focused on in this article included students’ comprehension of the articles, their predictions of population changes, and their overall learning of ecology.

The manipulation of task role was designed to encourage students to take the role of a student (i.e., their typical role) or to take on the perspective of junior scientist. The role of student was designed to focus students on learning specific ecology concepts, consistent with a standards-based approach. The role of junior scientist was designed to focus students on the use of scientific practices to explain phenomena in the world. A focus on the use of scientific practices is a core tenet of the updated U.S. national standards known as the Next Generation Science Standards [53]. At each school, the study took place either in a computer lab or on laptops in the science classroom. The students worked individually on a computer with headphones. A researcher was present in the classroom during the intervention to monitor the implementation of the study. The manipulation was embedded into the online learning environment which made it possible to assign different roles to students in the same class and to control for teacher effects. The task instructions were provided in writing on the computer screen and they were narrated aloud. Since the students were wearing headphones, it was possible for students sitting next to each to be in different experimental conditions.

The instructions for each role were embedded in several places in order to strengthen the manipulation.

Table 1. Introductory instructions for each of the experimental conditions.

Student Role	Junior Scientist Role
What do you think affects the survival of animals in an ecosystem?	How do you think that hurricanes affect the coquí (or anole) in the El Yunque rainforest?
By learning about the factors that limit the size of a population of animals, you can make predictions about how environmental changes will affect the animal.	By studying what happens to particular species after a hurricane, ecologists can predict what may happen in future hurricanes.
Over the next two days, you will join your classmates in learning about how a hurricane called Hugo changed the environment in an example ecosystem and how those environmental changes affected a particular carnivore.	Over the next two days, you will join ecologists in the El Yunque tropical rainforest to investigate how a particular hurricane called Hugo changed the environment and how those environmental changes affected a species called the coquí (or anole).

shows the introductory instructions for each condition. The bolded words highlight the key differences between the conditions. In the student condition, the instructions focus participants’ attention on general concepts related to ecology. In contrast, the instructions for the junior scientist condition focus participants’ attention on the specific species (coquí or anole), specific environment (El Yunque), and specific changes (hurricane). Messages that were specific to each condition were also conveyed in terms of their purpose of reading, the task for the day, and what they could expect to gain from the task. The instructions were presented to students in written form. The instructions were also read aloud to the students. At the end of the set of two days, students were asked to describe what they did in order to check whether students adopted different roles. The responses were coded for their level of concreteness (associated with the scientist role) or abstractness (associated with the student role). Students in the scientist role described their activities using more specificity in the content of what they were learning, which provides evidence that participants adopted their respective roles.

The four days of instruction were organized such that students spent the first two days learning about the first species (i.e., coquí) and the second two days learning about the other species (i.e., anole). All students studied the same two species. They had access to five readings for each of the two species. While reading, the students took notes and summarized their notes. The online system tracked how long students spent on each reading. On the second day for a given species, the students were asked to summarize the key ideas from the prior day’s readings. They then made a prediction about the effects of hurricanes on the population of the targeted species and manipulated a model to use as evidence in an explanation of what happened to that species in the aftermath of the hurricane.

2.2. Participants

The sample included 476 seventh grade students from 8 urban and suburban elementary schools in the Chicago metropolitan area. Given the documented differences in general science assessment performance by race and gender in the United States [54], it was important for us to block on race and gender in the process of random assignment to ensure that the distribution of students by race and gender was equivalent across the two conditions. We, therefore, asked students to voluntarily report their race and gender. Of the 468 students who reported demographic information, 54% identified themselves as females. The sample included students who identified as Asian (1%), African American (13%), Caucasian (36%), and Hispanic (49%). Another 2% indicated that their race/ethnicity was different from the categories provided. Using a Chi-square statistic, there were no statistically significant differences in the distribution of students across conditions by race/ethnicity or gender. All parents of the students gave their informed consent and subjects gave their assent for inclusion before they participated in the study. Students who did not give assent or whose parents did not provide informed consent still engaged in the learning activities, but their data were not retained. The study was conducted in accordance with the Declaration of Helsinki, and the protocol was approved by the Ethics Committee of Northern Illinois University.

2.3. Measures

The focal outcome measures included students’ knowledge of ecology after the instruction, their comprehension of the readings, and the extent to which their predictions incorporated key ideas from the background readings. These measures were acquired at different times during the learning tasks, which made it possible to track students’ learning through the sequence of activities. We also measured ecology knowledge and reading fluency prior to the manipulation (i.e., pretest) in order to use as covariates.

Test of General Ecology Knowledge. A measure of general ecology knowledge was developed during a prior summative evaluation of the program [55]. This assessment is comprised of publicly released state test questions in ecology that mimic the kinds of performance expected of students on standards-based state exams. The assessment contained 14 multiple-choice questions and one constructed response task. The assessment was administered as a pretest and posttest. Students’ responses were scaled using Rasch modeling. The scale ranges from 0 to 25 points. Using latent variable modeling, the reliability was determined to be 0.87.

Percent of Articles Read. In order to control for the amount of content that students were exposed to, we estimated the percent of the total available readings that students read. During the reading activities, the online environment tracked how long students spent reading the background materials. We computed an estimate of the percent of each article that was read by multiplying a students’ reading fluency (words per minute) times the number of minutes spent reading each article divided by the number of words in each article. We averaged the percent read for all five articles across each of the two species. As a measure of initial reading fluency, we used two middle-school MAZE assessments. Each form takes around 3-min. MAZE assessments have a reliability of 0.80 and show correlations over 0.70 with longer reading test forms [56].

Reading Notes. While reading, students were encouraged to take notes using a notetaking template that contained rows for each of the limiting factors the students were reading about. Students could organize their notetaking by the writing about each limiting factor in the appropriate row. The reading notes were scored for the presence of key ideas from each of the readings and divided by the total number of key ideas possible. The Reading Notes score for each species was averaged to create an overall score.

Reading Summaries. After the students finished reading and taking notes on the first day, we asked them to summarize the main limiting factors that affect the population. Students could use their notes to create answers. At the beginning of the second day for each species, we asked students to again summarize the main limiting factors that affect the species. They could use their notes for their response. The summaries were scored for the presence of key ideas about each limiting factor and divided by the total number of key ideas possible. The Reading Summary Scores were averaged across the two scores for each species and across the two species.

Prediction Justifications. After the students drew their initial prediction about what will happen to each species after a hurricane (see Figure 3), they provided a justification of how the limiting factors influence the population changes over time. After the students tested their prediction, we asked them to develop hypotheses about how the limiting factors shape the population after a hurricane. The responses were scored for the extent to which the students’ explanations of the influence of each limiting factor was consistent with the directionality of the graph. The students’ prediction score and hypothesis score were averaged within a species and then averaged across species.

Modeling Explanations. After students completed their prediction, they engaged in modeling of the influence of the limiting factors on the population of each species (see Figure 4). We did not score the students’ final explanations from the modeling tasks. A review of the student responses revealed a floor effect in the quality of responses that would not have provided sufficient variation for statistical analysis. The quality of student responses to the modeling task will be discussed in the discussion section.

2.4. Data Preparation

Scoring of Student Responses. While reading, predicting, and modeling, students took notes and responded to academic prompts. We developed an outline of the key idea units related to each limiting factor as they are described in the background readings. We scored the prevalence of key ideas in the notes students took, summaries of the readings, and justification for their predictions. Two independent scorers coded for the presence or absence of idea units related to each limiting factor for each species. Coding discrepancies were identified and discussed with a third rater to reach consensus. The overall inter-rater reliability of the two independent ratings prior to discussion was 92% agreement and 0.82 using Kohen’s Kappa, which adjusts for chance agreement. We computed the percentage of limiting factors that were referenced and then averaged the percentages across the limiting factors to come up with an overall score for the notes, summaries, and predictions.

Missing Data. Studies in educational settings almost always have missing data. A frequent source of missing data is student absence on the day that assessments or surveys are administered. A straightforward, intuitive approach to analyzing datasets with missing data is to delete students with missing data and only analyze students from whom the researchers have a complete dataset [57]. However, this approach can be problematic if the source of missing data is correlated with outcome variables, as is often the case with absences and outcome measures. Instead, Baraldi & Enders [57] advocate for a more robust approach using multiple imputation. For missing data within a day that students were present, we employed a multiple imputation technique to estimate values for missing data on process and outcome variables. The analyses in the results section represent pooled outcomes based on the results of five versions of multiply imputed data.

Table 2. Descriptive statistics for the variables used in the regression models.

Variable	Mean (SD)	Scale (Range)
Pretest	13.21 (3.82)	Rasch Scale (0.00–25)
Posttest	13.48 (4.19)	Rasch Scale (3.44–25)
Percent Read	0.55 (0.18)	Percent (0.08–1.09)
Notes Score	0.26 (0.12)	Percent (0.03–0.62)
Summary Score	0.11 (0.07)	Percent (0.00–0.39)
Prediction Score	0.17 (0.14)	Percent (0.00–0.75)

provides the pooled descriptive statistics of the multiply imputed data for the variables that will be included in the regression models.

3. Results

In order to investigate the extent to which the role which students adopted affected the quality of their responses to the learning tasks and their knowledge of ecology, we developed a series of models to first test the increase in ecology from pretest to posttest and then test the experimental manipulation on the quality of responses to the academic prompts and the posttest. The average pretest score was 13.2 out of 25 and the average posttest score was 13.5 for an increase of three-tenths of a point. We used a one-tailed, paired t-test to determine that this increase was statistically significant (t (162) = 1.89, p = 0.03) with a small effect size of 0.07, adjusted for the correlation between the pretest and posttest.

In the next set of models presented in

Table 3. Regression results for four models examining the direct and indirect influence of student role on the quality of responses to academic prompts and posttest outcomes. The reported coefficients are unstandardized with standard errors in parentheses. Statistically significant coefficients are presented in bold.

Independent Variables	Posttest Model	Prediction Model	Summary Model	Notes Model
Constant	13.48 **	0.17 **	0.10 **	0.24 **
	(0.20)	(0.01)	(0.004)	(0.01)
Pretest	0.66 **	0.003	0.001	0.01 **
	(0.05)	(0.002)	(0.001)	(0.001)
Percent Read	2.26 *	0.10 *	0.007	0.24 **
	(0.95)	(0.04)	(0.02)	(0.03)
Scientist Role	0.06	0.005	0.01 *	0.03 **
	(0.27)	(0.01)	(0.007)	(0.01)
Notes Score	4.68 *	−0.07	0.21 **
	(1.80)	(0.08)	(0.04)
Summary Score	0.73	0.82 **
	(3.04)	(0.11)
Prediction Score	1.97
	(1.39)
R2	0.55	0.23	0.18	0.32

* p < 0.05; ** p < 0.01.

, we used multiple regression to investigate the extent to which a students’ role predicted the quality of students’ notes, reading summaries, and predictions as well as posttest outcomes. We also investigated the extent to which the quality of student responses to a task predicted the quality of responses to subsequent tasks. All four models controlled for prior ecology knowledge and content exposure, expressed as the estimation for the percent of content that students read in the background readings. All of the independent variables were centered around the overall average. Table 3 shows the results of these regression models. All four models were statistically significant. In the first model, the student role did not directly predict student posttest scores. After controlling for prior knowledge and exposure to content, the only factor that predicted posttest scores was the quality of student notes. In the second model, we examined the quality of students’ justification of their prediction as an outcome measure. The student role did not directly predict the quality of students’ justification of their prediction. After controlling for prior knowledge and exposure to content, the only factor that predicted quality of students’ justification of their prediction was the quality of their summary of the readings. In the third model, both the student role and quality of notetaking predicted the quality of reading summaries, after controlling for prior knowledge and exposure to content. Students who were assigned to the role of scientist had slightly higher quality reading summaries. In the fourth model, the student role predicted the quality of notetaking, after controlling for prior knowledge and exposure to content.

4. Discussion

The results indicate that taking on a scientist role did not have a direct correlation with the two primary outcome measures of this study—posttest scores and quality of justification of prediction. However, taking on a scientist role had an indirect effect on those outcomes through the direct effect of role on the quality of student notetaking and the quality of student summaries of the readings. Figure 5 shows a summary of these regression results. Over the course of a short 4-day intervention, the average performance on the posttest increased a small amount (0.3) over the average performance on the pretest. It is important to bear in mind that the assessment addresses content that is usually covered over a 4-week period in a typical middle school life science classroom. Therefore, even a small statistically significant effect from such as short intervention is an important outcome. In order to place the indirect effect of the scientist role on posttest outcomes in context, we estimated the indirect effect by multiplying the 0.03 increase in notetaking score due to scientist role by the 4.68 coefficient for the influence of notetaking on posttest performance. The combined effect of scientist role and quality of notetaking is 0.14, which is equivalent to 47% of the pretest to posttest increase in general ecology content. These results run counter to the results of prior research [28,29,30]. In those studies, students who took on a concrete role were less likely to transfer their understanding to a different context. These results suggest that students who take on the authentic role of a scientist, which is congruent with the task of explaining a complex phenomenon, were more likely to transfer their understanding to the abstract context of a standards-based assessment. These results are particularly noteworthy since the only manipulation was the manner in which the tasks were introduced. The students in each condition still engaged in the same set of activities. In reality, typical standards-based instruction does not involve investigations of a coherent phenomenon. Instead, it typically involves studies of disconnected phenomena. These results suggest that there may be an even greater disparity in outcomes between an authentic, coherent experience like Journey to El Yunque and the typical disconnected set of experiences in a standards-based approach.

An important part of the Journey to El Yunque experience is the use of models of population dynamics as evidence to explain the response of particular species to the changed environment after a hurricane disturbance. While the students were routinely able to set the starting conditions for the model parameters to achieve a model output that approximated the actual data, the quality of their use of these input parameters as evidence for an explanation was low overall. The literature on modeling and argumentation in education provides recommendations on potential future scaffolds that could benefit students in a future implementation of Journey to El Yunque. First, students need to be able to record evidence in a manner that will guide them to developing their explanation. In the case of Journey to El Yunque, students record their evidence from the readings in a table organized by limiting factors. However, this representation may not be sufficient for supporting students to see the connections between the factors. Alternative ways of representing evidence may be beneficial. For example, causal model diagrams have been used successfully in the past to organize evidence [58,59,60]. Second, more support needs to be provided around the development of the scientific practice of explanation and argumentation. For example, there have been a wide variety of instructional supports developed to support the evaluation of scientific arguments e.g., [9]. Third, evidence-based modeling and scientific argumentation are not tasks that can be successfully accomplished by following a recipe. Students need to develop a task model to understand the reason why they are engaged in a particular task and how that task will contribute to the primary goal of explanation e.g., [61,62]. The task model is important for helping students carry important information from one task to the next. It is important to present task instructions to activate appropriate schema [17,63,64]. Providing explicit support for the development of the task model through task model framing would be an important future scaffold. In future research, we intend to explore how these additional scaffolds can improve the quality of students’ final explanation using the Journey to El Yunque models as evidence.

5. Conclusions

Hurricane disturbances are a common occurrence along the Atlantic and Caribbean coasts and territories in the United States. The research articles in this volume represent the cutting edge for the field’s understanding of the response of ecosystems to such disturbances. These articles are crucial for the general public to understand how to manage natural and human systems in the face of potentially increased frequency of severe hurricanes. The Journey to El Yunque program represents one approach for using the scientific results of research on hurricane disturbance as a context for engaging students in authentic inquiry about the impact of hurricane disturbance on ecosystems. The cognitive apprenticeship model guided the development of the program to engage students in the authentic practices of a community of practice such as the Luquillo ecologists. A key component of the cognitive apprenticeship model is taking on the explicit role of a scientist while developing explanations of species responses to a hurricane disturbance. The results of our research suggest that taking on an authentic role has benefits for identifying important content ideas from background readings and transferring that knowledge to an assessment of general knowledge of ecology as well as applying that knowledge as justification for a prediction of how a species might respond to a hurricane disturbance. Future research will expand on these results to enhance the way in which the Journey to El Yunque program supports students in developing evidence-based explanations of the response of ecosystems to hurricane disturbances.