Building Competence in Science and Engineering

Abstract

Next Generation Science Standards science and engineering practices (NGSS S&E) are ways of eliciting reasoning and applying foundational ideas in science. Studies have revealed one major impediment to implementing the NGSS, namely, insufficient teacher preparation, which is a concern at all teaching levels. The present study examined a program grounded in research on how students learn science and engineering pedagogical content knowledge and strategies for incorporating NGSS S&E practices into instruction. The program provided guided teaching practice, content learning experiences in the physical sciences, engineering design tasks, and extended projects. Research questions included: To what extent did the Program increase teachers’ competence and confidence in science content, with emphasis on science and engineering practices? To what extent did the program increase teachers’ use of reformed teaching practices? This mixed-methods, quasi-experimental design examined teacher outcomes in the program for 24 months. The professional development (PD) findings revealed significant increases in teachers’ competence and confidence in integrating science and engineering practices in the classroom. These findings and their specificity contribute to current knowledge and can be utilized by districts in selecting PD to support teachers in preparing to implement the NGSS successfully.

1. Building Competence in Science and Engineering Content: Research to Inform Practice

The Next Generation Science Standards (NGSS) document attempts to provide educators and students nationwide with an internationally benchmarked education by articulating conceptual science performance expectations. Little exists in implementation strategies and national studies have already identified impediments to NGSS implementation, such as the lack of resources for effective science education, limited instructional time devoted to science, and insufficient teacher training [1]. In a recent national study on teacher readiness, most middle- and high-school teachers indicated they have no engineering training and are ill-prepared to effectively implement NGSS; engineering emerged as the content area of greatest need and created the greatest degree of anxiety. Teacher preparedness is a concern at all levels as the mandates of NGSS require conceptual and exploratory learning, which are not always employed in all science classrooms [2]. There is an urgency to identify the type of professional development (PD) that will prepare teachers to meet the challenges of the NGSS. It requires an investment of resources to develop the appropriate tools to support teachers [3]. We must align the resources spent on PD with the demands teachers will face with NGSS, and also conduct the necessary research required to learn from it to inform practice.

The present study examined a program that aimed to prepare middle-school teachers for NGSS by building competence and confidence in using science and engineering practices in the classroom. We investigated PD that would potentially meet the demands teachers will face during NGSS implementation. We propose to use the information we have learned from this study’s results to provide recommendations for teacher PD, as lead states begin the NGSS-adoption process.

1.1. Literature Review

Some teachers embrace an educational innovation with enthusiasm and incorporate it into their classroom teaching. Yet, others discard it and continue with their familiar teaching practices after only a few attempts [4], as all teachers are not amenable to innovation [5]. For instructors to persist in their efforts to implement new strategies, they need to have the expectation that they will succeed [5]. Individuals’ beliefs about their competence and outcomes expected of their actions serve to enhance interest in a specific area, and a strong self-efficacy helps individuals overcome setbacks and persist in the face of challenge [6]. Teachers’ low self-confidence and lack of competence in content become significant impediments to an innovation, such as the NGSS, as teachers will have to contend with not having the necessary equipment, materials, or training for successful implementation [1].

1.2. The Development of NGSS

NGSS is a set of science standards. Science and its fellow technology, engineering, and mathematics (STEM) disciplines are, and have been, for at least ten years, the focus of concern and reform efforts from educators nationwide [7,8]. The reasons for this are diverse. The concern that the United States is lagging behind other nations in STEM areas, and that this gap could potentially emerge as economic disaster in the future [8], has resulted in an influx of federal funding for STEM education and research, and the encouragement of institutions to pursue this research [7].

In the summer of 2011, a writing team of 41 educators worked on the first draft of the NGSS. On 9 April 2013, the finalized NGSS document was released. NGSS describe what all students should know and be able to do by the time they graduate from high school. NGSS are based on learning progressions of core ideas in the discipline, concepts that cut across disciplines and practices that will allow students to use their disciplinary knowledge in thoughtful ways. A difference from earlier 1996 standards from the National Research Council, NGSS Science and Engineering Practices are characterized as ways of identifying the reasoning behind, discourse about, and application of the core ideas in science [9].

The specific eight science and engineering practices outlined in the NGSS are: asking questions and defining problems, developing and using models, planning and carrying out investigations, analyzing and interpreting data, using mathematics and computational thinking, constructing explanations and designing solutions, engaging in argument from evidence, and obtaining, evaluating, and communicating information (2013). The process of creating the NGSS was driven by 26 lead states that contributed resources and provided support. They are expected to be trailblazers in the adoption of the NGSS. However, the choice to implement curricula and the form these curricula will take is ultimately at the discretion of individual states.

1.3. NGSS, Models, and Modeling Instruction

Recent research contributes to current understanding of how students develop and use models in middle- and high-school classrooms. Current modeling research has focused on argumentation in science education [10,11], model-based inquiry [12], software scaffolds supporting modeling practices [13] constructing and revising models [14], and integrating conscious and intuitive knowledge [15].

Educators often discuss the important role models play in science education [16]. Scientific disciplines are guided in their inquiries by models that scientists use to create explanations for data and to further investigate nature. The design, use, assessment, and revision of models and related explanations play a primary role in scientific inquiry and should be a prominent feature of students’ science education [12].

Researchers investigating modeling nationally and internationally have significantly influenced the conceptualization of modeling articulated in NGSS [13,14]. They have advocated for the role of models and modeling in school science, and also argue that modeling is a core practice in science and a central part of scientific literacy [13]. Scientific modeling includes the elements of the practice (constructing, using, evaluating, and revising scientific models) and the knowledge that guides and motivates the practice such as understanding the purpose of models [13].

NGSS [17] employs the use of core-science and engineering practices (identified above), which are at the foundation of modeling instruction, an evidence-based pedagogy for science education that was developed in the 1980s. Modeling instruction integrates a student-centered teaching method with a model-centered curriculum [18,19]. It applies structured-inquiry techniques to teaching fundamental skills in mathematical modeling, proportional reasoning, and data analysis, which contribute to critical thinking, including the ability to formulate hypotheses and evaluate them with rational argument and evidence.

Modeling pedagogy has three elements: the models, the modeling cycle, and classroom discourse management [18,19]. An understanding of these elements is the pedagogical content knowledge [20] needed for successful classroom adoption and implementation. A model is a representation of structure—a conceptual representation of a real thing [18,19]. The models around which learning is centered in modeling are basic relationships among quantities that form the content core of a discipline and these models are developed by students into tools for making sense of physical reality—for making predictions.

Modeling has been defined as an activity. With its foundation in the modeling cycle [18,19] is a three phase process: model construction, which takes place in the context of a paradigm lab that discovers a link between two physical quantities at the beginning of each instructional unit; model validation, in which students refine the basic model they have constructed by testing it in disparate initial conditions; and model deployment, in which students use the model to solve problems from diverse contexts.

Teachers learn modeling instruction by participating in a modeling workshop—an intensive, three-week 90-h immersion experience. Teachers are engaged in laboratory investigations and activities, creating experiments, collecting, analyzing, interpreting data, and engaging in classroom discourse to achieve collective sense-making. It is by active participation in the discourse that characterizes the modeling learning that teachers can become effective managers of modeling discourse in their classrooms. modeling instruction began in college and high school physics and has expanded across the science disciplines into chemistry, biology, physical science and middle school science.

1.4. Study Overview

This study examines in-service teachers’ outcomes in a program that aimed to increase teachers’ content knowledge (cognitive skill) and confidence (self-efficacy) in the use of science and engineering practices (SEP) in the classroom. Three high-needs districts and a state university formed a partnership and proposed to enhance the quality of science instruction for middle-school teachers. The partnership was cognizant of the fact that science instructors lacked adequate preparation in areas associated with science and engineering practices and proposed, through the use of modeling instruction, to increase teacher science-content knowledge of energy and matter. The team planned a formal needs assessment in 2012 to collect data on concrete deficiencies.

1.5. Needs Assessment Identifies Professional Development Focus

A survey was administered to partnership teachers to assess their knowledge related to the grant’s content focus. An online survey was administered in summer, 2012 to 250 science and math instructors in three districts to assess their science content knowledge with emphasis on SEP. The survey yielded a 68% response rate (n = 171). The assessment revealed that teachers in grades six through eight had the following limitations: lack of content knowledge and confidence in their ability to teach science content, with emphasis on scientific and engineering practices; minimal knowledge on integrating science and engineering content; and limited knowledge of how to design and deliver science and engineering activities. Teachers had no college coursework in the structure of matter, matter and energy flow in organisms, conservation and transfer of energy, the relationships of energy and forces, and energy in everyday life. This corresponded to no physics, chemistry or biology courses that could be considered fundamental to NGSS standards and practices.

The data were combined with research that highlights the correlation between teachers’ science-content knowledge and student achievement [20] and SEP and student achievement [21,22], particularly in high poverty areas, which were project cornerstones. Teacher deficiencies were a concern in these districts and administrators wanted to identify PD to best train teachers to meet NGSS challenges.

Based on the needs assessment, partners identified three goals targeting middle-school teachers: increase teachers’ physical science content knowledge in energy and matter; increase teachers’ confidence in incorporating NGSS SEP into their instruction; and increase teachers’ use of reformed teaching practices.

1.6. Professional Development Model

The PD model was designed to move the partnership toward accomplishing these goals, which started 27 October 2012 and was completed by Summer of 2014. PD included 236 h with three six-hour Saturday PD sessions during each academic year and two three-week summer institutes in 2013 and 2014. Teachers participated in 227 h of PD overall. The Partnership for Success (PAS) program engaged 27 teachers in grades six through eight. The practices of engineering design were interrelated with scientific practices to create the context of the learning environment. PAS provided teachers with a PD program grounded in research on how students learn science and engineering pedagogical content knowledge and strategies for incorporating NGSS S into instruction. To do so, PD provided guided teaching practice, content learning experiences in the physical sciences, and engineering design tasks and extended projects. modeling instruction was also integrated into the PD model.

PAS activities were selected to provide teachers with content preparation in a core scientific concept—energy—and to provide explicit practice and experience in using SEP. Teachers worked through activities and sense-making, confronting misconceptions, and learning to argue from evidence just as their students will be expected to do. They engaged in classroom discourse that reflected the type of discourse they would be expected to mediate in their own classrooms. Teachers went from “student mode,” in whole-group discussions, to “teacher mode” deliberations, in which they explored the instructional implications and identified the theoretical underpinnings and disciplinary links to what they were learning. As PAS participants were middle-school teachers, faculty often helped them appreciate both the horizontal continuum of the energy concept across disciplines, and the vertical trajectory of conceptual development across grade levels, which resulted in a coherent model of energy storage and transfer.

Both summers focused on crosscutting models of energy and the structure of matter. The first summer institute delved into macroscopic models of energy and the structure of matter and focused on developing SEP in the context of motion, forces, and mechanical and gravitational energy. Time was given to the development of operational definitions the use of scientific language and management of classroom discourse. Saturday sessions, after this first Institute, gave teachers an opportunity to share their successes and challenges as they gained confidence in the use of new teaching strategies. PAS engaged them in additional engineering design activities to help them transition from macroscopic to microscopic models of energy and the structure of matter to help frame the content for the second summer. The second summer institute focused on chemistry, ecology, and the earth sciences. More time was spent understanding the structure of matter and the role of energy and systems in these content areas.

The mechanism used to deliver instruction was the Modeling Method of Instruction. The design of instruction followed the modeling cycle, thus, participants engaged in whole group pre-laboratory discussions and small-group laboratory activities, followed by analysis and synthesis of results. These results were shared via whiteboard meetings—whole group discussion and sense-making around the relationships explored in the laboratory activities. This model construction activity was then followed by a series of model elaboration and deployment activities. Activities done in small groups were followed by whole group discussion to allow participants to place what they learned in the context of their own teaching assignment.

Iterative engineering design activities were used as a capstone activity in the summer institute. Teachers were involved in a guided curriculum design activity in which they worked together in groups. They used the modeling cycle to design or redesign a curriculum unit of their choosing and incorporated activities that utilized SEP. Units designed by participants were also made available electronically to other participants so they could integrate them in their own classrooms with the goal of giving feedback to the unit designer.

1.7. Partner Roles

District personnel were responsible for basic communication, facilities and district credit, as well as overall project management. University personnel were responsible for initial planning and delivery of professional development. The researcher was responsible for collection and analysis of aggregated data, quarterly reports (formative assessment), and dissemination of findings in a summative report. While each partner in the project had these specific roles, PD curriculum, leadership in PD sessions, data collection and analysis, and production of project deliverables were accomplished in full cooperation and participation.

2. Method

2.1. Research Design

This study employed a mixed-methods approach and thus the investigator collected, analyzed, and drew inferences from both quantitative and qualitative data in a single study. The investigator held the assumption that the combination of quantitative and qualitative approaches provides greater understanding of the research problem than either approach alone [23]. The researcher used a quasi-experimental, matched-comparison group design, using multiple methods and statistical tests to measure progress toward meeting the established outcomes. This model provides a good alternative, as a randomized controlled trial was not feasible. The research design included both quantitative and qualitative methods and employed analysis of variance (ANOVA) to determine differences between groups and qualitative analysis to code, categorize, and analyze teacher comments. The assumptions of homogeneity of variance, normality, and independence were tested and met.

Research questions that guided the study included:

• To what extent did the program increase middle school teachers’ science content knowledge, with emphasis on science and engineering practices?
• To what extent did the program increase middle school teachers’ self-confidence in teaching science content, with emphasis on science and engineering practices?
• To what extent did the program increase teachers’ use of reformed teaching practices?

2.2. Participants and Recruitment

District and school administrators developed strategies for the PAS program for recruitment and retention of teachers to maintain samples size of both groups (experimental and comparison groups). In year one, the project team recruited over 30 participating teachers from the three districts in the partnership and a comparison group equivalent on selected demographic characteristics (i.e., time teaching, grade band, and area of specialization). Matched comparison was based on number of years teaching (average of 13 years), grade level taught (i.e., middle school), and area of specialization (i.e., science). A t test analysis revealed no statistical difference between the groups based on the number of years teaching and grade level.

2.3. Quantitative Data

Data Sources. In order to answer the first research question (To what extent did the Program increase middle school teachers’ science content knowledge, with emphasis on science and engineering practices?) the investigator employed the Diagnostic Test for Mathematics and Science (DTAMS) as a pre–post measure, and it was administered to both groups. In addition, the Basic Energy Concept Inventory (BECI) served as a pre–post measure for the PAS group, as it was closely aligned to the intervention. The BECI, a 25-selected-response-item instrument, is used to capture commonly held misconceptions regarding energy.

To answer the second question (To what extent did the Program increase middle-school teachers’ confidence in science content, with emphasis on science and engineering practices?) a self-efficacy instrument, the Science Teaching Efficacy Beliefs Instrument or STEBI [24] was administered to both groups as a pre–post measure. The STEBI, with 24-items, assessed teachers’ confidence in science and engineering practices using a 5-point Likert scale (5 = strongly agree to 1 = strongly disagree).

To answer the third research question (To what extent did the Program increase teachers’ use of reformed teaching practices?) an observational tool, the Reformed Teacher Observation Protocol (RTOP, [25]) was used as a pre–post measure for both groups. The RTOP, a 25-item observational instrument, was designed to measure reformed teaching as defined by research in mathematics and science and national standards. All pre assessments were administered to both groups before the start of the intervention and the post-tests were administered to both groups after the intervention ended for the experimental group.

As another data source, an online survey was administered after each Saturday PD session and during the summer institutes. Teachers rated PAS in terms of its effectiveness in providing guidance and concrete examples to enable progress in the eight NGSS SEP. The survey used a five-point rating scale of effectiveness (5 = highly effective, 4 = effective, 3 = average, 2 = below average, and 1 = not effective).

Data Analysis. Quantitative data were analyzed two ways. Analysis of variance (ANOVA) determined differences between groups. Secondly, a paired samples t test was used to examine difference within groups to determine program efficacy. All analyses include 27 PAS and 29 comparison teachers. The assumptions of homogeneity of variance, normality, and independence were tested and met.

2.4. Qualitative Data

Teachers were given the opportunity to comment (on surveys) on the PAS program after each Saturday PD and during the summer institutes. They were asked to rate the program on specific criteria such as providing training on NGSS SEP but could also express views about PAS impact; as a result, themes associated with NGSS competence, confidence, and implementation in the classroom emerged. The researcher used the constant comparative method [26] as a conceptualizing method on the first level of abstraction. The initial phase involved conceptualizing all the incidents in the data. The researcher compared data and continually modified and sharpened the growing theory at the same time. Notes were compared to find differences and consistencies between codes, which helped reveal categories. Data were analyzed using a three-step process: data reduction, data display, and conclusion drawing and verification [27,28]. Data reduction helped to sort, focus, and condense excerpts, which helped organize the data to develop conclusions. Data display enabled review of the reduced data so that conclusions could be drawn. Teachers’ excerpts formed the basis for identifying categories, themes, and assertions.

2.5. Quantitative Results

To answer the first research question (To what extent did the Program increase teachers’ science content knowledge, with emphasis on science and engineering practices?) the DTAMS was employed. The DTAMS pre–post test was used to measure knowledge of core content concepts. Means are arrayed for the PAS and comparison groups in terms of pre–post test within each group (paired t tests) and post-test statistical comparisons between the groups (ANOVA) to determine differences. PAS significantly outscored the comparison group on the DTAMS content knowledge items and the difference was significant (p = 0.03).

2.6. DTAMS Results

Between Group Difference. This evaluation examined differences between groups on the DTAMS’ overall-content-knowledge mean score. The overall possible score for the content knowledge was 35. The PAS group (27 teachers) final knowledge mean score was 18.22 (SD = 5.4) and the comparison group (29 teachers) final knowledge mean was 15.31 (SD = 4.78), which revealed a significant difference favoring the PAS group (p = 0.03) as shown in

Table 1. DTAMS: Group Comparison Post-Means.

	PAS Post-DTAMS	Comparison Post DTAMS
	Mean (SD)	Mean (SD)
DTAMS Means	18.22 (5.40)	15.31 (4.78) *

* p < 0.05.

below.

Within Group Differences. For PAS, there was a modest gain from the pre- (M = 16.62, SD = 5.26) to post-DTAMS knowledge mean (M = 18.22, SD = 5.40) and the gain was not significant (p = 0.27). There was no significant difference for the comparison group from the pre-DTAMS knowledge mean (M = 16.20, SD = 5.0) to post-DTAMS mean (M = 15.31, SD = 4.78) (p = 0.51), as seen in

Table 2. DTAMS: Within Group Comparison Pre-Post DTAMS Means.

	Pre-DTAMS	Post DTAMS
	Mean (SD)	Mean (SD)
PAS Teachers	16.62 (5.26)	18.22 (5.40)
Comparison Teachers	16.20 (5.00)	15.31 (4.78)

below. There was no significant difference between the PAS and the comparison group on the pre-DTAMS knowledge mean score (p = 0.75).

2.7. Basic Energy Concept Inventory (BECI) Results

To answer the first research question, the Basic Energy Concept Inventory (BECI), an instrument to capture commonly held misconceptions regarding energy, was administered to the PAS group. The PAS program focused on energy content, and the BECI instrument was considered well aligned to the PAS intervention. Initially, teachers did not understand the structure of matter or potential energy, the structure of matter well enough to account for both warmth and coldness in terms of thermal energy, and did not account for energy that had dissipated into the environment. For the PAS administration of the pre- and a post-BECI, post-scores were higher (M = 15.7, SD = 3.03) than on the pre-BECI (M = 8.7, SD = 3.79) and the difference was significant (p < 0.001). Teachers mastered 63% of energy content (see

Table 3. BECI: PAS Group BECI Pre-Post Difference.

	PAS Pre-Test	PAS Post-Test
	Mean (SD)	Mean (SD)
BECI Means	8.70 (3.79)	15.7 (3.03) *

* p < 0.0001.

).

The overarching themes of the PAS workshops were energy and the structure of matter. Special care was taken, during the course of these workshops, to develop representational tools and practices that allowed the teachers to develop robust models of microscopic and macroscopic models of both of these core concepts, and teachers were encouraged to employ these tools across disciplines and grade levels. BECI increases revealed that PAS teachers left the program with a more robust microscopic model of energy transfer and storage.

2.8. Self-Efficacy Instrument Results

To answer the second question (To what extent did the Program increase middle school teachers’ confidence in science content, with emphasis on science and engineering practices?) a science teaching self-efficacy instrument was administered. Teachers had increased confidence in science and engineering practices and the ability to design and deliver engineering activities. In addition, teachers were less anxious about their engineering skills.

Within-Group Differences. Post-survey means revealed PAS teachers were more confident in their ability to design and deliver science and engineering activities and integrate SEP into their classroom. Data revealed teachers were less anxious about their engineering skills and more confident in the following areas: having the ability to answer students’ engineering questions (survey item 18); using SEP to enable integration into classroom instruction (survey item 19); and designing and delivering engineering activities (item 23). PAS teachers were more confident in designing and delivering science content with scientific and engineering practices (SEP) (p-value < 0.001), as seen in the Appendix A.

The increase in self-confidence for PAS teachers was also evident in the everyday actions of the teachers in the second year Saturday sessions and during the final summer institute regarding the depth and the types of questions they asked. There was an increase in teacher directed inquiry and analysis at the end of an activity. In addition, teachers were able to think deeply about what they were doing (during and after experiments) and about student thinking and learning in the context of engineering content and practices. In many instances, teachers were able to make suggestion on how to make the activities better.

Between Group Differences. There were differences between groups on the post self-efficacy survey relating to items on SEP and engineering skills, favoring PAS. Post-survey means showed PAS teachers were more confident (p < 0.01) in their ability to answer students’ engineering questions (M = 3.66, SD = 0.62) than the control (M = 3.03, SD = 1.14). PAS teachers were more confident (p < 0.001) in using SEP to enable integration into classroom instruction (M = 4.03, SD = 0.75) than the control (M = 3.24, SD = 1.02) and were more confident (p < 0.001) in designing and delivering science content with SEP (M = 4.11, SD = 0.75) than the comparison group (M = 3.00, SD = 1.30) as seen in

Table 4. Self-efficacy instrument post means: PAS and comparison group differences ¹.

Scientific and Engineering Practices Self-Efficacy Survey Item		PAS Post		Comparison Post
		Mean	SD	Mean	SD
18	I am typically able to answer students’ engineering questions	3.66	0.62	3.10	1.17 *
19	I am confident in using Scientific and Engineering Practices to enable integration into classroom instruction	4.03	0.75	3.24	1.02 **
20	I am typically anxious about my engineering skills	2.66	1.17	3.00	1.06
21	I currently have the necessary skills to integrate Scientific and Engineering Practices	4.00	0.73	3.06	1.25 **
22	I am currently able to design and deliver science activities	4.30	0.60	3.82	1.28
23	I am currently able to design and deliver engineering activities	3.77	0.89	3.17	1.07 *
24	I am currently able to design and deliver science content with science & engineering practices	4.11	0.75	3.00	1.30 **
Degree of Agreement (5 = strongly agree to 1 = strongly disagree)		n = 27		n = 29

* p-value = 0.01, ** p-value ≤ 0.001, ¹ Independent-samples t-test.

.

PAS PD in NGSS Scientific and Engineering Practices: Survey Results. To determine the extent to which PAS provided strategies and guidance to increase competence and confidence in NGSS SEP, an online survey was administered. Overall, the majority of teachers felt the program was highly Effective in providing guidance and examples in the following SEP: developing and using models (81% highly effective), engaging in argument from evidence (70%), asking questions and defining problems (67%), analyzing and interpreting data (67%), obtaining, evaluating and communicating information (67%), and planning and carrying out investigations (58%) as seen in

Table 5. PAS Provided effective training in NGSS science and engineering practices.

PAS Provided Guidance and Examples in	Teacher Rank of PAS Guidance and Providing Examples (%)
	Highly Effective	Effective	Average	Below Average	Not Effective
Asking questions/defining problems	67%	30%	3%
Developing and using models	81%	19%
Planning investigations	58%	42%
Analyzing/interpreting data	67%	30%	3%
Using math/computational thinking	19%	78%	3%
Construct explanations/design solutions	33%	67%
Engaging in argument from evidence	70%	30%
Obtaining, communicating information	67%	33%

below.

2.9. Reformed Teacher Observation Protocol (RTOP)

To answer the third research question (To what extent does the project increase participating teachers’ use of reformed teaching practices?) the RTOP was employed as the classroom observational tool. The post-RTOP data and significant differences (from pre- to post-RTOP) for the PAS group revealed integration of practices such as developing and using models, engaging in argument from evidence, asking questions and defining problems and analyzing data. The PAS group outscored the comparison group on the post-RTOP revealing growth over time, and also highlighted PAS teachers’ increased integration of NGSS S&E practices.

Within-Group Differences. The PAS group post-RTOP score (M = 73.44, SD = 14.32) revealed a 22-point gain from the pre RTOP score (M = 51.41, SD = 22.43) and the gain was significant (p < 0.001) as seen in

Table 6. RTOP: Within group comparison pre–post means.

	Pre-RTOP	Post RTOP
	Mean (SD)	Mean (SD)
PAS Teachers	51.41 (22.43)	73.44 (14.32) **
Comparison Teachers	57.38 (20.51)	56.10 (19.04)

** p < 0.001.

below. There was no significant difference for the Comparison group regarding the pre–post RTOP scores (p = 0.47). No significant difference was evident between the PAS and the Comparison groups on the pre-RTOP scores (p = 0.30).

Between Group Differences. There was a significant difference between groups on the post RTOP scores (p = 0.0003) favoring PAS as seen in Table 3. The mean for the PAS group was 73.44 (SD = 14.32) and the mean for the Comparison was 56.10 (SD = 19.04). There was a Cohen’s d ‘large’ effect size of 0.80, or one standard deviation difference between groups.

RTOP data revealed PAS teachers’ use of reformed teaching practices. Specifically, growth was observed in teachers’ integration of NGSS SEP as students were seen developing and using models, engaging in argument from evidence, asking questions and defining problems, and analyzing and interpreting data. PAS classrooms, during post-RTOP observations, frequently involved students developing explanations and employing critique and evaluation (promoting argumentation from evidence). Increases also emerged as students incorporated (RTOP, 9) elements of abstraction with symbolic representations and theory building. Also, (RTOP, 11) students used a variety of means and developed and used models (e.g., used models, drawings, graphs, and concrete materials to represent phenomena); asked questions and defined problems; made more predictions, estimations, and hypotheses, and devised means for testing them (RTOP, 12); communicated information and ideas to other using a variety of means (RTOP, 16) and were analytical and reflective in their learning (RTOP, 14).

3. Qualitative Results

NGSS Implementation in the Classroom

Comments indicated that when teachers started the program, they had “increased anxiety over NGSS requiring a new set of skills not found in their education” yet PAS “provided a new skill set” (C. Mason, survey response, June, 2014). Initial teacher comments (commenters’ names have been replaced by pseudonyms in this section) revealed anxiety associated with insufficient teacher training for successful NGSS implementation, impediments, which were similar to those identified in a prior study [1]. In addition to low self-confidence, the comments focused on lack of engineering content knowledge and the inability to design and deliver integrated science and engineering activities. Qualitative comments captured at the end of the program indicated teachers were more frequently embedding NGSS SEP in the classroom, as reflected in the following excerpt, “Before the PAS program, I did not integrate any engineering practices. Now, I incorporate engineering practices almost on a weekly basis. This has been an easy transition especially with the models, resources, and strategies provided by the program” (A. Sabato, survey response, June, 2014). Another echoed this sentiment, “Now I have many strategies to integrate science and engineering practices and I use models, graphs, and other elements of abstraction in my teaching (M. Rodriguez, survey response, June 2014). Another indicated, “I feel much more confident in using science and engineering practices and asking students to use models, analyze data, and be reflective in their learning” (B. Masters, survey response, June 2014). Teachers more often used “scientific writing, integrating claim, evidence, and reasoning into classroom projects” (A. Prosser, survey response, 27 June 2014) and required students to “ask questions, analyze and interpret data, and communicate information during class time” (A. Monroe, survey response, 27 June 2014). The majority of teachers use modeling instruction and ask students to “define problems, build and use models, collect and analyze data, and communicate information to classmates.”

Teachers provided details on how they were integrating NGSS SEP into the classroom. One noted, “I use models and modeling instruction in my classroom and allow students to build their own conceptual models. I have changed my expectations for my students’ lab reports and we focus on claims, evidence and reasoning” (T. Walker, survey response, 28 June 2014). Others have re-engineered the way they structure classes as a strategic process “Students have been involved in more experiments, and will be engaged in more open experimentation that will foster greater analysis and communication individually and between students” (A. Verde, survey response, June 2014). “PAS content and pedagogy have increased student critical thinking, confidence, and engagement” (C. Lorenzo, interview response, June 2015). Teachers are “promoting more student critical thinking and allow students to guide learning.” They have devoted more time to planning the inquiry line of questioning.

4. Discussion

National research suggests that there are several impediments to overcome for the adoption of NGSS and one barrier is inadequate teacher preparation [1]. Similarly, the PAS partnership recognized local teachers lacked adequate training in areas associated with science and engineering practices and proposed, through modeling-based learning experiences, to increase content knowledge in energy and matter. This study examined PAS efficacy and found that the PD potentially meets the demands middle-school teachers will face during NGSS implementation. PAS PD, which provided integrated science and engineering content, guided teaching practice, content learning experiences in the physical sciences, and engineering design tasks, increased teachers’ competence and confidence in using NGSS SEP. It is likely these results will help inform other educators and researchers as states and district begin the NGSS adoption process.

4.1. Program Builds Competence and Confidence in the Use of NGSS

Although current science teaching practices often emphasize the memorization of facts, the PAS team adopted the NGSS focus, which emphasizes the active construction of conceptual knowledge by “doing science” through science and engineering practices. Regarding content, the BECI attempted to capture gains in energy and the structure of matter, which were the overarching themes in PAS. PAS required the development and use of representational tools and practices that allowed teachers to develop microscopic and macroscopic models of these core concepts. Teachers were involved in deep, rich discussions to highlight naïve beliefs and replace them with coherent conceptual models. Using models as thinking tools in diverse problem contexts, teachers used them to frame their thinking in responding to BECI questions. Moreover, BECI increases revealed that PAS teachers left the program with a more robust microscopic model of energy transfer and storage.

PAS data revealed increased self-confidence as teachers were less anxious about their engineering skills and more confident in their ability to answer students’ engineering questions; were more confident in using scientific and engineering practices to enable integration into classroom instruction and in designing and delivering engineering activities. They also indicated they were better able to design, deliver, and integrate science content with SEP.

4.2. Program Teachers Use NGSS SEP in Classroom Instruction

During observations, PAS teachers were guided in their inquiries by models, which often created explanations for data. In addition, teachers’ design, use, assessment and refinement of models played a primary role in the program, supporting NGSS and prior research emphasizing the importance of models in science education [11,14,16]. Consistent with international and national research informing the NGSS, PAS made modeling a core practice in the PD [13,14]. PAS was also found effective integrating argumentation [10]; using model-based inquiry [12]; planning and carrying out investigations; constructing explanations and designing solutions; and obtaining, evaluating, and communicating information, thereby supporting NGSS [17].

5. Limitations

PAS program teachers were recruited by the three districts and were paid for their participation. Teachers self-selected into the study, and these teachers persisted in a PD program for 24 months. Since the teachers were not randomly selected, there are limitations to the study; teachers were motivated to learn new skills and persist in the program. These results may be generalizable to the population of teachers who would enroll in PD for academic growth and for those who are dedicated to improving their NGSS scientific engineering and practices.

6. Implications

PAS teachers were more competent and confident in using science content that emphasized science and engineering practices in the classroom. What implications are relevant to states adopting NGSS and districts serving middle school students? These findings and their specificity contribute to current knowledge and can be utilized by districts in selecting PD to support teachers in preparing to implement NGSS successfully. Teachers trained in the methods above and those who employ modeling instruction offer a profile of teachers who could become leaders in NGSS teacher professional development. They could also be employed as peer mentors in schools and districts to facilitate the transition to and implementation of the new standards. As NGSS move towards national adoption, it is crucial that educational leaders understand what these standards and the changes will mean for the teachers who implement them. To this end, our study examined a program that aimed to prepare middle-school teachers for NGSS by building competence and confidence in using science and engineering practices. This PD, incorporating modeling instruction, potentially meets the demands teachers will face during NGSS implementation. Our findings support prior research on elementary teachers, adding to the literature base on NGSS implementation. Consistent with results from Trygstad [1], our findings call for targeted professional development as teachers are concerned about receiving training in engineering content. The information we have learned from this study will help educators align the resources spent on PD with the demands teachers will face in a NGSS classroom.