Variations among Next Generation Science Standards and Other NRC Framework-Based Science Standards: Differences in Layout, Typography, and What Teachers Notice

Abstract

Science standards across 44 states in the United States are often assumed to be equivalent because they are all based on the National Research Council’s (NRC) Framework for K-12 Science Education. Twenty of those states adopted the Next Generation Science Standards (NGSS), which is based on the NRC Framework, and the 24 other states developed their own NRC Framework-based science standards. In this article, two related studies are described that focused on assessing this homogeneity assumption. In the first study, a comparative document analysis categorized the variety of ways performance expectations are presented. Analysis also focused on relative placement of information related to performance expectations and components of three-dimensional learning. To assess how variations affect teacher noticing, in the second study nearly 300 elementary school teachers viewed, in random order, seemingly similar fourth-grade standards from three states. Comparisons focused on teachers’ noticing of student objectives, elements that stood out, and teachers’ rationales regarding their noticing. Though both studies underscored that all NRC Framework-based science standards do integrate NRC Framework tenets, findings counter the assumption that NRC Framework-based science standards are necessarily equivalent to each other or to NGSS.

1. Introduction

In a discussion regarding the United States’ Next Generation Science Standards (NGSS), Sadler and Brown noted 19 states at the time had adopted NGSS and that NGSS served “as a framework (to varying degrees) for many others” [1] (p. 903). In this paper, the focus is on the parenthetical reference, “to varying degrees.” NGSS developers envisioned NGSS would be widely adopted across the United States, much like the preceding Common Core State Standards (CCSS) for Mathematics and English Language Arts [2]. However, NGSS was neither quickly nor commonly adopted. Negative reactions to CCSS and shifting political winds likely contributed to lackluster NGSS adoption [3,4]. Currently, only 20 U.S. states have adopted NGSS. Despite this, NGSS has been referenced as the United States’ “national science standards” [5] (p. 737).

There is an inclination to consider NGSS as the de facto U.S. science standards for K-12 because, aside from the 20 states that adopted NGSS, science standards in 24 other states are said to be “based” on the same National Research Council (NRC) framework from which NGSS emerged: A Framework for K-12 Science Education [6]. In fact, most of those 24 states’ science standards borrow language directly from NGSS. A cursory review of the 24 sets of NRC Framework-based science standards may lead one to agree with researchers and practitioners who have declared those sets of science standards essentially all versions of NGSS [7,8].

However, close examination reveals differences among NRC Framework-based science standards—some subtle, some considerable. Distinct differences exist regarding how performance expectations and the components of three-dimensional learning, espoused by the NRC Framework, are treated across the sets of science standards. The three components of learning are (a) crosscutting concepts (CCCs) (e.g., cause and effect), (b) science and engineering practices (SEPs) (e.g., analyzing and interpreting data), and (c) disciplinary core ideas (DCIs) (e.g., matter and its interaction). The interconnected representation of these three components has been described as the most critical shift in science education driven by the NRC Framework [9]. The three dimensions are integrated into performance expectations which are statements describing how students should engage in science and what students are expected to achieve and understand.

It is therefore important to understand how performance expectations and the components of three-dimensional learning are expressed across NRC Framework-based science standards. This is valuable to know because various adaptations may affect interpretations among users, namely teachers. As a correlative example from the medical field, it has been shown that simple layout and typography modifications, such as placement of text regarding drug risk information and font changes, substantially affect how users interact with material and influence resulting understandings [10,11]. Likewise, it is reasonable to conjecture that variations among science standards may correspond to variations in what teachers notice and consequently how teachers address instruction.

In this work, the assumption of analogous science standards across the 20 NGSS states and the 24 non-NGSS NRC Framework-based states was evaluated. Understanding differences is important because it confronts the assumption that 44 states effectively have “the same science standards.” Because the state science standards exist in varied arrangements, the first step was to design a classification scheme to organize the sets of standards based on presentation of student performance expectations (RQ1). The resulting classification system yielded a rational way to differentiate broad differences among the standards. The next step was to augment the classification system by determining how the sets of standards treat the components of three-dimensional learning (CCCs, DCIs, and SEPs) (RQ2). This was done by cataloguing and classifying how performance expectations and three-dimensional learning components are expressed and how they exist in relationship to one another within NRC Framework-based science standards.

Following, three sets of standards that outwardly appear similar were scrutinized to assess if and how seemingly small differences affect what teachers notice (RQ3). The work was guided by the following research questions and contributes to developing finer tuned discussions about the state of science standards and more broadly about the presentation of content standards from all disciplines.

RQ1.

How can NRC Framework-based science standards be classified based on presentation of performance expectations?

RQ2.

How does expression of performance expectations in relationship to the components of three-dimensional learning vary across NRC Framework-based states’ science standards?

RQ3.

In what ways do variations among NRC Framework-based states’ science standards affect what teachers notice?

1.1. Relevant Literature and Conceptual Framework

Examinations of content standards often focus on alignment between standards and related curriculum and assessments [12,13]. Less common are crosswalks between two or more sets of K-12 content standards; among those types of studies, international comparisons have provided insight regarding how cultural and political views reflect in educational standards [14]. International comparisons of standards have also served the purpose of determining hallmarks of academically high-achieving countries; in the United States, this was particularly true during the years leading up to establishment of No Child Left Behind legislation and the coinciding push for standards reform [15,16]. As state-level content standards in the United States became the norm, assessing alignment between school district standards and state standards was also of great interest [17]. Widespread adoption of CCSS in the 2010s led to various comparisons. For example, Conley et al. [18] examined the extent to which CCSS aligned to California and Massachusetts standards and Porter et al. [19] evaluated alignment among CCSS, state standards, and enacted curriculum in schools.

1.1.1. Science and Engineering Standards

Prior to the NRC Framework and resulting science standards, there was far greater variation among U.S. states’ K-12 science standards. The variation prompted cross-state comparisons, particularly to assess inclusion of evolution and climate change [20,21,22]. Dissimilarity among state science standards also led to attempts to rank or grade science standards relative to each other. Notable were several evaluations reports from the conservative Fordham Institute think tank which included criteria such as the need for standards to “stress the importance of clear, unambiguous terminology and rigorous definition” [23] (p. 4). The Fordham Institute reports called attention to misgivings among several states’ science standards such as including too many topics and convoluted organization [24]. However, though disseminated widely and influential among many school administrators, Hassard noted the Fordham reviews did not necessarily promote high-level cognitive challenge [25].

Considering NGSS’ emergence from the NRC Framework and that NGSS has served as a template for many of the 24 sets of NRC Framework-based science standards, attention paid to NGSS is not surprising. In fact, the Achieve organization, which headed NGSS development, created the Science Education Standards Comparison Tool [26]. The tool guides a user through a series of 21 two-part open-ended questions enabling comparison of a set of science standards to NGSS. The questions are thoughtful, albeit often cumbersome (e.g., how do the standards expect students to proficiently demonstrate the ability to support scientific hypotheses with evidence and sound scientific reasoning?). When using the Achieve tool, users consider their comparand standards and contrast their answers to those provided by Achieve regarding NGSS. However, due to the broad nature of its questions and because the Achieve tool substantiates answers to all 21 questions with evidence drawn from either the NGSS Introduction or Appendices, the tool is not seen as useful for analysis of usability or content [27].

As NGSS and seemingly similar NRC Framework-based science standards were adopted, state-to-state crosswalk comparisons became less prevalent in the research literature. Among enduring comparison studies were Vazquez’ examination of the placement of evolution in science standards wherein states that had adopted NGSS were compared to others [22]. The treatment of engineering in science standards also remained a topic for comparison studies. Moore et al. analyzed the distribution of engineering-related standards across the United States prior to NGSS and compared that to NGSS [28]. Related, Lopez and Goodridge examined the state of engineering in science standards by comparing science standards from states that had adopted NGSS with states that had “effectively adopted the NGSS” and with states that had done neither [29] (p. 4). Their focus on engineering in K-12 standards was important because prior to the NRC Framework, the engineering domain was nontraditional for K-12 science standards [30]. Lopez and Goodridge recorded that 70% of the states at the time used an NRC Framework-based set of science standards, but noticed that some of the NGSS-like states organized standards differently than NGSS and some states severely modified inclusion of engineering standards or even left them out completely [29].

Lopez and Goodridge’s observations demonstrated unsuitability of the assumption that all NRC Framework-based science standards are equivalent. However, their study shed light only on inclusion of engineering. Therefore, a broader approach was taken here to investigate how NRC Framework-based science standards compare to NGSS, to NRC tenets, and to each other. Unlike studies that have assessed the degree to which sets of content standards include specific criteria (e.g., [18,19]), the objectives of this work focused on providing researchers and practitioners improved ways to consider comparability of NRC Framework-based state science standards.

1.1.2. Framework to Assess Variations among Science Standards

Assessing variations among science standards that are based on the NRC Framework requires consideration of the Framework’s chief principles. Therefore, a first step is to ensure understanding of guidance provided by the NRC Framework to science standards developers. Second, an aligned framework to organize the presentations and the effects of variation is obliged.

Proposals to further science learning through student-centered learning and inquiry-based investigations precede the NRC Framework by decades [31]. Yet, the NRC Framework advanced conversations about what science instruction and learning looks like by laying out a distinct vision centered on the integration of CCCs, SEPs, and DCIs, i.e., three-dimensional learning. The NRC Framework asserts that collectively these elements “define what it means to know science, not as separate elements but as intertwined aspects of knowledge and understanding” [32] (p. 320). The NRC Framework committee suggested that the integration and representation of three-dimensional learning should occur across the standards but particularly in performance expectations. Performance expectations represent what students are expected to achieve and performance expectations are sometimes even referred to as “the standards.” For example, in Iowa, which adopted NGSS in 2015, the state department of education states, “Iowa’s science standards are the performance expectations of the Next Generation Science Standards” [33].

NRC Framework-based science standards are alike in that they all present performance expectations. The structure of performance expectations are also quite similar across the sets of state science standards—the statements indicate students should apply science and engineering practices to understanding disciplinary core ideas in ways that will further understanding of crosscutting concepts. Most simply, the statements are what students should be able to do. However, the layouts and the use of typography within which those performance expectations exist across state standards vary considerably. Layout and typography are rarely studied in the context of content standards. Layout guides interpretation of information and typeface cues what requires most attention [34,35]. These components of presentation, as well as omission and inclusion of information, are quite important in both signaling intent of developers and determining how standards will be interpreted [36].

1.1.3. Overall Methodology

The purpose of this work was to consider layout differences among NRC Framework-based science standards and explore if variations affect how attention is drawn. To that end, a three-step framework process was followed. The first step was to address the relative macro level of layout (RQ1). Characterizing the various types of layouts of science standards is an important way to reduce the “noise” but also allows researchers to consider efficacy of different layouts. Researchers conducting international comparisons of textbooks emphasized that the layout of and relationship between instructional goals and key information yields a type of “signature” which strongly influences how teachers use and interpret textbooks [37]. This concept about the importance of layout design and the relationships between objectives and key information was adopted. Because performance expectations are the instructional goals of content standards and because performance expectations are core to all sets of NRC Framework-based science standards, the manner of presentation of performance expectations was central to classifying the standards (RQ1).

Likewise, it was important to characterize the various ways in which the components of three-dimensional learning (CCCs, DCIs, SEPs) are laid out and how those elements are presented in relationship to performance expectations (RQ2). Of particular importance are the alignment and proximity of associated items—in this case, performance expectations and components of three-dimensional learning. The adjacency of key information within documents signify the “principles that organize and unify [a] document and show relationships between the various objects” [38] (p. 243). To assess these relations among performance expectations and three-dimensional learning components, Pauwels’ framework for analyzing textual and visual material was adapted [39]. Pauwel emphasizes that layouts and expressions of material can represent underlying perspectives of the culture of the developers. Decoding that information helps us to understand intended and unintended persuasions.

Specifically, after characterizing the types of layouts, the second step was determining the relationship of “distance” between fundamental information. In the context of NRC Framework-based science standards, the measurement of interest is the distance from performance expectations to supporting information about the three dimensions of learning. Though performance expectations should embody all three dimensions, the position and ease of locating supporting information may affect how teachers wholly interpret science standards. At the same time, cluttered presentations may disenchant users. No matter, the placement of information contributes to what Valverde et al. referred to as a document’s signature and potentially the teaching ideology it projects [37].

The third step was to assess if differences among science standards affect what teachers notice (RQ3). To this end, a hermeneutic methodological lens was applied. Hermeneutic inquiry is rooted in examining everyday understanding of common objects, including textual material [40]. Hermeneutic frameworks involve identifying objectively verifiable aspects and determining users’ open-ended interpretations. Therefore, to address the research question’s focus on understanding what individual teachers immediately notice, a hermeneutic framework was preferable to other common methodological approaches such as phenomenology, which centers on lived experiences, or symbolic interactionism, which examines shared meanings resulting from social interaction.

In the context of content standards, recognition of significant components as well as interpretations of the whole of content standards may shift due to developers’ intentional and unintentional design choices. Specifically, developers draw attention in various ways, such as through use of color, typeface, layout design, and through insertion and absence of information [10]. It is known that even the most ubiquitous of design choices, namely font style, can affect readers. Fonts can connote personality of written material and can even evoke emotional responses [41,42]. Other mechanisms, such as type size and spacing around material (i.e., whitespace) also guide readers’ attention and can indicate a designer’s intended hierarchy of importance [43].

Within this study’s three-step framework process, the emphasis was not on evaluating specific design choices, such as whether a serif font is more readable than a sans serif font. Returning to the initial goal of determining if NRC Framework-based science standards are equivalent, the focus was instead on assessing design differences and how those differences draw a teacher’s attention differently. Most research related to design layout is situated in traditional texts such as advertisements and expository text and focuses on customary dependent variables such as readability and memory. What is not clear is how variations affect what users notice within content standards and their interpretations regarding what is important. The particulars of how noticing was assessed in this study are detailed when discussing methods.

To address the research questions, two studies were carried out. Study 1 addressed RQ1 and RQ2. This was a document analysis of NRC Framework-based science standards with the goal of categorizing the science standards and determining their treatment of the components of three-dimensional learning. Study 2 addressed RQ3 and focused on how a sample of teachers interacted with three purposively selected sets of science standards. Organization of the Methods and Results sections are branched based on the two studies. Following, findings of both studies are brought together in the Discussion section.

2. Study 1 Methods: Layouts and Treatment of Three-Dimensional Learning

Study 1 was a document analysis designed to yield characterization of the layouts of NRC Framework-based science standards and to indicate how performance expectations and three-dimensional learning components are presented in association to one another.

2.1. Sample of Standards

The 24 K-12 state science standards said to be based on the NRC Framework were the data sources for this study. The list of 24 states came from the National Science Teaching Association (NSTA), which tracks whether a state adopted NGSS, developed science standards based on the NRC Framework, or did neither [44]. Readers are referred to the NSTA’s site to view an interactive map indicating the state of science standards across all 50 states.

Twenty of the science standards were available as single PDF documents that include standards for all grades, K-12. The science standards from Louisiana, Indiana, and Georgia were available as separate grade-level and/or discipline-based documents. West Virginia’s science standards were available as grade-specific screens only online (i.e., not as downloadable documents).

During preliminary inspection, ease of use, inclusion of supplementary material, navigation, and general look and feel of the science standards were noted. Through this exploration, it was confirmed that all 24 sets of standards were indeed influenced by, if not thoroughly based on, the NRC Framework. In fact, several of the sets of standards included language that was frequently identical to that found in NGSS.

Many of the standards documents acknowledge in their introductory material the influence of expected elements such as state assessments, standards committees, elected and appointed officials, and public input. However, it was noticed that the science standards of Arizona and South Carolina were further shaped by influences beyond those anticipated and beyond the NRC Framework. Arizona’s science standards additionally preface each grade section with a repeated list of ten core ideas for knowing science (e.g., all matter is made of very small particles) and three core ideas for using science (e.g., knowledge produced by science is used to solve problems and/or create products), adopted from the publication Working with Big Ideas of Science Education [45]. However, Arizona’s performance expectations appear to be shaped chiefly by the NRC Framework. South Carolina indicates their science standards development team utilized the NRC Framework plus eight other resources, including the state’s previous K-12 science standards and a publication from the conservative Fordham Institute [23]. This mix of resources likely contributes to South Carolina’s science standards appearing the most dissimilar to NGSS among the science standards reviewed.

2.2. Data Analysis

The process of categorizing the science standards focused on how performance expectations are conveyed. Creating groups for the science standards followed document analysis guidelines of conducting cursory and thorough examinations, tagging documents, creating preliminary groupings and iteratively resolving until final groups are fixed [46,47]. As an assessment of alignment with the NRC Framework, it was important to determine the extent to which the standards are “based” on the NRC Framework’s three major dimensions that the NRC stresses should be integrated into science education: CCCs, SEPs, and DCIs.

Even among the shortest sets of science standards, which are essentially lists of performance expectations, it was apparent that all 24 sets include the three NRC dimensions of learning in some manner. However, the ways this is accomplished, and the attention called to the three dimensions varies. For example, New York’s science standards provide distinct textboxes outlining the CCCs, SEPs, and DCI component ideas that correspond to each performance expectation. As a comparison, although Utah’s science standards provides descriptions of the three dimensions in its introductory chapter, when presenting grade-level performance expectations, typeface highlights CCCs (underlined) and SEPs (boldface). For example, Utah’s performance expectation for Standard 4.2.4 is to “design a device that converts energy from one form to another.”

Considering the time constraints of teachers, particularly elementary teachers when tasked with teaching science [48], it was conjectured that teachers are inclined to read content standards from the “middle out” by first, and perhaps primarily, reading the portions of the standards that are most relevant to their grade or discipline. In doing so, across all of the science standards examined, a teacher who is “getting right to it” and looking up the standards for their grade level will at the very least encounter DCI-based headings (e.g., energy, life science). However, state science standards vary considerably regarding how they present SEPs and CCCs, as well as DCI component ideas. In all cases, it is arguable that SEPs are integrated into performance expectation statements and performance expectations support understanding of CCCs. For example, the performance expectation “plan and conduct an investigation to provide evidence of the effects of balanced and unbalanced forces on the motion of an object,” integrates the SEP of planning and carrying out investigations and should promote understanding of the cause and effect CCC. Yet, due to the variation in presentation and because addressing SEPs and CCCs may be nontraditional for many teachers, it was conjectured that adjacency or distance between performance expectations and information that explained or at least laid out the NRC SEPs and CCCs was a metric of interest.

Similarly, the distance from performance expectations to corresponding information about disciplinary component ideas was of interest. Component ideas are the more specific concepts within DCIs. For example, the concept that “climate describes a range of an area’s typical weather conditions and the extent to which those conditions vary over years” is a component idea of the larger core idea of weather and climate. Though component idea information found in science standards is less than what is needed for teachers to be thoroughly knowledgeable about a concept, the information does provide foundational material that can help to shape understanding.

To catalog distance, the standards documents containing fourth-grade standards were examined. Nearest distance from performance expectations to elaboration about SEPs, CCCs, and DCI component ideas was recorded using the following magnitudes and these values are applied in Table 1 of Study 1’s Results section:

• 1 = Found in the same subsection as grade-level PE
• 2 = Found in subsection different from PE subsection but within grade-level section
• 3 = Found in a section outside of grade-level section but within grade-band section
• 4 = Found in a section outside of grade-level and grade-band sections
• 5 = Not found in core science standards document

Table 1. Distance from PEs to explanation of three dimensions.

Layout Type and State	CCCs	SEPs	DCI Component Ideas
Rolling layout
Alabama	5	5	5
Georgia	5	5	5
Indiana	5	2	5
Massachusetts	4	4	5
Minnesota	4	4	4
Mississippi	3	3	1
Montana	4	4	5
Nebraska	4	4	5
South Dakota	4	4	1
Tennessee	4	4	5
West Virginia	5	5	5
Related-PEs discrete layout
Arizona	1	4	2
Idaho	5	5	1
New York	1	1	1
NGSS	1	1	1
South Carolina	4	4	1
Utah	4	4	1
Single-PE discrete layout
Alaska	1	1	1
Colorado	1	1	1
Louisiana	1	1	1
North Dakota	1	1	1
Oklahoma	1	1	1
Wyoming	1	1	1
Columnar
Missouri	5	5	5
Wisconsin *	-	-	-

1 = same subsection as PE; 2 = different section than PE but within grade-level section; 3 = outside of grade-level section but within grade-band section; 4 = outside of grade-level and grade-band sections; 5 = not found in the science standards document. * Wisconsin presents standards by grade band with color-coded sections for CCCs and SEPs.

Table 1. Distance from PEs to explanation of three dimensions.

Layout Type and State	CCCs	SEPs	DCI Component Ideas
Rolling layout
Alabama	5	5	5
Georgia	5	5	5
Indiana	5	2	5
Massachusetts	4	4	5
Minnesota	4	4	4
Mississippi	3	3	1
Montana	4	4	5
Nebraska	4	4	5
South Dakota	4	4	1
Tennessee	4	4	5
West Virginia	5	5	5
Related-PEs discrete layout
Arizona	1	4	2
Idaho	5	5	1
New York	1	1	1
NGSS	1	1	1
South Carolina	4	4	1
Utah	4	4	1
Single-PE discrete layout
Alaska	1	1	1
Colorado	1	1	1
Louisiana	1	1	1
North Dakota	1	1	1
Oklahoma	1	1	1
Wyoming	1	1	1
Columnar
Missouri	5	5	5
Wisconsin *	-	-	-

1 = same subsection as PE; 2 = different section than PE but within grade-level section; 3 = outside of grade-level section but within grade-band section; 4 = outside of grade-level and grade-band sections; 5 = not found in the science standards document. * Wisconsin presents standards by grade band with color-coded sections for CCCs and SEPs.

It is not assumed that this measurement system correlates to quality. The NRC Framework recommends that standards should be “clear, concise, and comprehensible to science educators” [6] (p. 299). Therefore, positioning more information closer to the performance expectations may not necessarily be the best means to achieve that goal. However, the extent to which attention is drawn to explicit information about the three dimensions is considered potentially valuable, particularly for understanding how teachers may interpret science standards and the degree to which such information is considered significant or extraneous.

3. Study 1 Results

3.1. Categorization

Results of the categorization process yielded three major classifications of standards based on presentation of performance expectations: rolling, discrete, and columnar. Among the 24 states examined, 11 states present grade-level performance expectations (PEs) as rolling layouts. Rolling layouts provide all grade-level PEs on one or more successive pages with discipline topics (e.g., physical science) as the prominent breaks. Rolling layouts primarily present a series of statements indicating what students should do (i.e., performance expectations). States such as Tennessee provide clean rolling layouts comprised only of PEs. Other states with rolling layouts additionally provide sub-objectives (e.g., Alabama) or clarification statements (e.g., Massachusetts). In all cases, rolling layouts primarily present a succession of student objectives with CCCs and SEPs integrated into the student objective statements (i.e., the PEs). Figure 1 provides examples of rolling layouts from Montana and Alabama.

Eleven other states present grade-level PEs as discrete layouts. Discrete layouts, such as NGSS, group related sets of PEs or present single PEs on dedicated pages. Discrete layouts provide adjacent supportive information such as background information or explication of CCCs. The eleven states with discrete layouts are further classified as a related-PEs layout or single-PE layout. Among these eleven states, five present PEs in related sets, as does NGSS. The six other states present PEs one at a time. Figure 2 provides an example of the difference between related-PEs (New York) and single-PE (Alaska) discrete layouts. Single-PE discrete layouts are also found in Figure 3, Figure 4 and Figure 5. Among states with discrete layouts, Alaska additionally prefaces each grade-level PE section with a rolling layout—essentially an advance organizer.

The layouts for Missouri and Wisconsin are neither rolling nor discrete, but instead described as columnar. Missouri presents K-5 PEs across 31 pages in six columns: sections of rows are demarked by DCIs and columns represent grades. Wisconsin similarly presents PEs in columnar format but instead of specifying grade-specific standards, their standards indicate the PEs are to be addressed by grade bands (K-2, 3–5, 6–8, 9–12). Some states, in addition to their discrete or rolling layouts, provide learning progression charts that resemble the Missouri and Wisconsin columnar designs.

3.2. Distance from PEs to Elaboration of Three Dimensions of Learning

Table 1 summarizes results of the distance between a performance expectation and information about SEPs, CCCs, and DCI-related component ideas, ranging from 1 equal to the information found in the same subsection as the PE to 5 indicating the information was not found in the science standards. It was found that information about SEPs and CCCs was most specific to performance expectations when presented nearest them. That is to say, when SEP and CCC information was found in the same subsection as a performance expectation (coded as a 1 in Table 1), the information was specific to the performance expectation and not simply a list of SEPs and CCCs. When presented in the same subsection as a performance expectation, specificity of SEPs and CCCs varied from simple typographical emphasis (e.g., bold-facing the corresponding SEP in a list of all SEPs, see Figure 4 for example) to providing only the specific SEP and CCC that match the performance expectation.

Due to their more elaborate structures, related-PE and single-PE discrete layouts generally provide SEP, CCC, and DCI information nearer to performance expectations. However, this is not uniformly the case among related-PE discrete layouts. For example, although Idaho’s related-PE layout includes DCI information next to performance expectations, such as definitions of energy and explanation of the relationship between energy and forces when presenting energy-related third-grade performance expectations, CCCs and SEPs are not explained within the document.

Though it was expected that rolling layouts, owing to their structure, would not provide nearby information about the three dimensions, it was surprising that science standards with rolling layouts generally only included the information in introductions or appendices, and in some cases not at all. For example, although Alabama’s science standard states that “students are introduced to disciplinary core ideas and crosscutting concepts in the domains of Physical Science; Life Science; Earth and Space Science; and Engineering, Technology, and Applications of Science through content and participation in scientific and engineering practices”, no explication is provided in the document regarding what exactly are crosscutting concepts or science and engineering practices.

4. Study 2 Methods: What Teachers Notice

As stated, states use various layouts to organize and present performance expectations and they apply various typography methods to draw attention. Additionally, included information can vary—for example, one state may include information about connections to mathematics standards while other states may not provide any peripheral information. To explore how these mechanisms guide attention, in Study 2, standards from three states with single-PE layouts were selected: Colorado (Figure 3), Oklahoma (Figure 4), and Wyoming (Figure 5). The reason only single-PE layouts were chosen for this closer examination was to understand if visual design choices affect teachers’ noticing even when overall layout and information are seemingly equivalent. Further, it was not an objective of this study to determine a “winner” among the layout and typography choices. If results indicated teacher attention is guided differently by seemingly analogous layouts, then those results contribute to a call for greater attention to the design of content standards. Therefore, the objective was to determine if evidence supported the conjecture that layout and typography affects what teachers notice. If so, future study should address if varied noticing corresponds with differences in what gets taught.

4.1. Sample of Teachers

Elementary school teachers from a state that had not adopted NGSS nor established their own NRC Framework-based science standards were solicited to complete an online survey. Teachers from a non-NRC state were selected because there was concern that teachers from NRC states would be too accustomed, if not adherent, to the layout and the way attention is drawn to three-dimensional learning in their own state standards.

A pool of potential respondents was developed by collecting publicly available email addresses of elementary school teachers (K-5) in the non-NRC state. School districts were randomly selected from a list provided by the state’s department of education. Two hundred and eighty-eight elementary school teachers responded. The teachers represented 24 school districts and had an average of 13.2 years of teaching experience (standard deviation (SD) = 8.9 years). Teachers completed the survey anonymously and upon completion received a link to enter a drawing for a gift card.

4.2. Survey and Data Analysis

An intent of the survey was to support moving beyond traditional document analysis that centers primarily on coding written text, and to consider how differences among analogous standards with presumed similar information affects interaction with the standards. The survey was structured to address RQ3 regarding what teachers notice when interacting with standards. Noticing was discerned as two aspects. The first was assessing teachers’ identification of performance expectations because these are the core statements of science standards; performance expectations indicate what students should be able to do and they are comprised of three-dimensional learning components. Therefore, it was of interest to learn the extent to which layout and material differences among the three state science standards directed teachers differently when identifying what they considered the chief learning objectives.

The second aspect of noticing supported by the survey was an open-ended exploration asking what element on the standards stood out and why. This supports understanding how different layouts draw attention differently and provides information regarding the situated meaning ascribed to different elements of textual material [49]. This aligns with the hermeneutic lens of noticing specific components in common material as well as rationalizing such identifications.

On three successive screens, presented in random order, the teachers saw the pages of the Colorado, Oklahoma, and Wyoming science standards that contain the nearest neighbor to NGSS PE 4-PS4-2: develop a model to describe that light reflecting from objects and entering the eye allows objects to be seen. Teachers were presented with three prompts on each of these three screens. The first prompt is a task retrieval based on Cawthon and Vande Moere [50]. The task requires identification of the student performance objective from the layout. Specifically, teachers were asked to “click on the region of the following that best represents what students should know, understand, and/or be able to do that is measurable and observable.” The wording of this prompt is paraphrased from the NRC Framework’s definition of a performance expectation [6] (p. 218). In other words, teachers were asked to identify the performance expectation but were not provided the term “performance expectation” in the prompt.

The second and third prompts asked teachers to provide open responses to the questions, “As a teacher, what in the above stands out to you as needing most attention?” and “Why?” Responses to the what stands out question were coded using predetermined (a priori) codes based on the NRC three dimensions of learning and state-specific elements (e.g., Colorado’s elaboration of grade level expectations), as well as emergent coding developed for responses that did not fit a priori categories. Responses to the why item were coded using an inductive, constant comparative method. Combining inductive and emergent coding methods is common practice among qualitative researchers and in this case allowed for assessing the degree to which prescribed science standards elements (e.g., the DCI) stood out to teachers but also allowed for understanding the degree to which tacit elements (e.g., challenge posed to students) were raising attention.

For both the what stands out and why questions, responses were assessed through multiple reads and 30% of the data were re-coded two months after initial coding to ensure reliability as outlined by Mackey and Gass [51]. The survey did not include a Back button; therefore, respondents could not revisit items already completed. Metadata of the amount of time teachers took before their first click (i.e., seconds to highlight the performance expectation) and total time on a screen were collected. These metadata and teacher responses were denoted based on whether a state’s layout (CO, OK, or WY) was randomly presented to respondents first, second, or third.

5. Study 2 Results

5.1. Teachers Identification of Performance Expectations

As seen in Figure 3, Figure 4 and Figure 5, the performance expectation examined was worded identically for Colorado, Oklahoma, and Wyoming: “Develop a model to describe that light reflecting from objects and entering the eye allows objects to be seen.” In response to the item asking teachers to “click on the region … that best represents what students should know, understand, and/or be able to do that is measurable and observable,” approximately half of the teachers identified the performance expectation.

Though the plurality of teachers identified the performance expectation in this exercise, the array of responses differed among the three state standards. Because it was conjectured that answering the same prompt on three successive screens might condition teachers to select the same response as selected on a previous screen,

Table 2. Percent of responses identifying different parts of standards as the performance expectation.

	State Viewed 1st *			State Viewed 1st, 2nd, or 3rd
	CO (n = 104)	OK (n = 102)	WY (n = 82)	CO (n = 255)	OK (n = 253)	WY (n = 242)
Performance Expectation	51.0	49.0	39.0	50.2	54.5	46.3
Disciplinary Core Idea	14.4	2.9	12.2	9.0	4.0	5.4
Crosscutting Concept	2.9	1.0	9.8	4.7	1.2	14.0
SEP about developing a model	7.7	19.6	24.4	11.0	18.2	21.5
Other SEPs—not specific to models (OK only)	-	25.5	-	-	20.6	-
Elaboration on grade level expectation (CO only)	14.4	-	-	13.7	-	-
Prepared grads statement (CO only)	8.7	-	-	10.2	-	-
Assessment boundary (OK and WY)	-	2.0	3.7	-	1.6	2.1
Cross-curricular connection (WY only)	-	-	11.0	-	-	10.7
Other	1.0	-	-	1.2	-	-

CO = Colorado, OK = Oklahoma, WY = Wyoming, SEP = science and engineering practice. * Columns refer to responses when the state viewed first by respondents (teachers shown three states in random order).

summarizes overall responses, as well as responses when viewing a particular state’s standard first. The mean amount of time teachers took before clicking on their choice for the performance expectation varied. Considering only when a state was randomly shown first, teachers took an average of 33.8 seconds when viewing Colorado (n = 104, SD = 53.4), 43.6 seconds when viewing Oklahoma (n = 102, SD = 68.5), and 30.4 seconds when viewing Wyoming (n = 82, SD = 30.4); however, analysis of variance indicated these differences were not statistically different.

To determine if significant differences existed in the identification of the performance expectation among the three states’ science standards, a chi-square test of homogeneity was conducted. The dependent variable was set as binary, i.e., either the performance expectation was identified, or another area was identified as being the measurable objective. Although Table 2 indicates teachers were least likely to identify the performance expectation when viewing the Wyoming standards, results revealed no statistically significant differences regarding identification of the PE among the states, χ² (2, N = 288) = 2.92, p > 0.05. In a similar analysis of the “distractors” of the DCI, CCC, and the model-focused SEP, a statistically significant difference was found: teachers were more likely to select the DCI as best representing a measurable student performance goal when viewing the Colorado standard as compared to the Oklahoma standard, χ² (2, N = 288) = 9.49, p < 0.05.

5.2. What Teachers Notice and Why

Analysis of open responses to the what stands out and why questions were limited to responses to the first state a teacher was randomly presented by the survey software. Data were constrained this way because several teachers, when viewing a second or third state’s standard, provided general reflections about preferences for one layout over another and did not necessarily respond to the questions. Additionally, likely due to survey fatigue, some teachers did not respond to the questions for their second and third states.

In response to the item asking teachers what stands out as needing most attention, 262 teachers provided responses. Eighty-three percent of the responses indicated a specific element from the standard, such as the crosscutting concept statement or a science and engineering practice, or specified a particular element, such as a rubric, was missing. Among the other seventeen percent of responses, approximately two-thirds were broad statements of praise or criticism; other statements among the seventeen percent were concerns or reflections about how the standard was affecting thinking about teaching and learning. Examples of these latter responses include thoughts about needing to connect the standard to learning in other grades (1.7%) and commentary about how students will find the standard challenging (3.1%). Generally, when teachers did not specify a particular element of the standard in response to the what stands out question, they echoed their response again when replying to the why question. In some cases, teachers provided “see above” type of replies to the why question. Therefore, the seventeen percent of the responses that did not specify a particular element of the standards were re-examined and coded as responses to the why question if the statements provided rationales or concerns not already offered in response to the why question. Based on this,

Table 3. What teachers identified as needing most attention as percent of responses per state.

	CO	OK	WY	p
Found in all three states’ standards
Performance Expectation (PE)	13.9	13.9	25.0	0.247
Disciplinary Core Idea (DCI)	25.3	5.0	10.9	0.001
SEP about developing a model	19.0	28.7	25.0	0.077
Crosscutting Concept (CCC)	7.6	1.0	12.5	0.024
Noted something missing (e.g., rubric)	6.3	5.9	7.8	0.949
Vocabulary	5.1	3.0	1.6	0.521
Other	2.5	0.0	1.6	0.363
Not in all three states’ standards
Other SEPs—not specific to models	NA	35.6	NA	–
Prepared graduates statement	10.1	NA	NA	–
Elaboration on grade level expectation	10.1	NA	NA	–
Assessment boundary	NA	6.9	3.1	–
Cross-curricular connection	NA	NA	12.5	–

summarizes the most prevalent responses regarding what stands out to the teachers. Likewise,

Table 4. Reasons attention was raised (why responses) as percent of responses per state.

	CO	OK	WY	p
Guides teaching, prompts instructional thinking
teacher stated it indicates what students should do or know and/or is foundational, including guides assessment	22.2	22.9	18.8	0.846
teacher stated how it benefits students (e.g., promotes critical thinking)	13.1	18.1	20.0	0.201
teacher provided consideration of how to teach the standard (e.g., hands-on)	14.1	10.5	18.9	0.085
teacher provided consideration of pre- and co-requisite skills including other grades and/or content areas	4.0	6.7	12.6	0.026
Raises concern because…
students have deficits or will find it challenging	20.2	15.2	10.5	0.332
standard lacks clarity, vague, poorly worded	19.2	11.4	10.5	0.233
teachers have shortages (time, materials, or training)	5.1	7.6	8.4	0.469
Design-related reason
layout or typography pointed to it being important	2.0	7.6	2.1	0.086

summarizes the most common responses to the why question.

A series of chi-square analyses with Bonferroni post hoc tests were conducted to assess differences between states. Results indicated that in response to the what stands out question (Table 3), teachers were significantly more likely to point out that the DCI required the most attention when viewing the Colorado standard as compared to the Oklahoma standard (p < 0.05). Similarly, teachers were significantly more likely to spotlight the CCC as needing most attention when viewing the Wyoming standard as compared to the Oklahoma standard (p < 0.05). Examination of the coded responses to the why question (Table 4) revealed teachers were significantly more likely to mention they were thinking about connections to other disciplines and other grades when viewing the Wyoming standard as compared to the Colorado standard (p < 0.05). Considerations regarding what underpinned the differences are provided in the Discussion section.

In an effort to discern if patterns could be detected in the why data for cases when the what stands out responses yielded significant differences, data were limited to just when teachers indicated the DCI stood out and limited again to examine just when teachers indicated the CCC stood out. This resulted in small data subsets and no discernible patterns were detected that distinguished why teachers were significantly more inclined to say the DCI stood out when viewing the Colorado standard as compared to the Oklahoma standard or why the CCC stood out significantly more when viewing the Wyoming standard as compared to the Oklahoma standard. In the Discussion section, suppositions regarding these differences are provided based on observations related to structural differences among the three state standards.

In Table 4, coded categories regarding why attention was raised are grouped as (a) guiding teaching by evoking thinking about pedagogy, learning, and student benefits; (b) effecting uneasiness by raising concerns related to ambiguity of the standard, student deficits, and shortages (time, materials, training); and (c) statements about the standard’s design (e.g., font size) informing importance. As an example of the standard guiding teaching, Teacher #13, when viewing the Colorado standard, indicated the performance expectation raised their attention because “it helps me understand the focus of my instruction and the activity my students will do.” As an example of the standard effecting uneasiness, Teacher #240, when viewing the Wyoming standard, indicated students developing models was concerning because “that concept is very difficult for students to understand because it deals with abstract concepts and the wording is difficult for 4th graders.” Finally, among the instances of a teacher plainly stating a standard’s design signaled importance, Teacher #177, when viewing the Oklahoma standard, stated the model-related SEP drew their attention “because it is all in bold and that is the first thing my eyes go to”.

6. Discussion

The genesis of this work grew from interest in scrutinizing assumptions about the comparability of NRC Framework-based science standards. It is understandable to think that science standards based on the same framework are likely equivalent. While it is not refuted that the 24 sets of science standards examined do indeed integrate tenets of the NRC Framework, particularly the three dimensions of learning, findings counter the notion that NRC Framework-based science standards are necessarily commensurate to each other or to NGSS.

Study 1’s examination of the NRC Framework-based science standards called attention to the use of various layouts, as well as to the ways and degrees of integration of the Framework’s three key dimensions of learning. The NRC Committee recommended that standards emerging from the NRC Framework [6] should (a) “include performance expectations that integrate the scientific and engineering practices with the crosscutting concepts and disciplinary core ideas” (p. 301) and (b) “emphasize all three dimensions” (p. 300). All of the science standards examined meet this first criteria of providing performance expectations that integrate the three dimensions of learning; however, emphasis placed on the dimensions varies as characterized by the distance from performance expectations to information about the three dimensions (see Table 1).

Of course, emphasis or de-emphasis within a set of science standards on the three dimensions may be offset by professional development and curriculum materials. Given this, developers of streamlined science standards, particularly developers of rolling layout science standards, may have considered the standards primarily as starting points of student objectives. In terms of aesthetics and simplicity, several of the sets of science standards certainly appear more immediately understandable to a novice user than NGSS with its varied use of typeface, competing eye-catching colors, and multiple textboxes. In fact, the seeming complex organization of NGSS, which has been pointed out by practitioners [52,53], may have pressed science standards developers toward modifications. The goal of providing clear straightforward lines between standards and associated assessments also affects how developers design content standards [3].

Nonetheless, NRC Framework-based science standards are to “provide a mechanism for communicating educational priorities” [6] (p. 299). Placing information about the three dimensions farther from the performance expectations and in some cases not including such information in the standards document may be an attempt to provide a clean set of understandable goals, but as Bowen asserted the “absence, sparseness, or incompleteness of documents should suggest something about the object of the investigation or the people involved” [46] (p. 33). In other words, placing maximum emphasis on performance expectations and minimal prominence on the three dimensions may affect teachers’ views about how to approach teaching the standards, particularly when they are working matters out on their own without aligned professional development or materials.

To assess how science standards variation might affect what teachers notice, Study 2 took a conservative approach by comparing three sets of standards that ostensibly appeared equivalent. While not surprising, an important first finding was that teachers do not notice what is not evident. Specifically, although the Colorado, Oklahoma, and Wyoming standards were generally similar, the unique elements of each drew the attention of at least some teachers (see Table 3). For example, Wyoming’s inclusion of cross-curricular connections to mathematics and English language arts accounted for 12.5% of the what stands out responses; this tracks to responses to the why question related to non-science disciplines and grades other than fourth-grade. Specifically, teachers provided rationales about other disciplines and grades nearly twice as often when viewing the Wyoming standard than when viewing the Oklahoma standard and three times more often than when viewing the Colorado standard (p < 0.05).

The effect of conspicuous information is also apparent when considering that Colorado and Wyoming provide only the corresponding SEP related to the fourth-grade performance expectation that was examined, i.e., developing and using models. On the other hand, Oklahoma provides a list of eight SEPs and emphasizes the modeling SEP by setting it in boldface. Though not significantly different from the other two states, that glaring typeface amid a list is perhaps the reason that a higher proportion of teachers noticed the modeling SEP on the Oklahoma (28.7%) science standard than the Colorado (19.0%) and Wyoming (25.0%) science standards. Providing the full list of SEPs also led to the plurality of Oklahoma responses (35.6%) highlighting the seven non-corresponding SEPs. Arguably, presenting the non-corresponding SEPs distracts teachers; however, it can also be contended that attention paid to other SEPs and not just the one at hand supports teachers thinking more broadly about mindsets and skills students should develop.

Moving beyond this point, the Colorado/Oklahoma/Wyoming comparison data provided other meaningful findings related to components found on all three standards. Notably, there were significant differences in teachers noticing central components of the three dimensions of learning. All three state standards included word-for-word the same DCI (“an object can be seen when light reflected from its surface enters the eyes”) and the same CCC (“cause and effect relationships are routinely identified”). As noted, teachers highlighted the DCI significantly more on the Colorado standard than on the Oklahoma standard (p < 0.05) and considerably more on the Colorado standard than on the Wyoming standard. Limits of this study do not permit a full analysis of why the DCI jumped out so much more on the Colorado standard, but two conjectures are offered. First, Colorado labels the DCI as the “Grade Level Expectation,” while Oklahoma and Wyoming list it under the heading of “Disciplinary Core Ideas” and provide the subheading of “Electromagnetic Radiation.” Forgoing the title of “Disciplinary Core Idea” and the technical term “electromagnetic radiation” and instead providing a simpler heading possibly led to teachers noticing the DCI more often on the Colorado standard. Also, nontrivial is the fact that the font size of the DCI on the Colorado standard is larger than most other parts of the standard, while on the Oklahoma and Wyoming standards the DCI’s font size is the same or smaller compared with other elements near it. Although this discussion about headings and font size may seem to be getting into the weeds, these are important navigation and signaling aids that have long been considered by experts in fields such as marketing and e-learning but are not commonly discussed when thinking about content standards [54,55].

Though the differences among the science standards of Colorado, Oklahoma, and Wyoming discussed here are revealing, the chief takeaway from Study 2 is that there were differences in teacher responses. Study 1 classified the NRC Framework-based science standards into distinct categories and indicated considerable differences in the placement of key information. Yet, Study 2 revealed that even when limited to just three well-matched state standards from the same layout category, teacher attention is drawn differently. Therefore, researchers and practitioners should reconsider messaging about the consistency of NRC Framework-based science standards and seriously consider implications on interpretations.

6.1. Limitations

The teachers in Study 2 were from a state that had not adopted NGSS nor created their own NRC Framework-based standards. The reason for this selection was to have teachers who were less likely to have predispositions for their own state’s NRC Framework-based science standards and to effectively have “fresh eyes.” However, it is acknowledged that such an assumption of objectivity may be problematic. Given the pervasive reach of NGSS and the NRC Framework, many educators in non-NRC states are likely cognizant of three-dimensional learning. Therefore, though the approach was an attempt to improve objectivity, it is recognized the sample was not necessarily a clean slate. Further, the restrained sample limits generalizability of findings. Generalizability is also limited because the study focused on one fourth-grade performance expectation; therefore, inferences that expand to other grades and disciplines can only be made with caution.

Particular to Study 2 is the consideration of ecological validity. The survey prompted teachers to point out what they saw as the student objective and to identify what they believed needed most attention. This manufactured exercise does not necessarily match real-world actions of teachers when they interact with science standards. People possess numerous strategies for taking in informational text and these include previewing, scanning, annotating, and studying [56]. Therefore, the survey’s prompts likely pushed respondents from the approaches they typically use when encountering new expository material and consequently the study’s results should be filtered through this understanding.

Finally, the design of both studies must be considered. No prior research matched the goals of this study. This led to adapting methods from disciplines such as media design. Adapting methods for a new type of application itself can provide uncertainty. Traditional methods, such as content analyses of entire sets of standards, did not fit the need. Therefore, the methods applied are viewed as one set of approaches and not necessarily the only ways to examine assumptions about NRC Framework-based science standards being synonymous to one another.

6.2. Future Research

This study draws attention to variations among NRC Framework-based science standards and encourages practitioners and researchers to think deeply about how nuanced and distinct differences may affect interpretation, usage, and eventually classroom implementation of standards. Further study is therefore recommended to assess how those differences affect sensemaking around the science standards. Given the distinctions among the NRC Framework-based science standards, it is likewise of interest to determine if corresponding differences exist in professional development, assessments, and curriculum among the states. Possibly, these elements are generally homogenous among all NRC Framework-based states, including NGSS states. If, however, a state’s science standards are used as a primary source when crafting professional development, assessments, and curriculum, there may be dissimilarities across states, even to the point of varied attention paid to the NRC Framework’s three dimensions of learning. Similar to the objectives of Goodman’s study [57], which found the quality of state math standards was associated with eighth-grade student math achievement, consequent investigations may then examine if variations in science standards are correlated with both instructional practices and student achievement.

Finally, the why question data from Study 2 provided a tangential finding of interest. A relatively high proportion of teachers indicated the standards evoked concerns about (a) students finding it challenging, (b) teachers lacking resources and time, and (c) at least part of the standard being unclear. On aggregate, 44.5% of Colorado, 34.2% of Oklahoma, and 29.4% of Wyoming responses were among these three categories. This high proportion of discordant reflections is a signal for educational leaders and professional development providers to address these first-order concerns if NRC Framework ideals are to be attained. This is especially true when considering teachers’ difficulty with the language of the standards juxtaposed with the NRC Framework’s recommendation for standards to use “language that is comprehensible to individuals who are not scientists” [6] (p. 299).

7. Conclusions

This study highlights differences among various science standards layouts that purportedly characterize the same NRC tenets. It shows that even seemingly similar science standards can draw teachers’ attention differently. In a sense, this work is a strong extension of Lopez and Goodridge incidentally noting differences of how K-12 science standards included engineering standards [29]. In going further, the work presented here provides means to better define what Valverde et al. referred to as “signatures” of K-12 science standards [37]. Returning to the assumption that prompted this investigation, i.e., science standards with roots in the NRC Framework are essentially all the same, it was found that NRC Framework-based science standards’ signatures have a good deal in common and the integration of the three dimensions of learning across the standards makes them connected, but they are not cohesive. Yet, in conclusion, it is not the intent to simply point to the findings to support the argument that there are indeed differences among NRC Framework-based science standards. Instead, it is important to advise how mindsets should shift and what can be done to accommodate the acknowledgement of differences of K-12 science standards across the United States. To this end, recommendations for practitioners, researchers, and policymakers are provided in the closing implications section.

Implications

Though this study highlighted several differences among U.S. sciences standards, no claims regarding what is “best” are made. Therefore, a first recommendation is for researchers to work in tandem with practitioners to establish means to assess the effect of design variations. This is especially important because possibly, and particularly in the absence of professional development and supporting materials, varied layouts of content standards lead not only to drawing attention differently but to diverse teaching practices.

Of course, even if empirical study reveals efficacious layouts, effecting change is not as simple as broadcasting results and anticipating quick revisions. Decisions regarding layout of science standards in the United States are tied to bureaucratic, organizational, and habitual practices—all of which can be difficult to alter within departments of education [58]. Consequently, much of the onus lies with addressing the formal procedures that exist for establishing K-12 content standards of all disciplines. For example, K-12 science standards developers may have little say regarding how standards are laid out if it is required that content standards from all disciplines in a state align to a preset template. Likewise, it may be difficult to convince individuals to shift away from customary and familiar layouts. Therefore, the recommendation for policymakers, particularly administrators in state education agencies, is to engage thoughtfully with considerations and research about layout and not necessarily adhere to instinct or to routine. Further, as Romney wrote, “visually designing documents can be a complicated, nuanced process requiring years of study and experience” [38] (p. 242); therefore, current practices of how and to whom the task of designing standards is assigned must be seriously considered.

Particular to researchers, it is suggested that they consider the presentation and style of science standards as a possible contributing variable when examining differences among states. National-level curriculum developers and professional development providers should also be attentive to the variations. Although three-dimensional learning concepts are part of all NRC Framework-based science standards, those concepts are presented in various ways and with varying depth across the United States. Therefore, teachers using their state’s science standards as a lens to decode three-dimensional learning may yield dissimilar interpretations. Since the release of NGSS in 2013, multiple professional development programs and curricular materials have been designed to support teachers and their students. Considering K-12 science standards across the United States are not homogenous, it is recommended to curriculum developers and professional development providers to be thoughtful about their audience and to consider if materials designed primarily with NGSS in mind need to be adapted for other NRC Framework-based states.