What Are US Undergraduates Taught and What Have They Learned About US Continental Crust and Its Sedimentary Basins?

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

Dear Authors,

I appreciated your paper since the same problems we have in Italy with our first year university students (and obviously with the public and policymakers). [actually we also have problems with geology graduates!]

I have some comments/suggestions, whose use is not mandatory.

1) In Italy, introductory courses are based on phisical geography rather than on geology. Years ago some university faculties proposed a course named "introduction to earth science". Basically, it served to introduce the vocabulary and make known the various disciplines that characterise the earth sciences. The lack of the vocabulary is the main problem with students and even more so with the population. In my opinion this is not linked to the quality of books or that one of teachers. It corresponds to a generalized ignorance that corresponds to a lack of awareness on risks and natural resources and on the defense from the former and on the sustainable use of the latter. How to introduce some easy geological topics and a essential geological vocabulary from primary schools should be the priority.

 2) It would probably be better to ask the same questions at the beginning of the course and after its conclusion. This could suggest how useful the course was to improve geological knowledge in a broader sense.

3) I am not an expert but the number of about 50 students could be considered not a significant statistical sample. 

4) textbooks are influenced by the sensitivity (scientific background) of the authors. In Italy, most geophysics experts are physicists and not geologists and therefore even in this case the vocabulary (and the importance of the topics) is not the same for everyone. I teach basin analysis and textbooks are strongly unbalanced on different topics among themselves. In my opinion your 4 topics on which is based your research appear distant and unconnected to non-geologists, as first-year students should be considered, even if they have taken an introductory geology course. Even the textbooks could show a different approach to the topics, creating confusion among students.

About the text, I have a few suggestions

line 5 - add "US" in your address

line 31 - I prefer "human needs/use" to "civilization" (that means "human social and cultural development")

lines 50 and 51 - a) from what they..... (probably should be added "from" at the beginning of the phrase)

lines 50 to 53 - Formatting is required

line 123 - UTD (?). The acronym probably stands for "University of Texas at Dallas" but was not introduced before. Since it is only used once, it is better not to introduce it.

line 133 - extra credit (?). Please explain to non-US readers how your courses and exams are structured.

Lines 147-155. Well done! I agree! [but your work is multiplied! lines 169-171]

line 212. heterogeneity! see my comments on textbooks. Furthermore, in Italy physicists receive much more money for research than geologists. And geophysicist receive funding from both channels.

line 225 - "I" calculated... You are 2! "We" calculated ...

figure 2 - in the middle there is a number "1" which in my opinion is not needed. It is possible to center the legend to each rectangle. Why 3 lines for "student's geophisics mean"? try to graphically "standardize" your standard deviation [it's a joke]

line 619 - A point lacks after "to understand".

 

 

 

 

Author Response

Responses to reviewers, re: manuscript of “What are US undergraduates taught and what have they learned about US continental crust and its sedimentary basins?” by Clinton W. Crowley & Robert J. Stern. Our responses are in blue text, below:

July 9, 2025

Reviewer 1

Comments/suggestions

• Thank you! No changes required.
• This is an excellent suggestion, and we plan to use an approach like this in our future surveys.
• We agree that more students would produce more meaningful statistics. We think that a cross-section of students at several universities across the nation would be optimal. We compensated for our lack of resources needed to do this larger survey by changing from an easily-deployable multiple choice quiz to a more time-intensive interview process. This demonstrates our concern to get as much quality out of the limited numbers of students that we were able to access.
• We agree with this comment, but our purpose is to find out what students know, or not know, about these particular topics we believe to be critical for geoscience students to understand. Even if the topics were not taught or leaned in the courses, that information is valuable to the scientific community.

We appreciate the comments!

line 5 - add "US" in your address

Good idea--done

line 31 - I prefer "human needs/use" to "civilization" (that means "human social and cultural development")

Agreed, done

lines 50 and 51 - a) from what they..... (probably should be added "from" at the beginning of the phrase)

              Agreed, that makes the list consistent

lines 50 to 53 - Formatting is required

This will be taken care of in production

line 123 - UTD (?). The acronym probably stands for "University of Texas at Dallas" but was not introduced before. Since it is only used once, it is better not to introduce it.

              Good point, but I think it would be worthwhile to highlight that this study was done at only one school.

line 133 - extra credit (?). Please explain to non-US readers how your courses and exams are structured.

              Agreed, done

Lines 147-155. Well done! I agree! [but your work is multiplied! lines 169-171]

               We don’t see the repetition, so we will leave this as is

line 212. heterogeneity! see my comments on textbooks. Furthermore, in Italy physicists receive much more money for research than geologists. And geophysicist receive funding from both channels.

              No changes required

line 225 - "I" calculated... You are 2! "We" calculated ...

Good catch—change made.

figure 2 - in the middle there is a number "1" which in my opinion is not needed. It is possible to center the legend to each rectangle. Why 3 lines for "student's geophisics mean"? try to graphically "standardize" your standard deviation [it's a joke]

Also good catch—figure corrected and cleaned up

line 619 - A point lacks after "to understand".

Period added

 

We wish to thank our reviewers for their edits and suggestions, whose expert perspective has improved the manuscript.

Sincerely,

Clinton W. Crowley

Robert J. Stern

University of Texas at Dallas

 

Author Response File: Author Response.docx

Reviewer 2 Report

Comments and Suggestions for Authors

This is a timely and pedagogically valuable study addressing a critical gap in geoscience education. The research design is innovative, blending textbook analysis with direct student assessment to compare taught vs. learned curriculum. However, methodological limitations weaken the impact. With revisions, this could be a foundational study for geoscience education reform.

Recommendation: Major revision required.

1. Index Keyword Analysis. While innovative, using index keyword frequency as a proxy for taught curriculum is indirect and potentially misleading. Indexing varies by publisher, and the approach may not reflect depth of coverage or pedagogical emphasis. Suggested Improvement: Supplement the index-based method with chapter-level content analysis or textbook word-frequency analysis via full-text parsing (e.g., using NLP). Alternatively, perform a comparative thematic coding of selected chapters across textbooks.
2. Sample Representativeness: Student sample (n=51) is small, dominated by non-geoscience majors (98%), and limited to one institution. Suggestion: Replicate surveys across 5+ universities with stratified sampling (majors, course formats). Include pre/post-testing to measure knowledge growth.
3. Cognitive Load Proxy: Claims about cognitive load are inferred, not measured. Experiment Suggestion: Implement cognitive load surveys (e.g., NASA-TLX) during geophysics vs. geology lessons to quantify differences.
4. Statistical Rigor and Presentation. The manuscript relies on descriptive statistics (means and SDs). While helpful, the absence of inferential statistical analysis weakens the strength of conclusions. Suggested Improvement: Apply basic inferential tests to evaluate differences between topic categories, even with small sample sizes (e.g., Kruskal-Wallis test).
5.  Video Design Section. Unvalidated Assumptions: Threshold concepts are hypothesized but not tested. Experiment Suggestion: A/B test video versions (e.g., scaffolded vs. non-scaffolded) with 200+ students, measuring: Knowledge retention (post-tests).Cognitive load (self-report scales).Engagement (viewer analytics).

Author Response

Responses to reviewers, re: manuscript of “What are US undergraduates taught and what have they learned about US continental crust and its sedimentary basins?” by Clinton W. Crowley & Robert J. Stern. Our responses are in red text, below:

July 9, 2025

Reviewer 2:

As a reviewer, I genuinely struggle with evaluating this academic paper on American geology education. The authors surveyed six mainstream geology textbooks, analyzing teaching content through statistical frequency of index keywords, while assessing learning outcomes by interviewing 51 students. The study revealed that geophysics content constituted the highest proportion in textbooks (averaging 51%), yet students performed weakest in precisely this domain; conversely, geology content accounted for only 24% of textbooks but demonstrated the best student mastery.

This research innovatively substitutes textbook index analysis for classroom observation, employs open-ended questions to assess genuine comprehension rather than multiple-choice tests, and explicitly distinguishes between the concepts of "intended curriculum" and "attained curriculum." However, the small sample size (6 textbooks/51 students) may limit the generalizability of conclusions.

Using rigorous mixed methods (textbook textual analysis + in-depth student interviews), the study exposes a significant disconnect between "teaching" and "learning" in introductory geology education, particularly highlighting the pedagogical challenges of abstract topics (geophysics). The proposed ‌video-based threshold concepts strategy‌ offers actionable solutions for optimizing geoscience education, bearing universal reference value for STEM instructional design. Three minor issues warrant attention:

1.Excessive keywords‌, 3-5 core terms suffice;

Suggestion adopted, reduced keywords to 5

2.Untested intervention efficacy - subsequent validation requires controlled experiments to evaluate learning gains;

There was no intervention in this study, so there was no need to evaluate efficacy

3.Limited sample diversity - all participants came from UTD, with only 1 geology major; cross-institutional sampling expansion is needed.

We agree that a larger study involving more students across more universities would be better, but we feel our approach emphasizing quality over quantity has significant advantages.

Multiple-choice is more easily deployed, and could be implemented at multiple schools in the same semester. Our experience with a multiple-choice quiz did not help us know what students knew about the four geoscience concepts. We then adopted the open-ended written answer format, and asked students where they got their information. In-person interviews are time-intensive, averaging 30 minutes per participant including scheduling and the meeting itself. Compared to multiple-choice, we believe that open-ended questions and interviews, commonly used in geoscience concept surveys, produced much better insights into learned curriculum.

We realized that to meet with students at other universities, we would have to travel there and meet over several days at each location. This would require extending the study period over several semesters, or hiring more qualified interviewers. Given these considerations, and without external funding, we feel that our approach is the best way to use the resources already available to us.

4.The conclusion is somewhat redundant. It is recommended to simplify it.

We agree, and condensed the conclusions section to half its original length.

5.The graphics can be made more attractive.

We agree, and have improved figures 1-3 to match styles. We hope this is acceptable.

We wish to thank our reviewers for their edits and suggestions, whose expert perspective has improved the manuscript.

Sincerely,

Clinton W. Crowley

Robert J. Stern

University of Texas at Dallas

Author Response File: Author Response.docx

Reviewer 3 Report

Comments and Suggestions for Authors

As a reviewer, I genuinely struggle with evaluating this academic paper on American geology education. The authors surveyed six mainstream geology textbooks, analyzing teaching content through statistical frequency of index keywords, while assessing learning outcomes by interviewing 51 students. The study revealed that geophysics content constituted the highest proportion in textbooks (averaging 51%), yet students performed weakest in precisely this domain; conversely, geology content accounted for only 24% of textbooks but demonstrated the best student mastery.

This research innovatively substitutes textbook index analysis for classroom observation, employs open-ended questions to assess genuine comprehension rather than multiple-choice tests, and explicitly distinguishes between the concepts of "intended curriculum" and "attained curriculum." However, the small sample size (6 textbooks/51 students) may limit the generalizability of conclusions.

Using rigorous mixed methods (textbook textual analysis + in-depth student interviews), the study exposes a significant disconnect between "teaching" and "learning" in introductory geology education, particularly highlighting the pedagogical challenges of abstract topics (geophysics). The proposed ‌video-based threshold concepts strategy‌ offers actionable solutions for optimizing geoscience education, bearing universal reference value for STEM instructional design. Three minor issues warrant attention:

1.Excessive keywords‌, 3-5 core terms suffice;

2.Untested intervention efficacy - subsequent validation requires controlled experiments to evaluate learning gains;

3.Limited sample diversity - all participants came from UTD, with only 1 geology major; cross-institutional sampling expansion is needed.

4.The conclusion is somewhat redundant. It is recommended to simplify it.

5.The graphics can be made more attractive.

Author Response

Responses to reviewers, re: manuscript of “What are US undergraduates taught and what have they learned about US continental crust and its sedimentary basins?” by Clinton W. Crowley & Robert J. Stern. Our responses are in red text, below:

July 9, 2025

Reviewer 3:

Comments and Suggestions for Authors

This is a timely and pedagogically valuable study addressing a critical gap in geoscience education. The research design is innovative, blending textbook analysis with direct student assessment to compare taught vs. learned curriculum. However, methodological limitations weaken the impact. With revisions, this could be a foundational study for geoscience education reform.

We thank the reviewer for the complimentary comments.

Recommendation: Major revision required.

1. Index Keyword Analysis. While innovative, using index keyword frequency as a proxy for taught curriculum is indirect and potentially misleading. Indexing varies by publisher, and the approach may not reflect depth of coverage or pedagogical emphasis. Suggested Improvement: Supplement the index-based method with chapter-level content analysis or textbook word-frequency analysis via full-text parsing (e.g., using NLP). Alternatively, perform a comparative thematic coding of selected chapters across textbooks.

This is a good point, and we have added an entire paragraph to address this:

To what extent could our use of index terms as a proxy for taught curriculum be misleading? The possible reasons for this could be 1) taught curriculum has no relationship with textbook contents. 2) index terms vary greatly from textbook to textbook, and 3) emphasis of different topics in textbooks.  The first concern is the most important, but without being in the classes, we cannot evaluate this possibility. We can address the second and third concerns. Indexing standards are non-binding [19], but index terms vary only slightly by publisher; we believe using index terms as a proxy for taught curriculum is unlikely to be more than slightly misleading for that reason. The textbooks we surveyed all emphasized the same principal topics in introductory geology, so we do not think their indices are substantially different because of textbook content. Supplementary document 2 summarizes incidence of keyword by each topic of interest. Geophysics keywords appear most frequently in all six textbooks. Sedimentary basin keywords appear the least frequently in four of the six textbooks, tying for lowest frequency in the fifth textbook and is next to last in the sixth.  Radiometric dating keywords are mentioned more frequently than geology keywords in 2/3 of the textbooks, but neither topic approaches the sum of geophysics keywords.  For this reason, we think it is appropriate to treat the six textbooks as a group, as a way of accessing the taught curriculum for the topics we are interested in. Our method provides the “quick and dirty look” we need to guide our composition of videos on these topics. We feel our method gave us a enough understanding of taught curriculum to compare it meaningfully with learned curriculum as well.

We do not know how to do this kind of analysis. This seems to be an entire separate project, and we would have to learn far more than we know now to do justice to it.

2. Sample Representativeness: Student sample (n=51) is small, dominated by non-geoscience majors (98%), and limited to one institution. Suggestion: Replicate surveys across 5+ universities with stratified sampling (majors, course formats). Include pre/post-testing to measure knowledge growth.

We agree that the sample is small, we address this concern in our response to Reviewer 2’s point 3, copied here:

Reviewer 2, point 3. Limited sample diversity - all participants came from UTD, with only 1 geology major; cross-institutional sampling expansion is needed.

We agree that a larger study involving more students across more universities would be better, but we feel our approach emphasizing quality over quantity has significant advantages.

Multiple-choice is more easily deployed, and could be implemented at multiple schools in the same semester. Our experience with a multiple-choice quiz did not help us know what students knew about the four geoscience concepts. We then adopted the open-ended written answer format, and asked students where they got their information. In-person interviews are time-intensive, averaging 30 minutes per participant including scheduling and the meeting itself. Compared to multiple-choice, we believe that open-ended questions and interviews, commonly used in geoscience concept surveys, produced much better insights into learned curriculum.

We realized that to meet with students at other universities, we would have to travel there and meet over several days at each location. This would require extending the study period over several semesters, or hiring more qualified interviewers. Given these considerations, and without external funding, we feel that our approach is the best way to use the resources already available to us.

3. Cognitive Load Proxy: Claims about cognitive load are inferred, not measured. Experiment Suggestion: Implement cognitive load surveys (e.g., NASA-TLX) during geophysics vs. geology lessons to quantify differences.

We think that getting a better understanding of students’ cognitive load for each of the four topics is a good suggestion. The reviewer specifically mentioned the NASA Task Load Index (NASA-TLX) which we looked into:

The Official NASA Task Load Index (TLX) is a subjective workload assessment tool which allows users to perform subjective workload assessments on operator(s) working with various human-machine interface systems.

This technique, designed to measure cognitive load for astronauts’ tasks in a spacecraft, would need to be modified. To implement a cognitive load survey along the lines of NASA-TLX would require modifying the survey, then designing the class around a series of these surveys. Modifying and implementing the survey would be a significant and worthwhile project in its own right, but is far beyond the scope of our present effort.

4. Statistical Rigor and Presentation. The manuscript relies on descriptive statistics (means and SDs). While helpful, the absence of inferential statistical analysis weakens the strength of conclusions. Suggested Improvement: Apply basic inferential tests to evaluate differences between topic categories, even with small sample sizes (e.g., Kruskal-Wallis test).

We used an online calculator (https://www.statskingdom.com/kruskal-wallis-calculator.html) to test our keyword counts, with these results:

The Kruskal-Wallis H test indicated that there is a non-significant difference in the dependent variable between the different groups, χ²(5) = 4.39, p = .494, with a mean rank score of 13.5 for Group1 (Grotzinger Jordan), 15 for Group 2 (Marshak), 12.75 for Group 3 (Plummer Carlson Hammersley), 13 for Group 4 (Reynolds Johnson Morin Carter), 14.75 for Group 5 (Tarbuck Lutgens), 6 for Group 6 (Wicander Monroe).

We consulted with a statistical expert, who said that the results for both the Kruskal-Wallis H test and the ANOVA test showed no significant difference between the groups. The expert told us that having only 4 data points per group means we can’t claim statistical evidence supporting a claim that there is a difference between your groups. We added these results to the text.

5.  Video Design Section. Unvalidated Assumptions: Threshold concepts are hypothesized but not tested. Experiment Suggestion: A/B test video versions (e.g., scaffolded vs. non-scaffolded) with 200+ students, measuring: Knowledge retention (post-tests).Cognitive load (self-report scales).Engagement (viewer analytics).

We agree that this kind of test to quantify the usefulness of threshold concepts is important. For this study, no video intervention was used, and given the scope of our study, there is no way to test the effectiveness of threshold concepts. These suggestions will be useful as we consider how to plan the intervention and assess the effectiveness of the videos. No changes to the manuscript are required because the study’s aim was to assess the background state of student knowledge on our topics of interest.

We wish to thank our reviewers for their edits and suggestions, whose expert perspective has improved the manuscript.

Sincerely,

Clinton W. Crowley

Robert J. Stern

University of Texas at Dallas

 

Author Response File: Author Response.docx

Round 2

Reviewer 2 Report

Comments and Suggestions for Authors

This manuscript makes a substantial and original contribution to geoscience education research. It offers a robust critique of current instructional practices, introduces a creative method for curriculum analysis, and lays a well-theorized foundation for future instructional design interventions. The authors have responded constructively to reviewer concerns, strengthening the manuscript further.

I recommend acceptance with no further revisions.