Diverging Paths: How German University Curricula Differ from Computing Education Guidelines

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

This manuscript details an examination of the ways in which Computer Science programs in Germany incorporate outcomes-based education approaches and frameworks into their student-facing program descriptions. Whilst it tackles some important questions about how programs might be viewed and valued, and how program design can be influenced by broader frameworks, in its current state these goals are unclear. It would benefit from a restructuring - that clearly progresses from the broader context of outcomes-based education and its adoption, to the German context, through to the research questions - and with more focus on the study goals and context. Much of the background historical information on learning outcomes and their use in higher education could be omitted in favour of a focused, critical overview, for example (this longer work may be better suited to a review article of OBE more generally).
In addition, some new questions arise during the course of the analyses, which are then examined in the results. Whilst I understand that this is common in complex analytical situations, any issues that are tackled in the analyses should be foregrounded and introduced earlier in the manuscript, to avoid any confusion. 

Overall, the manuscript appears to try to cover everything that has been uncovered as a result of the program analyses, and in the process has lost its core message. For instance, the RQs aim to detail a comparison of GI recommendations and institutional programs with respect to LOs - alignment, misalignment, and gaps. The findings and discussion are generally thorough and detailed, providing close analyses of learning outcome sets and a clear response to the RQs. The introductory material, and the manuscript overall, would benefit from major revision to tighten up the focus, structure and messages, so as not to dampen the impact of the findings. In addition, the secondary finding that the professors tasked with teaching/curriculum design may not have sufficient knowledge of frameworks or learning outcomes generally, could be further explored, and incorporated into the conclusions/recommendations for possible program improvements.
Please see the attached pdf for specific comments.  

 

Comments for author File: Comments.pdf

Author Response

The following sections contain: 

• Our rationale of the larger-scale changes, i.e. the restructuring of sections 1 & 2; 

• The comments from the reviewers in an itemized form as well as our response to said items. 

To be able to quickly identify our responses, we highlighted them in yellow. 

Overview over the Changes to the Paper 

Based on the comments of both reviewers, we felt that the article mainly needed a change to how the study was contextualized. Thus, most editing efforts went into reworking the Introduction and the Theoretical Background: 

• The introduction has been notably focused on the place of learning outcomes in the EU and German and, to a lesser extent, international higher education context. 
• In the introduction, the European developments in education have been related to the development in the US. In addition, a short paragraph about learning outcomes in different international accreditation units has been added. 
• The background section has now been focused on the GI recommendations. It only includes aspects that either are predecessors or are influencing the GI recommendations directly. 
• The description of the GI recommendations as well as their context in Germany has been extended. 
• The research questions have been moved to the Methods section to help contextualize the chosen methodology. To compensate, the Introduction now instead features a sort of “mission statement” for the article. 
• Text has been added which primes the secondary results discussed in the paper early on even though they are not a direct result of the RQs. In addition, discussion of the secondary findings was extended, and they are now included in the conclusion, embedding them holistically throughout the paper. 
• The paragraph introducing the Literature Survey as a chosen method has been notably expanded to motivate both the method and the data source.
• Figures 2-4 have been reworked in response to the review comments. This is most notable in Fig. 2. To stay consistent across the table, the figures, and the text, subject domain number (roman numerals) has been set as monotype font in all places. While we did not see any way to put the table through a major rework we introduced some column spacing to visually separate the “Total” column and the index columns (“Number” and “Subject Domain”) from the data columns. Additionally, the text size of the learning outcome examples has been (re-)set to normal size.

Comments on specific points of the Reviewers 

Review 1 

• This manuscript details an examination of the ways in which Computer Science programs in Germany incorporate outcomes-based education approaches and frameworks into their student-facing program descriptions. Whilst it tackles some important questions about how programs might be viewed and valued, and how program design can be influenced by broader frameworks, in its current state these goals are unclear. It would benefit from a restructuring - that clearly progresses from the broader context of outcomes-based education and its adoption, to the German context, through to the research questions - and with more focus on the study goals and context. Much of the background historical information on learning outcomes and their use in higher education could be omitted in favour of a focused, critical overview, for example (this longer work may be better suited to a review article of OBE more generally). 

Author comment: The introduction and background sections have been rewritten in large parts to streamline the flow of argument and remove some tangents, however interesting they might seem to us. We hope that the reviewer now finds these parts to be sufficiently succinct. 

• In addition, some new questions arise during the course of the analyses, which are then examined in the results. Whilst I understand that this is common in complex analytical situations, any issues that are tackled in the analyses should be foregrounded and introduced earlier in the manuscript, to avoid any confusion. 

Author comment: As part of the aforementioned revision, the secondary finding on issues with the writing on learning outcomes has been included in the paper’s purpose statement and in the introductory paragraphs of the method section. We hope that this way, they are primed enough to not be surprising in the results and discussion sections, even though they are not part of the primary research questions.  

• Overall, the manuscript appears to try to cover everything that has been uncovered as a result of the program analyses, and in the process has lost its core message. For instance, the RQs aim to detail a comparison of GI recommendations and institutional programs with respect to LOs - alignment, misalignment, and gaps. The findings and discussion are generally thorough and detailed, providing close analyses of learning outcome sets and a clear response to the RQs. The introductory material, and the manuscript overall, would benefit from major revision to tighten up the focus, structure and messages, so as not to dampen the impact of the findings. 

Author comment: The article, especially sections 1-3, has been restructured according to the comments. For details, see the section above. 

• In addition, the secondary finding that the professors tasked with teaching/curriculum design may not have sufficient knowledge of frameworks or learning outcomes generally, could be further explored, and incorporated into the conclusions/recommendations for possible program improvements. 

Author comment: We have included a new paragraph in the conclusion summarizing the secondary findings and including several measures to improve LO quality. In addition, we extended the discussion in Section 6.3. 

• Please see the attached pdf for specific comments. 

Author comment: The comments in the attached PDF files will be discussed in detail in the next section. 

Review 1 PDF comments 

• p. 1, l. 8 -- Do not start sentence with numeral 

Author comment: Restructured Sentence. 

• p. 1, l. 21 -- could be useful to briefly compare the European approach to others globally to situate the study and context 

Author comment: We included a reference which explicitly discusses the context of CBE in the US (and the usage of learning outcomes in accreditation internationally). 

• p. 1, l. 32 -- interesting context, with the relatively recent application of learning outcomes to courses for the purposes of accreditation/assurance of learning 

Author comment: Thank you for the warm words. 

• p. 2, l. 45 -- this whole section is interesting contextual background, but needs to be far more targeted and streamlined. It is unclear where the article is going at this point; I would recommend a restructuring (as in comment on p. 3) and to streamline the information to direct to the RQs and to highlight their relevance. As far as I can tell, the main points to include are: 
  • recommendations for curriculum design from GI
  • uptake is patchy and institutions may use other frameworks/guidelines as well 
  • important to have clear guidelines to ensure coherence
  • study seeks to examine...RQs 

Author comment: The position and relevance of the GI recommendations has been highlighted in the introduction. The details have all been moved to the theoretical background, which has been streamlined to focus on the GI recommendations.  

A new paragraph in the introduction leads to a sort of “mission statement” of the article which includes the motivational paragraph for curriculum design drawing from Annala et al. (2015). The specific research questions have been moved to the methods section to align with the other comments. 

• p. 2, l. 65 -- I question whether this is the best way to structure the article, with the RQs upfront prior to presenting the theoretical background. I encourage the authors to consider restructuring the introductory sections. 

Author comment: The RQs have been moved down into the methods section, putting them after the introduction and background sections. The introduction retains a brief presentation of the paper’s purpose. 

• p. 3, l. 77 -- Frame this instead with how your learning outcomes work sits within the broader global higher ed landscape - a new structure could be, for example:  

• • quick intro to context (e.g. this study seeks to examine how learning outcomes are incorporated into the national guidelines for computer science curriculum in Germany)
  • broad background on why, and how, learning outcomes are incorporated into higher ed curriculum globally (i.e. section 2) and overview of research
  • specific contextual factors and (i.e. much of the historical background in the section 1 introduction, which can be streamlined)
  • RQs, leading into methods 

Author comment: Was taken into consideration during the restructuring of the article (See above). Although the introduction is still on the longer side (first point made in the comment), we feel that this is justified since we included the global perspective there. 

• p. 4, l. 128 -- continue on sentence, or start new sentence with 'For example' (i.e. don't start with E.g.) 

Author comment: Used “For example”. 

• p. 5, l. 158 -- this is a key point of this section, to foreground the work within Germany and the context of the paper 

Author comment: Has been kept and integrated into the introduction during the restructuring of Section 1 & 2. 

• p. 5, l. 178 -- this is where some initial background (i.e. section 1) for the GI analysis should come in, for clarity. 

Author comment: Has been addressed by the restructuring of Section 1 & 2. 

• p. 7, l. 219 -- this is where we need to see the RQs so that the methods can follow on logically 

Author comment: We pulled the formulation of the RQs down from the introduction into the methods section, with the introduction now featuring a high-level purpose statement. 

• p. 7, l. 227 -- need to clarify here the intent of the analyses - to examine how programs are presented to students? To provide an objective summary of how programs assure learning? Would be helpful to revisit the RQs 

Author comment: We interpreted this comment to asking the question of “which point of view did we take during the study and why?”. The relevant paragraph has been extended to provide our point of view as well as reasons both for why we chose this specific search index and for why we chose to investigate module descriptions as a data basis. 

• p. 8, Figure 2 -- coloration of the figure is busy and distracting – simplify 

Author notes: The figure was transposed to allow presentation in a single (sub)plot. Additionally, the index number of the subject domains have been added. We removed the horizontal grid lines and the hatching of the orange bars. To increase readability of the bars we want to retain the vertical grid lines (cf. Few (2005)) to help gauge (relative) bar length. Furthermore, the grey background serves both the purpose of clearly delineating the figure content from the text while also wrapping the entire figure together as a visual whole (see ggplot documentation – the figures were created using the matplotlib ggplot style, the argument is analogous), thus following best practice (see the references in the ggplot documentation). 

Few, S. (2005). Grid Lines in Graphs are Rearely Useful. Perceptual Edge. Available: https://www.perceptualedge.com/articles/dmreview/grid_lines.pdf 
GGPlot Documentation (n.d.). 17.2 Complete themes. Available: https://ggplot2-book.org/themes.html#sec-themes 

• p. 9, Figure 3 -- recommend to simplify figure (e.g. can remove background gridlines and shading) 

Author comment: We removed the vertical gridlines. Additionally, we rescaled the figure a bit and added the index number to the subject domain labels. We are unsure what the reviewer meant with ‘shading’. We are seeing two possible interpretations: the plot background color, and the different colored bars. 

For the background color, see the notes for Figure 2. 

For the different bar colors: As stated in the figure legend, the grey bars carry a certain semantic which directly relates to the posed RQs. This dimension of meaning would be lost when coloring all bars the same way. We included a note on the bar colors in the figure caption (which we now realized had been missing) and hope that the reviewer agrees to this compromise. 

• p. 9, l. 299 -- whilst I understand that new questions can arise during the course of analyses, these issues should at least be foregrounded prior 

Author comment: Priming for the “flatness” analysis (the paragraph where this comment was placed) has been added to the method section. 

• p. 10, l. 313 -- (highlighted the word “professors”) 

Author comment: We assume this to mean that the term “professor” is seen as wrong here. A possible intended alternatives might be “lecturers”. However, in the German higher education system, it is almost always the case that a professor (which is both a position and a rank) is chiefly responsible for any given lecture, with their staff providing seminars or tutorials that accompany the lecture. Thus, as long as R1 does not explain what issue there is with the term, we intend to keep using it. 

• p. 23, l. 755 -- additional conclusions/recommendations around how LO creation should be directed or supported would be useful here, for example to counter the general lack of knolwedge from professors or other educators. 

Author comment: We have included a new paragraph in the conclusion summarizing the secondary findings and including several measures to improve LO quality. In addition, we extended the discussion in Section 6.3. 

Reviewer 2 Report

Comments and Suggestions for Authors

The paper addresses an important topic that is generally underexplored. A few papers out there focus on national curricula recommendations that go beyond the ACM/IEEE/BCS recommendations. As such, it’s great to see a paper that has a national/local focus. I enjoyed the methodological approach. The use of QCA was very well justified and nicely executed. It was also good to see how well the data collection and coding processes were carried out. Overall, I believe the paper contributes valuable empirical evidence relevant for accreditation, curriculum design, and educational policy.

 

It’s relatively easy to argue that the paper has a regional focus and, therefore, limited generalisability. I consider its regional focus, however, to be a strength. While I don't have an issue with the length of the paper, sometimes the details obscure its core arguments. I would suggest making some sections, such as the theoretical background, succinct. It would be interesting to see a more critical discussion of international frameworks rather than dismissing them as irrelevant. Some figures and tables are dense, making their interpretation difficult. Perhaps a more concise textual summary would be better.

Author Response

The following sections contains: 

• Our rationale of the larger-scale changes, i.e. the restructuring of sections 1 & 2; 

• The comments from the reviewers in an itemized form as well as our response to said items. 

To be able to quickly identify our responses, we highlighted them in yellow. 

Overview over the Changes to the Paper 

Based on the comments of both reviewers, we felt that the article mainly needed a change to how the study was contextualized. Thus, most editing efforts went into reworking the Introduction and the Theoretical Background: 

• The introduction has been notably focused on the place of learning outcomes in the EU and German and, to a lesser extent, international higher education context.
• In the introduction, the European developments in education have been related to the development in the US. In addition, a short paragraph about learning outcomes in different international accreditation units has been added.
• The background section has now been focused on the GI recommendations. It only includes aspects that either are predecessors or are influencing the GI recommendations directly.
• The description of the GI recommendations as well as their context in Germany has been extended. 
• The research questions have been moved to the Methods section to help contextualize the chosen methodology. To compensate, the Introduction now instead features a sort of “mission statement” for the article. 
• Text has been added which primes the secondary results discussed in the paper early on even though they are not a direct result of the RQs. In addition, discussion of the secondary findings was extended, and they are now included in the conclusion, embedding them holistically throughout the paper. 
• The paragraph introducing the Literature Survey as a chosen method has been notably expanded to motivate both the method and the data source. 
• Figures 2-4 have been reworked in response to the review comments. This is most notable in Fig. 2. To stay consistent across the table, the figures, and the text, subject domain number (roman numerals) has been set as monotype font in all places. While we did not see any way to put the table through a major rework we introduced some column spacing to visually separate the “Total” column and the index columns (“Number” and “Subject Domain”) from the data columns. Additionally, the text size of the learning outcome examples has been (re-)set to normal size.

Comments on specific points of the Reviewers 

Review 2 

• The paper addresses an important topic that is generally underexplored. A few papers out there focus on national curricula recommendations that go beyond the ACM/IEEE/BCS recommendations. As such, it’s great to see a paper that has a national/local focus. I enjoyed the methodological approach. The use of QCA was very well justified and nicely executed. It was also good to see how well the data collection and coding processes were carried out. Overall, I believe the paper contributes valuable empirical evidence relevant for accreditation, curriculum design, and educational policy. 

Author comment: Thank you for the kind words. 

• It’s relatively easy to argue that the paper has a regional focus and, therefore, limited generalisability. I consider its regional focus, however, to be a strength. 

Author comment: The discussion of the difference between the EU the US systems has been strengthened. While this does nothing to amend the regional focus of the paper it should facilitate prospective readers contextualizing the work even if they are from outside the EU. 

• While I don't have an issue with the length of the paper, sometimes the details obscure its core arguments. I would suggest making some sections, such as the theoretical background, succinct. 

Author comment: The introductory and background sections have been streamlined and shortened. The focus is now on the aspects strictly necessary to understand the GI model and the study. 

• It would be interesting to see a more critical discussion of international frameworks rather than dismissing them as irrelevant. 

Author comment: This is, unfortunately, beyond the scope of the study. The CS2023 are over 500 pages long; comparing them to the GI model (with 25 pages) would both be unbalanced (due to differences in granularity) and unduly superficial (as the comparison would not be the focus of either the background or the paper’s discussion). In addition, the GI decided to explicitly not align with other recommendations, thus such an inclusion would further distract from the focus of the paper. 
An undertaking like comparing different recommendations would most likely justify its own paper. 

• Some figures and tables are dense, making their interpretation difficult. Perhaps a more concise textual summary would be better. 

Author comment: We cleaned up the figures and table and improved labels & captions where possible. While this does not add more explanations, it does (hopefully) make figures easier to interpret. This was tested with different people. 

We tried to add text, but every time we did, we basically duplicated the text found elsewhere (mostly in the results). We therefore decided against adding more text. 

We are willing to implement more changes to improve the figures; however, without more detailed problem descriptions and with user testing showing no problems, we are not sure how to improve further over the current state of the paper. 

Round 2

Reviewer 1 Report

Comments and Suggestions for Authors

Thank you to the authors for the thorough revisions they have made to their paper in response to the reviewer reports. The clarity of the revised paper is greatly improved and the progression of ideas flows nicely. The ways in which the data and findings are displayed are also clear and coherent. 

There are some additional items that the authors appear to have included in response to my initial review but that now read as potentially apologetic or otherwise have been inserted as a reaction to my prior review. For example: 
- Lines 221-231 'Please note...' 'while discussing the minutiae...' and 'it should suffice to say...'
It would be great to see a more deliberate inclusion of these statements so that they are assured, and blend well within the paper sections; if the authors feel that my suggestions were inappropriate, for example, then I would not expect them to include phrases that are counter to their beliefs or findings.

I am happy to recommend publication once the revised manuscript with accepted changes is closely checked for coherence, clarity and accuracy.  I commend the authors on their significant work in both the study itself and in their revisions to the manuscript. 

Author Response

Dear Reviewer, 

thank you for taking the time to review our revised manuscript. Please find below our responses to your comments. As we did last time, our response will be highlighted yellow. 

• Thank you to the authors for the thorough revisions they have made to their paper in response to the reviewer reports. The clarity of the revised paper is greatly improved and the progression of ideas flows nicely. The ways in which the data and findings are displayed are also clear and coherent.  

Author comment: Thank you for the kind words. We are glad that you could appreciate the rework. 

• There are some additional items that the authors appear to have included in response to my initial review but that now read as potentially apologetic or otherwise have been inserted as a reaction to my prior review. For example:  
  - Lines 221-231 'Please note...' 'while discussing the minutiae...' and 'it should suffice to say...' 
  It would be great to see a more deliberate inclusion of these statements so that they are assured, and blend well within the paper sections; if the authors feel that my suggestions were inappropriate, for example, then I would not expect them to include phrases that are counter to their beliefs or findings. 

Author comment: The specific paragraph that was taken as an example had already been part of the first version of the manuscript. It had been moved down to where it is now as part of the streamlining effort. We moved the enumeration into a footnote to further smoothen the reading process and added some text to better highlight why we thought this paragraph was necessary. Rest assured that we would have tried to argue against a comment of yours if it were contrary to our beliefs or findings; any and all comments we took to heart, we did so because we agreed with them. 

• I am happy to recommend publication once the revised manuscript with accepted changes is closely checked for coherence, clarity and accuracy.  I commend the authors on their significant work in both the study itself and in their revisions to the manuscript. 

Author comment: We did a final pass and gave the manuscript to two other colleagues who did not read the manuscript beforehand with the explicit task to find inconsistencies. This resulted in some minor edits which we hope will aid with readability as well as the mentioned coherence, clarity and accuracy. We hope that the manuscript now also passes reviewer scrutiny. Please refer to the diff PDF which is provided in the source code ZIP archive for a version with highlighted edits.