What’s in a Grade? Teacher Candidates’ Experiences of Grading in Higher Education: A Phenomenographic Study

Round 1

Reviewer 1 Report

Abstract: I would suggest deleting or rephrasing the first lines, improve the internal syntactical relation between phrases. 

In conclusion to this section, we argue that the Bologna agreement has made an im-pact on the Swedish educational environments. We hold that some of the impact may be predominantly cosmetic, whereby new regulations around learning outcomes and assess-ment practices may have been re-worded but without significant changes in practice. ---- It is not clear if these are conclusions based on the sources cited or this if a hypothesis.

Further, we identify that many university teachers lack formal training in assessment and grading and we argue that teacher candidates’ experiences of formal education in general and assessment and grading in particular may impact their future practices. We argue that assessment and grading practices are connected to outcome-based education. For the rest of this paper however, we disconnect assessment and grading practices and focus on the latter. We do this to enable us to focus on students’ understanding of grading practices. ---- At the section where this affirmation appears it's not clear if it is based on authors' experiences or theoretic sources. 

Section 5: adding the complete list of interview questions would be great. Observation: basing the study about grading system in Sweden on 1 university only with 13 interviews does not seem an adequate method. The authors include this in limitations of the study. I miss a strong argumentation about why only 1 university was explored. In my opinion, this is the weakest point of the study, although generally the originality and research aims are great. 

Section 6: The table with participants characteristics (language, age, etc. would be helpful, not only the range).

Results section: The extracts from the interviews visually are disconnected from the authors' comment.

Author Response

Dear editor and reviewer,

We would like to thank the editor and reviewers for the insightful comments on the paper. A point-by-point response to the reviewer concerns can be found below. We have revised the entire document in order to achieve better consistency and readability.  We hope the final text now fulfils the expectations from the editor and reviewers. Please note this text contains response to both reviewers as there is some overlap. 

Reviewer: Abstract: I would suggest deleting or rephrasing the first lines, improve the internal syntactical relation between phrases. 

Comment: We agree. The abstract has been revised to make a more direct comment about our study.

Reviewer: In conclusion to this section, we argue that the Bologna agreement has made an im-pact on the Swedish educational environments. We hold that some of the impact may be predominantly cosmetic, whereby new regulations around learning outcomes and assess-ment practices may have been re-worded but without significant changes in practice. ---- It is not clear if these are conclusions based on the sources cited or this if a hypothesis.

Further, we identify that many university teachers lack formal training in assessment and grading and we argue that teacher candidates’ experiences of formal education in general and assessment and grading in particular may impact their future practices. We argue that assessment and grading practices are connected to outcome-based education. For the rest of this paper however, we disconnect assessment and grading practices and focus on the latter. We do this to enable us to focus on students’ understanding of grading practices. ---- At the section where this affirmation appears it's not clear if it is based on authors' experiences or theoretic sources. 

Comment: We have modified this section to acknowledge that these notions are based on our experiences as teacher educators and in doing so wish to tone down any potential generalizations. The section now reads: In conclusion to this section, we argue, based on our experiences that the Bologna agreement may have had implications on the Swedish higher education environments. We hold that some of the impact may be predominantly cosmetic, whereby new regulations around learning outcomes and assessment practices may have been re-worded but without significant changes in practice.  

Moreover, we have decided to omit the section that reads: Further, we identify that many university teachers lack formal training in assessment and grading and we argue that teacher candidates’ experiences of formal education in general and assessment and grading in particular may impact their future practices. We made this decision to better balance the claims we are making in our study and tone down the risk of making broader generalizations.

Reviewer:

• Section 5: adding the complete list of interview questions would be great.

Comment: We have now added the list of questions used as a supplementary file.

Reviewer Observation: basing the study about grading system in Sweden on 1 university only with 13 interviews does not seem an adequate method. The authors include this in limitations of the study. I miss a strong argumentation about why only 1 university was explored. In my opinion, this is the weakest point of the study, although generally the originality and research aims are great. 

Comment:  We acknowledge that our site constitutes a limited sample, but it is among the first done focusing on these students and with a focus on their experience of grades and grade descriptors. We see this as the first step in a process and have already begun the work in approaching other teacher education sites in order to do comparative work. We acknowledge the shortcoming in the limitations section and write: This study focuses on one HE institution which is a limitation, but we see this as the first in a number of studies examining teacher candidates experiences of grading and grade descriptors. By choosing one site we could have a better understanding of the context the respondents shared. We believe that this field needs close case-study like examination, and more comparative work.

Reviewer: Section 6: The table with participants characteristics (language, age, etc. would be helpful, not only the range).

Comment: We have added an abbreviated table that contains more detailed information, but without any risk of disclosing the identities of the respondents.

Reviewer: Results section: The extracts from the interviews visually are disconnected from the authors' comment.

Comment: We have revised the results section.

Reviewer: The authors report findings from a phenomenographic study of 13 teacher education candidates, who were interviewed to identify their perceptions and experiences of grading in their academic program. Responses were organized in four broad categories: use of grades as a means of articulating identity as a student, grades as motivational tools, interpretation and personal understanding of the grading process, and grading as a feedback mechanism for acculturating students into the academic community (developing shared values in the discipline).

The sample seems to be very small (n = 13) to justify making broad generalizations about student perceptions of grading. The authors do not discuss the degree to which this sample is representative of candidates enrolled in teacher education programs. Please include a discussion about the adequacy of a sample of this size for the conclusions drawn.

Comment:  Thank you. In the limitations section we have elaborated further on the limitation in having only one site (university). In the context we describe the: The interview respondents were chosen using a purposeful sampling procedure (Patton, 1990). Given the focus of describing respondent’s variation respondents with different ages and potential experiences were sought after.

Reviewer: Please define “academization of teacher training.” The authors argue that this is an important driver for current trends in grading but do not clearly articulate what they mean by this term (introduced on page 1 and referred to elsewhere throughout the submission). How does “academization” differ from the Bologna process?

I did not see a clear relation between the discussion of learning outcomes (page 2) and the questions posed about student understanding of grading. Please make this more explicit. How does this discussion support the authors’ rationale for conducting the research they report?

Comment: . Previously teacher training in Sweden was offered under the pre-text of vocational training with a focus on teaching methodologies. A move towards a more academic approach to teaching may bring with it certain consequences. We have now attempted to tighten up what we mean in this section by adding the following sentence: Such a thrust towards the academization may come at the expense of a previous focus on teaching methodology and vocational approaches.

When describing the Bologna agreement and its impacts learning outcomes have come to play an important role. This has now been highlighted in the text. Our understanding and intention have been to relay that Bolonga may, to some extent, have brought on more instrumental ways of looking at higher education governance. We added the following text. One of the major impacts of the implementation of Bologna in the Swedish context was the introduction of such learning outcomes.

Reviewer: In the discussion (final paragraph of discussion on page 11), please provide a concrete example that illustrates how grading should function to acculturate teacher candidates to the discipline.

Comment: We have added a note in this section to bring the reader’s attention to how descriptors could be used to create an opportunity for dialogue.

Reviewer: A major (and disturbing) finding is the frequent reference to the degree to which respondents devalued grades because teachers are in such short supply and all teacher candidates can expect to earn a certificate and (presumably) find placement in a school regardless of the grades earned (and, I assume, competencies attained) during their training. This strikes me as a serious criticism of the state of professional preparation of new teachers. What are the implications of these findings for the likely quality of graduates as future teachers?

Comment: Thank you. We agree with this sentiment, but choose on the side of caution, and do not wish to speculate on this matter. We have however, modified the conclusion to add a concern, and a reflection. The conclusion now begins with: This study demonstrates teacher candidates’ understanding of grading practices in higher education, suggesting that while teacher candidates accept the existing grading system, they have difficulty in explaining its finer points. As we point out in the limitations section, this paper has a small sample and caution should be applied when considering the results. However, as teacher educators ourselves, we are concerned that student candidates are, at times, unable to discern the difference between a different grade descriptors and grades, and fear that might have an impact on their own future practice as teachers.

Reviewer: Some technical problems: My understanding is that the numbers for references indicate the order of citation in text, but this convention is not followed in this submission. The numbers have no intrinsic meaning. The references are (mostly) in alphabetical order (a few late additions appear at the end of the list, which made the task of checking reference citations difficult). Still, some cited works appear to be missing and the list appears to include works that are never cited in the text.

It is only necessary to cite the authors of an idea once in a sentence. The authors managed to cite three times in one sentence (page 2, first paragraph under “Drivers of learning”).

Comment: Thank you for pointing this out. We have used APA guidelines to revise the references, and we have omitted superfluous references. As per APA we present the references in the text and then alphabetically in the document.

Reviewer 2 Report

Please see my comments in the attached file.

Comments for author File: Comments.pdf

Author Response

Dear editor and reviewer,

We would like to thank the editor and reviewers for the insightful comments on the paper. A point-by-point response to the reviewer concerns can be found below. We have revised the entire document in order to achieve better consistency and readability.  We hope the final text now fulfils the expectations from the editor and reviewers. Please note this text contains response to both reviewers as there is some overlap. 

Reviewer: Abstract: I would suggest deleting or rephrasing the first lines, improve the internal syntactical relation between phrases. 

Comment: We agree. The abstract has been revised to make a more direct comment about our study.

Reviewer: In conclusion to this section, we argue that the Bologna agreement has made an im-pact on the Swedish educational environments. We hold that some of the impact may be predominantly cosmetic, whereby new regulations around learning outcomes and assess-ment practices may have been re-worded but without significant changes in practice. ---- It is not clear if these are conclusions based on the sources cited or this if a hypothesis.

Further, we identify that many university teachers lack formal training in assessment and grading and we argue that teacher candidates’ experiences of formal education in general and assessment and grading in particular may impact their future practices. We argue that assessment and grading practices are connected to outcome-based education. For the rest of this paper however, we disconnect assessment and grading practices and focus on the latter. We do this to enable us to focus on students’ understanding of grading practices. ---- At the section where this affirmation appears it's not clear if it is based on authors' experiences or theoretic sources. 

Comment: We have modified this section to acknowledge that these notions are based on our experiences as teacher educators and in doing so wish to tone down any potential generalizations. The section now reads: In conclusion to this section, we argue, based on our experiences that the Bologna agreement may have had implications on the Swedish higher education environments. We hold that some of the impact may be predominantly cosmetic, whereby new regulations around learning outcomes and assessment practices may have been re-worded but without significant changes in practice.  

Moreover, we have decided to omit the section that reads: Further, we identify that many university teachers lack formal training in assessment and grading and we argue that teacher candidates’ experiences of formal education in general and assessment and grading in particular may impact their future practices. We made this decision to better balance the claims we are making in our study and tone down the risk of making broader generalizations.

Reviewer:

• Section 5: adding the complete list of interview questions would be great.

Comment: We have now added the list of questions used as a supplementary file.

Reviewer Observation: basing the study about grading system in Sweden on 1 university only with 13 interviews does not seem an adequate method. The authors include this in limitations of the study. I miss a strong argumentation about why only 1 university was explored. In my opinion, this is the weakest point of the study, although generally the originality and research aims are great. 

Comment:  We acknowledge that our site constitutes a limited sample, but it is among the first done focusing on these students and with a focus on their experience of grades and grade descriptors. We see this as the first step in a process and have already begun the work in approaching other teacher education sites in order to do comparative work. We acknowledge the shortcoming in the limitations section and write: This study focuses on one HE institution which is a limitation, but we see this as the first in a number of studies examining teacher candidates experiences of grading and grade descriptors. By choosing one site we could have a better understanding of the context the respondents shared. We believe that this field needs close case-study like examination, and more comparative work.

Reviewer: Section 6: The table with participants characteristics (language, age, etc. would be helpful, not only the range).

Comment: We have added an abbreviated table that contains more detailed information, but without any risk of disclosing the identities of the respondents.

Reviewer: Results section: The extracts from the interviews visually are disconnected from the authors' comment.

Comment: We have revised the results section.

Reviewer: The authors report findings from a phenomenographic study of 13 teacher education candidates, who were interviewed to identify their perceptions and experiences of grading in their academic program. Responses were organized in four broad categories: use of grades as a means of articulating identity as a student, grades as motivational tools, interpretation and personal understanding of the grading process, and grading as a feedback mechanism for acculturating students into the academic community (developing shared values in the discipline).

The sample seems to be very small (n = 13) to justify making broad generalizations about student perceptions of grading. The authors do not discuss the degree to which this sample is representative of candidates enrolled in teacher education programs. Please include a discussion about the adequacy of a sample of this size for the conclusions drawn.

Comment:  Thank you. In the limitations section we have elaborated further on the limitation in having only one site (university). In the context we describe the: The interview respondents were chosen using a purposeful sampling procedure (Patton, 1990). Given the focus of describing respondent’s variation respondents with different ages and potential experiences were sought after.

Reviewer: Please define “academization of teacher training.” The authors argue that this is an important driver for current trends in grading but do not clearly articulate what they mean by this term (introduced on page 1 and referred to elsewhere throughout the submission). How does “academization” differ from the Bologna process?

I did not see a clear relation between the discussion of learning outcomes (page 2) and the questions posed about student understanding of grading. Please make this more explicit. How does this discussion support the authors’ rationale for conducting the research they report?

Comment: . Previously teacher training in Sweden was offered under the pre-text of vocational training with a focus on teaching methodologies. A move towards a more academic approach to teaching may bring with it certain consequences. We have now attempted to tighten up what we mean in this section by adding the following sentence: Such a thrust towards the academization may come at the expense of a previous focus on teaching methodology and vocational approaches.

When describing the Bologna agreement and its impacts learning outcomes have come to play an important role. This has now been highlighted in the text. Our understanding and intention have been to relay that Bolonga may, to some extent, have brought on more instrumental ways of looking at higher education governance. We added the following text. One of the major impacts of the implementation of Bologna in the Swedish context was the introduction of such learning outcomes.

Reviewer: In the discussion (final paragraph of discussion on page 11), please provide a concrete example that illustrates how grading should function to acculturate teacher candidates to the discipline.

Comment: We have added a note in this section to bring the reader’s attention to how descriptors could be used to create an opportunity for dialogue.

Reviewer: A major (and disturbing) finding is the frequent reference to the degree to which respondents devalued grades because teachers are in such short supply and all teacher candidates can expect to earn a certificate and (presumably) find placement in a school regardless of the grades earned (and, I assume, competencies attained) during their training. This strikes me as a serious criticism of the state of professional preparation of new teachers. What are the implications of these findings for the likely quality of graduates as future teachers?

Comment: Thank you. We agree with this sentiment, but choose on the side of caution, and do not wish to speculate on this matter. We have however, modified the conclusion to add a concern, and a reflection. The conclusion now begins with: This study demonstrates teacher candidates’ understanding of grading practices in higher education, suggesting that while teacher candidates accept the existing grading system, they have difficulty in explaining its finer points. As we point out in the limitations section, this paper has a small sample and caution should be applied when considering the results. However, as teacher educators ourselves, we are concerned that student candidates are, at times, unable to discern the difference between a different grade descriptors and grades, and fear that might have an impact on their own future practice as teachers.

Reviewer: Some technical problems: My understanding is that the numbers for references indicate the order of citation in text, but this convention is not followed in this submission. The numbers have no intrinsic meaning. The references are (mostly) in alphabetical order (a few late additions appear at the end of the list, which made the task of checking reference citations difficult). Still, some cited works appear to be missing and the list appears to include works that are never cited in the text.

It is only necessary to cite the authors of an idea once in a sentence. The authors managed to cite three times in one sentence (page 2, first paragraph under “Drivers of learning”).

Comment: Thank you for pointing this out. We have used APA guidelines to revise the references, and we have omitted superfluous references. As per APA we present the references in the text and then alphabetically in the document.

Round 2

Reviewer 1 Report

Dear author/s,

Thank you for making the changes, everything seem to be in order now. 

Author Response

Thank you

Reviewer 2 Report

Please see uploaded file

Comments for author File: Comments.pdf

Author Response

 Review 1296187

What’s in a grade? - Resubmission

Reviewer:The author has made some minor changes, which largely fail to address the questions I raised.

 I asked about the small sample size. The author acknowledges this as a limitation (page 12) but did not address my question about whether the size was adequate to support the conclusions drawn. Nor did the author discuss the degree to which the sample represents the population of teacher education candidates enrolled in the institution’s program. Instead, the author described a sampling approach intended to maximize the diversity of responses. I do not see the value of maximizing diversity of responses if the underlying population would not be accurately characterized as highly variable in the perceptions of grading requested in the question prompts.

Comment: Thank you. We apologize for this oversight. The sample size is in the range of what is expected in phenomenographicv studies, and we have clarified that further, by noting: “It was probable that qualitative differences in conceptions could be expressed in this sample and it is common that phenomenographic studies contain, in the region of 10-15 respondents (Marton, 1981). Other studies will need to be done in order to validate or contradict the findings presented here.”

We also acknowledge that it would be unreasonable to claim we have resolved the research question exhaustively, and we call upon other researchers to review and contextualise our findings.

Reviewer: The author added language about “academization” but did not provide a clear definition of this term, as I requested.

Comment: Academization in our texts acts as a background, and previously we wrote that academization could mean a move away from vocational focus, “Such a thrust towards the academization may come at the expense of a previous focus on teaching methodology and vocational approaches.”, but realise that this is insufficient. We have now modified the text, adding: “Academisation in this sense involves a move towards academic values and in particular conducting academic research, even for students who will not go on to become researchers, which, may lead to a loss of connections to professional practice(Magnell & Geschwind, 2019).” to acknowledge what is understood more broadly as Academisation and hope that these two notions will focus the readers’ attention on what we mean by the concept.

Reviewer: I requested a discussion of the implications of the many respondents who devalued grades, provided the “got by” to receive a professional credential, and the general perception that grades represented a hoop to be jumped through rather an evidence of attainment of a valued professional skill.

Comment: We agree this needs further examination and discussion. We have made three comments in the discussion and conclusion section, highlighting this as troublesome, calling on future work, and reflecting on the implications. We ask you to consult this revised section(s). We are careful, however not to generalise based on these findings.

Reviewer: The author added a missing reference but the reference list continues to be out of order. (This is largely a copy editing issue, but I thought I would point it out again.)

Comment: We have reviewed the text and edited the references, but do not see the errors you identify. Should you find the text satisfactory, we hope the copy editor might syphon these out. We humbly apologize.

Round 3

Reviewer 2 Report

Thank you for addressing my questions. The new material on "academization" now explicitly describes the author's intended meaning for this term.

The author also clarified expectations for sample sizes for this type of research (thanks for adding a citation) and has appropriately added qualifiers about over-generalization of findings from a small sample.

I appreciate the additional material related to the interpretation of the findings (re: the value and meaning of grades for students).

The final two references (# 36 and #37) are out of alphabetical order. These can easily be corrected in copy editing.

Author Response

Thank you for your comments and positive response. The last comment: 

The final two references (# 36 and #37) are out of alphabetical order. These can easily be corrected in copy editing.

Is one we cannot change as this follows the Swedish language convention where Å precedes Ö. We thank you for your understanding.