Quality of Feedback in Higher Education: A Review of Literature

: In raising the standards for professional educators, higher educators must be prepared to provide the highest quality feedback on student performance and work products toward improved outcomes. This review of the literature examined the major ﬁndings of 70 quantitative, mixed methods, or qualitative studies found in higher education journals across a range of disciplines. Multiple recommendations and results for feedback emerged which fall into the categories described by Susan Brookhart. This review found research for each of Brookhart’s categories, with results indicating di ﬀ erences between the perceptions of adherence to sound feedback practices versus the reality of implementation, the potential for innovative tool use, and a disagreement about the e ﬀ ectiveness of peers for providing e ﬀ ective feedback. Indicators for quality within the research conﬁrmed the importance of commonly accepted standards such as positivity, speciﬁcity, timeliness, and encouraging active student participation. Additionally, trends and themes indicated a need for the consistent implementation of the feedback exchange process and ﬂexibility to account for student input / preferences. Greater consistency toward the application of these quality indicators should be undertaken when determining the quality of higher education feedback for preservice teachers prior to undertaking summative licensure assessments.


Introduction
As with public schools, institutions of higher education are facing increased accountability pressures in recent years to assess student performance [1]. For educator preparation programs (EPPs; i.e., institutions of higher education designed to prepare candidates for teaching across the U.S.), this pressure is particularly acute [2,3]. Nationally, summative assessments such as performance-based assessments (e.g., validated tasks including demonstrations of teacher-candidate knowledge and skills such as edTPA or PPAT) and licensure exams are being used more frequently to ascertain teacher-candidate effectiveness prior to a licensure recommendation [4,5]. The high-stakes nature of these assessments has caused many EPPs to create formative activities in coursework that are aligned with these summative assessments, allowing candidates to build their capacity toward the successful completion of these assessments throughout a program of study [6,7]. These formative activities (such as planning differentiated lessons or analyzing student data) are embedded into course assignments and provide faculty opportunities to give candidates extensive feedback on their performance with enough time to adjust instruction based on the results. Given that passing scores may be required on the summative assessments to meet licensure requirements, the quality of faculty feedback given to candidates on formative assignments can be critical in assisting candidates with improving their own 1.
Are features of quality feedback verified in the higher education current literature? 2.
What are the perceptions of faculty and students in higher education regarding what is considered quality feedback?
The Methods outline the parameters for the review, the inclusion and exclusion criteria, and descriptors of the codes. The results are organized by research design and codes. The discussion highlights the overall major findings within the review.

Methods
The parameters for conducting this literature review included a targeted review of materials that fell within a specific time frame. The focus of the current review considered literature on feedback published between 2000 and 2018, which included updated frameworks for feedback development and higher standards for empirical evidence. Readily available electronic databases (considered standards in the U.S.) were searched as the main source of identifying appropriate information (e.g., Academic Search Complete, Proquest, Sage Complete, JSTOR, WorldCat.org, EBSCO Host, Ed Research Complete, and Google search/scholar for web resources). Search terms generally included the words higher education with combinations of the words feedback, assessment, and evaluation along with various descriptors like quality, timely, task focused, analytic, and descriptive. Literature were also identified by manual searches of higher education-focused journals (e.g., Assessment and Evaluation in Higher Education) and tree searches of the reference sections of previously identified literature.
Articles were reviewed for alignment with the following inclusion and exclusion criteria. Including theoretical, conceptual, and data-based works (both qualitative and quantitative), the authors of this review sorted the collection of literature into three categories for separate analyses: conceptual foundations, quantitative, and qualitative. All work included in the conceptual foundations was used to inform the literature search terms and categories. Only literature that conducted a methodological study was included in the results of the literature review. The literature review materials included journal articles, conference papers, and dissertations. Other materials such as books, book chapters, and websites were also considered in the development of the quality feedback coding measure as well as in the development of the search term list for the literature review.

•
Included feedback provided to students in higher education (e.g., a university or community college).

•
Addressed either the content of the feedback provided or the format of feedback (delivery mechanisms).
Exclusion criteria: • Articles prior to 2000. • Studies including participants from grade school or post-school employment.

•
Articles which fail to clearly describe the focus of study as the content and format of feedback.
Using the existing literature about the features of quality feedback, a coding form was created to identify the categories within each article (see Table 1). Brookhart [12] was written for practicing classroom teachers, not necessarily faculty in higher education. The content, however, was deemed to be applicable for both young children and adult learners. Therefore, the content was adapted through the lens of higher education. Not all codes outlined within Brookhart [12] were included in the study. For example, within Function, the code of "judgement" was not included in this review, as the research literature typically did not measure "judgement" as a purpose of providing feedback within a higher education setting. Considerations of student interactions with feedback (e.g., Carless and Boud [14]), are taken into account through the code of Tone. All articles were coded using the same codes regardless of the methodology across studies. A number of articles (n = 7, 10%) with representation from each methodology category included were double coded by the first two authors to establish reliability of coding. Operational definitions were created to increase the reliability of coding across articles (see Table 1). Additionally, qualitative studies included a narrative findings summary for themes by category captured in an excel file. The percentage of agreement for the categories ranged from 91% to 100%.

Results
The initial review resulted in 17 quantitative, 11 mixed methods studies, and 50 qualitative studies (see Supplemental Materials). After a closer examination of these studies, 10 did not meet the inclusion criteria. Thus, the final analysis included 15 (21.4%) quantitative, 11 (15.7%) mixed methods, and 44 (62.9%) qualitative studies for a total of 70 studies. Of these accepted studies, the majority were conducted or published internationally (outside of the United States). These studies were coded for their characteristic features (e.g., participants, design, results), data collection descriptions (e.g., timing, mode, audience) and for explicit outcomes from the analyses. For quantitative studies, coding was conducted to examine the presence/absence or type of various feedback qualities including the feedback's focus, means of comparison, function, valence, and quality (clarity, specificity, and tone). For the qualitative studies, a findings summary was completed and a data reduction process was conducted. Mixed methods studies were included in both processes for their relevant data.

Characteristics of Current Literature
To begin the analyses, the authors noted descriptive information for each study (i.e., who provided the feedback; the mode in which that feedback was provided; methodological designs). Of the included works, the majority of the research focused on the instructor to student feedback (61%), but some literature also focused on peer to student feedback (18%) with two comparisons between instructors and peers. Various methodologies for the mode of feedback were noted, including evaluations of innovative technology applications (e.g., audio feedback by Lunt and Curan [15]; Macgregor, Spiers, and Taylor [16]), while most focused on written feedback (e.g., Kisnanto [17]). Blog, vlog, and virtual feedback were also represented, although viewed as part of a new wave of applications for technology education (e.g., Williams and Jacobs [18]; Xie, Ke, and Sharma [19]). Tutor to student feedback was also present (17%) as was automated feedback (4%). The quantitative designs were varied between experimental (e.g., group = 22.3%) and non-experimental (e.g., survey = 12%), with few examples relying on other means (e.g., action research, n = 1). Every study examined at least one of either the identified strategies or the content for effective feedback. Across all qualitative studies examined, none included an explicit description of all feedback features, but each study included at least one of the characteristic features.

Focus of the Studied Feedback
Using the levels feedback focus outlined by Hattie and Timperley [20], the included studies were sorted into those which focused the task (defined here as the completion of work tasks or content of the work), the process of the task (defined here as the process for producing the work), the self-regulation of student activities (defined here as student behaviors associated with the work), or the students themselves (defined here as directly addressing student characteristics). While many included studies did not describe the feedback in sufficient detail to be coded for this feature (n = 53), the majority of those that could focused on the work or the content of the work (19.4%, n = 13). None of the included studies focused solely on the process, although an additional four studies focused on both content and process (e.g., Van der berg, Admiral, and Pilot [21]). Few studies had a focus on student behavior (e.g., Minehart, Rudolph, Pian-Smith, and Raemer [22]) and only one study was found which mentioned feedback that included characteristics and descriptors of the students themselves (e.g., "Your willingness to change your tone has improved!", Herbert and Vorauer [23]).

Means of Comparison and Feedback Function
A description of the means for comparison was rarely included (9%), but when it was, the feedback were criterion referenced as opposed to norm-referenced. Although the feedback function was typically not explicitly stated, descriptions of the feedback were sufficient to identify whether the feedback was descriptive (providing advice without implications for or a connected to a grade) or evaluative (with implications for or connected to a grade). The majority of the qualitative studies examined evaluative feedback (19.4% of all included studies), while descriptive was included for seven others, six of which were provided in conjunction with evaluative feedback.

Valence and Tone
The valence of the feedback was coded to indicate the frequency of positive versus negative feedback in the literature. While the majority of the qualitative studies did not mention or provide a sufficient description of the valence (noted for 28.8%), examinations of both positive and negative were compared (e.g., Plakht, Shiyovich, Nusbaum, and Raizer [24]) or noted in student opinions of feedback effectiveness (e.g., McCarthy [25]). Tone was also coded by identifying feedback designed for active participation versus passive participation by the learner. The vast majority of the included studies did not include the relevant information for this code (with only 3% identified), but those that did described the detriments of passive learning feedback from computer-generated feedback (e.g., Herbert and Vorauer [23]) and peer provided feedback (e.g., Wilkins, Shin, and Ainsworth [26]).

Understandable and Actionable
Similarly, only six studies evaluated evidence for understandable or actionable qualities (e.g., Herbert and Vorauer [23], Lunt and Curan, [15]). All studies that mentioned this characteristic included a description of the presence of vague feedback that was less helpful in applying suggestions and making impacts on future student work.

Specificity
Only five studies evaluated evidence for specificity (e.g., Lunt and Curran [15]) with two examining the difference between vague vs. specific comments that promotes the next steps for students (e.g., Wilkins, Shin, and Ainsworth [26]). In the current literature, there is a lack of evidence to sufficiently describe how specific feedback in higher education has a causal relationship with positive results. The limited evidence that was found, however, consistently suggests that specificity is required for quality so that the student is able to understand how to proceed. Unfortunately, the feedback for higher education students included in the studies that examined this tended to be vague, and thus less impactful and effective.

Qualitative Themes in Current Literature
A qualitative analysis of qualitative literature supported the quantitative summary based on various aspects of effective (and ineffective) feedback and feedback practices. Within the 41 qualitative studies, evaluations of feedback delivery (timeliness, tone, valence; n = 23), feedback forms/mediums/modes (n = 25), feedback content (n = 14), and feedback functions or purposes (n = 9) were the most frequent foci. Additionally, many of the qualitative studies of feedback within high education examined various failings or barriers to quality feedback (n = 22) as well as the technology used or required to support quality feedback (n = 18). Of these initial codes, several themes (described below) emerged from the findings.

Feedback Delivery
Feedback delivery examinations described influences on how the message was received (e.g., Huxham [27]). This included factors like feedback timeliness, tone, and valence. Timeliness was a particularly prominent theme across many studies (e.g., Lynam and Cachia [28]), many times with participants noting that timely feedback was more likely to be received and acted upon. Additionally, feedback was evaluated based on the individual providing it (instructor, tutor, peer-with instructor feedback being preferred) as well as the tone and language chosen to convey the message (which was noted by participants as having an impact on usability and reception). Peer feedback was examined (e.g., Wilkins, Shin, and Ainsworth [26]) and found to increase reflective thinking and collaboration, but most conclusions suggest that peer feedback cannot replace instructor feedback but can supplement it.

Feedback Forms
In the literature, the various forms by which feedback was provided were evaluated frequently, and ranged from descriptive (e.g., Mathisen [29]) to comparative across perceptions (e.g., Gould and Day [30]). Both a focus for the students and the instructors within the qualitative studies, feedback forms were noted to impact the effectiveness of the feedback's message, the timeliness and actionability of the feedback in application to future work. Hennessy and Forester [31] conducted a comparison of forms (audio and written) and concluded that more than one mode may increase access to the positive effects of feedback, confirming the findings of prior inquiry (e.g., Miller [32]). Several innovations for use within typical forms were found (e.g., Bloxham and Campbell [33] examined the use of cover sheets to spur productive dialogue between tutors and students), and outcomes in these types of studies indicated the potential for impact on overall feedback quality and its use (e.g., Sopina and McNeill [34]).

Feedback Content
Characteristics of the feedback content were also present in the literature, providing descriptions for included elements. Preferences or beliefs about what should be said within the feedback were discovered related to valence and actionable content, (e.g., Poulos and Mahony [35]), and most recently, considerations of affect and personalization to student growth [36]. Other research emphasized the need to create a connection between the feedback and assessment criteria as a general standard [37].

Feedback Functions and Purposes
Dawson and colleagues [36] are one example from the literature that examined perceptions of the functions of feedback. Students and instructors indicated that improvement of work was defined by the increased understanding of the content, but also by self-reflection and critical thinking indicators.

Discussion
The current literature review sought to extend current understanding by using existing research to identify the features of quality feedback in higher education as well as the perceptions of faculty and students in higher education regarding what is considered quality feedback. The outcomes confirmed the findings of previous reviews that there is a wide range of foci in the research literature. Despite this, clear themes emerged which indicate that standards for quality feedback in higher education can be established and applied through a quality framework.
The identification of critical features and functions of feedback is of upmost importance if feedback is to be determined to be effective. O'Donovan, Rust and Price [38] noted a failure between theory and practice for the effectiveness of feedback. This literature review contributes by identifying indicators which promote quality, a necessary step prior to feedback being effective. While this review utilized feedback characteristics originally outline for classroom teachers, the utility of the adapted framework was applicable and appropriate given the research literature surrounding feedback in higher education. It is necessary for IHEs to determine if faculty (or the structures they utilize such as peer feedback opportunities) are providing feedback that has quality elements for not only content, but for the strategies or processes as well. It is possible to develop a tool for use in institutions of higher education (IHEs) which can measure the implementation of essential feedback characteristics and provide feedback to faculty regarding the number of quality indicators found in their current practice. Providing faculty with feedback can lead to a variety of preferred outcomes such as the reduction or even elimination of the perceived mismatch between instructor practices and learning goals. This type of feedback could also promote active relationships between instructors and students surrounding the feedback and improvement exchanges. Additionally, faculty may be more thoughtful and willing to use innovative tools and techniques to increase the effectiveness of higher education feedback. The quantitative analysis within this review strengthens this call for action on the part of educators in EPPs to reform practice.

Consistent Definitions of Features
For the coded samples associated with feedback content, there seemed to be relative standards to follow across disciplines. Most of the characteristics described here (including timing, focus, means of comparison, valence, and quality for clarity, specificity, and tone) are repeated throughout the literature with a consistent definition of their quality. For example, timing is generally considered to be high quality when it falls within close proximity to the initial learning and production of work products from both the instructor's perspectives [37] and the students' perspectives [35].

Feature Implementation Inconsistency
The implementation of high quality feedback features in the literature were inconsistent. Substantial mismatches were noted in some cases (e.g., Mulliner and Tucker [37]) indicating that even when instructors understand the elements of high-quality feedback, the effective implementation of those quality elements can remain elusive. The use of effective feedback strategies is also inconsistent.
There is a need for implementation standards for new application of feedback practices, such as with audio feedback (e.g., Hennessey and Forester [31]).

Perceptions
Overall, perceptions emerged as a theme. Dissonance between various conceptualizations of feedback and feedback practices was apparent. Two branches emerged within this theme, both of which have the potential to influence the quality of feedback in higher education.

Instructor Perception vs. Implementation
This theme was associated with multiple examples of instructor feedback practices that did not match the instructors' espoused beliefs about high quality feedback practices (e.g., Orrell, [39]). Student action was one area in which perception did not match practice. Some articles demonstrated a lack of expectations to act on the provided feedback (e.g., Stern and Solomon, [40] described as steps for preliminary guidance, ongoing clarification, supplementary supports, and feeding forward by Hounsell, McCune, Hounsell, and Litjens [41]), Another mismatch was the delivery of feedback on formative or summative assessments, which favored summative assessment (describing the end result of student learning) only (e.g., Orrell [39]). The literature highlighted that feedback practices should reflect the evidence-based methods designed to achieve primary learning goals, which indicate a cyclical approach to apply feedback. For example, Hyland [42] interviewed faculty to examine disciplinary writing feedback and found that their ability to shape student writing toward disciplinary approved standards was considered a primary outcome.

Instructor Perceptions vs. Student Perceptions
The mismatch theme was also noted in articles which addressed instructor and student perceptions of the effectiveness and usefulness of provided feedback (e.g., Weaver [43]). There is also evidence that differing perceptions affect the assessment process as well (e.g., Carless [44]). Certain trends in the quality feedback literature are also at odds with student perceptions, such as demonstrated student preferences for written print feedback over electronic means [45]. Dawson et al. [36] found that while some characteristics of feedback would match between students and faculty, perspectives differ on the finer goals and the ultimate point of feedback improvement. These mismatches are cause for the reevaluation of quality feedback priorities which includes the flexibility to account for student preferences.

Impact of Student Perceptions
Also of note were the instances of student perceptions which have the potential to impact the outcomes of feedback utilization and application to future work. For example, students perceive individual feedback as higher quality than group feedback (e.g., Mulliner and Tucker [37]). The previously noted dissonance between instructor and student ideas of the purpose, most effective form/mode, or valence or the feedback are influential in the student's receptiveness to constructive criticism. Studies are emerging on the ways that students interact with feedback (e.g., Carless and Boud [14]), and calls for continued research investigating the ways in which students appreciate, judge, and manage their affect before taking action are echoed by the current review. Student perceptions of valence in the literature indicated support for constructive criticism in some cases (e.g., Brandt [46]) and a sensitivity to any feedback associated with judgement that could be discouraging to students (e.g., Zhou and Deneen [47]). Indications that including student preferences for the feedback process may increase students' active participation were present [41]. Active student participation is a primary goal for high-quality feedback and requires student and staff buy-in for effective implementation (e.g. Hart and Wakeman [8]).

Potential of Peers
Multiple articles led to the development of a theme on the potential for peers to provide effective feedback to students in higher education classrooms. This potential is highly disputed with indications that staff are likely to reject peer feedback models due to a lack of reliability, a lack of expertise, power struggle concerns, and a lack of time [48]. While some of the literature described the limitations of using peers (e.g., unexplained incongruences and inconsistent expectations [46]), others described their great potential to promote advanced cognitive strategies in their peers [49]. Wilkins, Shin and Aimsworth [26] found that peer feedback provided more opportunities for student reflection and collaboration but was strictly supplemental to the feedback provided by university staff. Ertmer et al. [50] described instructor feedback as more preferred by students (who did not see the value in peer feedback), again demonstrating the potential for student perceptions to impact the feedback process.

Innovative Tool Use
Prominent descriptions, comparisons, and new applications for various supporting technology were noted across the higher education feedback literature. From video feedback [29] to the effective development and acceptance of automatic computer-generated feedback [51], the literature describes promising results for new technology applications. Computer-generated content feedback was also emerging in the literature (e.g., Eddington and Foxworth [52]), with indications that automating some forms of feedback could save time for higher educators and save money for educational institutions [53]. Being on the forefront of many educational feedback supporting technologies, further study would provide a stronger research base and evidence for untested tools and strategies.

Timing, Mode, and Audience Findings
Timing was included in 24% of the identified studies and consistently emphasized timeliness as a virtue of quality feedback. Thirty-two percent of the studies examined written feedback (either paper or electronic), however several studies compared different modes of feedback delivery (e.g., Macgregor, Spiers, and Taylor [16]). Future studies could consider the strengths and weakness of each possible mode, which may influence the feedback exchange. It also may be of interest to further study the impact of providing a variety of modes [10]. In describing the audience for the feedback, indirect or vague descriptions were provided for 52% of the studies. Two studies compared group vs. individual feedback [37,54], with the clear majority providing evidence for individual student feedback.
Consideration of the audience was largely unexamined in the experimental research despite its inclusion in high quality feedback frameworks (e.g., Brookhart [12]). In addition to the tailoring of feedback to address individual student goals, audience considerations include factors about the students that may impact the utilization of the feedback, such as level of study (undergraduate versus master's level students; e.g., Poulos and Mahony [35]) or previous preparation (initial licensure/lateral entry students versus students who have previous lesson plan writing experience). This aspect of high-quality feedback seems largely absent in the literature, however evaluations of the dynamic dimensions of feedback (cognitive, social-affective, and structural) suggest that the conceptualization of feedback as dialogue includes the audience as an active participant [55]. Future examinations should include the recent literature themes of the audience and the impact of recipient participation in the feedback discussion, moving beyond feedback type, provider, and mode as basic descriptors for quality (e.g., Wu [56]).

Conclusions
This study contributes to the field by identifying quality feedback practice trends which result in the biggest impact for success for students in higher education. These include both widely accepted quality standards (e.g., feedback should be positive, specific, timely, and encourage active student engagement) as well as less-known features with emerging evidence in the literature (e.g., the impact of peer feedback practices, novel tool use, and novel feedback forms). To prepare preservice educators for high stakes teacher preparation summative licensure assessments, EPPs will need to become more consistent regarding the content of the feedback provided to students and in the process in which their programs provide such feedback [7]. The included research in this review confirms that effective feedback should be actionable throughout a student's learning process, rather than occurring once on a summative assessment with no chance of application or improvement by the student.
As the measurement of quality feedback and its delivery by higher education faculty is not a typical occurrence other than for the characteristics of timing and tone, it is challenging to require any mode, form, or quality characteristic. This review, however, helps to outline best practices-practical implications for what individual faculty can do to maximize the quality and impact of the feedback provided to students. By utilizing quality feedback practices outlined here that are responsive to student preferences and needs, the time spent on feedback by faculty can directly influence student learning. Moreover, the use of these practices by higher education faculty would address student preference and subsequently, student satisfaction.
This review found research for each of the categories regarding feedback strategies and content, with results indicating differences between the perceptions of adherence to sound feedback practices versus the reality of implementation, the potential of innovative tool use, and a disagreement about the effectiveness of peers for providing effective feedback. Quality feedback requires consideration of strategies, processes, and content, but also a means of addressing potential mismatches in perception of students and faculty within higher education for what is effective. Beyond the recommendations of Gibbs and Simpson [11] as well as Brookhart [12], additional factors described in recent literature (e.g., features available in distance learning [56]) are likely to impact the quality of the feedback. Future work in the evaluation of feedback, particularly on tasks related to high stakes assessment, are necessary to be sure students in higher education are receiving feedback that prompts the adaptations and modifications necessary for student learning and development, as well as the lasting impacts demanded by the rigors of the teaching profession.
Supplementary Materials: The following are available online at http://www.mdpi.com/2227-7102/10/3/60/s1, A table of studies included in this analysis is available at the request of the corresponding author. Literature that was reviewed but not referenced above is included in the reference list .