Education Sciences the Effects of Facilitating Feedback on Online Learners' Cognitive Engagement: Evidence from the Asynchronous Online Discussion

With a large-scale online K-12 teacher professional development course as the research context, this study examined the effects of facilitating feedback on online learners' cognitive engagement using quasi-experiment method. A total of 1,540 discussion messages from 110 learners (65 in the experimental group and 45 in the control group) were both quantitatively and qualitatively analyzed. Results revealed that facilitating feedback significantly impacted learners' cognitive engagement: (1) the level of cognitive engagement presented in treatment group was significantly higher than that of control group; (2) cognitive engagement levels of original postings increased overtime in the treatment group, while decreased in the control group; (3) the difference in discussion quantities between two groups was not significant. Effective feedback strategies and the importance of facilitating feedback in creating quality online instruction were discussed.


Introduction
With the fast advances in emerging web technologies, distance education has created greater opportunities for learners without constraints on geographic location, socio-economic condition, or time of their choosing [1,2], and is especially appreciated by adult learners who have fulltime work, family responsibilities, and multiple social obligations [3,4].According to Jonnasen, the cognitive tools embedded in the online environment provide learners with more experience of knowledge construction, self-directed inquiry, and collaborative learning [5].This constructive environment keeps alive the control-decentralized learning paradigm, in which learners have more opportunities to participate and contribute with their own voice [6,7].In addition, many researchers advocate that distance education has great potential in reaching more students and reducing the cost (i.e., [8][9][10]).All these benefits bring great promise to distance education, and the demand for it will only continue to grow [11].
Although distance education has its strengths as mentioned above, it also faces some challenges due to physical separation [12].This lack of physical presence often results in learners' feeling of isolation, which is the primary reason for higher dropout rates in distance education [13].In addition, increased chances of misunderstanding, difficulties in tracking learners' learning, and more psychological distance between learners are some of the issues we encounter in distance education [14][15][16][17].To address these issues, great attention has been paid to communication-orientated activities in online learning.
Learners' opportunity to interact with other participants is an essential component in instruction [5,18], and learning is more likely to occur when such interaction is enhanced [19].In this regard, online discussion becomes an important component of the online learning experience [20][21][22].Laurillard advocates this point and further suggests that, in distance education, the pedagogical benefits of technology rely primarily on how successfully it supports the dialogue between students, or between teacher and students [23].When suitably organized, such dialogue could provide a communal space for learners to socially construct knowledge, in which they are encouraged to take responsibility for sustained and collective knowledge advancement [24].Scandamalia and Bereiter indicate that knowledge building is more likely to occur in a community that is typified by discourse [25].This is because the dialogic mode provides more opportunities for learners to become co-creators of collective understanding and contributors of community knowledge base rather than passive information receivers [26].They make argument, diagnose misconception, negotiating perspectives, share personal experiences, challenge accepted beliefs, and demand reason for conclusions, and all of these are important cognitive presence in deep learning [27,28].Meyer believes that the socio-cognitive dynamics of online discourse could establish a constructive, collaborative, intentional, and authentic learning experience for learners [29], in which not only could they gain a thorough understanding of the subject content, but also obtain critical thinking skills and inquiry capacities that are prerequisite for an independent thinker.
However, the sole existence of discussion activities still cannot guarantee the effectiveness of distance instruction [28].Researchers have identified some potential problems that might arise in online discussions.Meyer and Smith et al. found that learners are less motivated to participate in online discussions because written communication is more labor-intensive, and learners have to spend more time to read and respond to others' postings [30,31].Moreover, some studies reported low level of learning in online discussion [30,32,33].Specifically, the discussions are merely of social nature and stay at the superficial level.In some cases, the conversation is even not related to the pre-determined topic.[34,35].
In situations like these, facilitating can play an important role in developing critical and in-depth discussions and improving online learning outcomes.According to Vandehaar, what matters in distance education is not the technology format, but the teaching and support from the instructor [36].Feedback from the instructor/facilitator is one of the essential heuristics to promote learning [37].Empirical evidence is shown by some research revealing that more effective learning occurs when learners are provided with instructors' feedback [38][39][40].Feedback could provide learners the outcome information regarding the correctness of their response [41], and additional information regarding the nature of outcomes and the qualities of their cognitive processing [39].Fabro and Garrison reveal that instructor's constructive critique into students' thinking process significantly influences learners' higher-order learning [42].In an online course, the instructor's presence by giving feedback has the potential to reduce the insecurity of lonely study [43], keep awareness of learning purpose [43], and improve self-confidence and self-esteem of students [45].
To better understand the feedback pedagogy, feedback effects have been explored by scholars and adopted in various practices.For example, Nicol and Macfarlane-Dick proposed seven principles of good feedback practice for self-regulated learning [46].Chai reported the feedback effects on knowledge acquisition in online learning [47].Zumbach, Hillers, and Reimann examined the feedback mechanism in problem-based online learning [48].Butler and Winne studied the feedback in self-regulated learning from the theoretical view [39].Effective feedback strategies in online instruction are identified including providing probing questions [45], clarifying the learning purpose/goal [40], encouraging positive motivational beliefs and self-esteem [46], provoking reflection [44,45], defining students' roles/responsibilities, and encouraging students to explicitly confront others' ideas [48].
A large body of research has explored the feedback effects on traditional classroom teaching (i.e., [39,49,50]).Some studies examined the feedback effects in online learning generally at test achievement level (i.e., [47,48,51]).The impact of feedback on students' discussion activities, especially the specific intellectual aspects of discussion, in online learning environment has not been fully examined yet.Cognitive engagement is a concept proposed to describe such intellectual aspects in learning.In this study, we analyze the learner-generated data to examine the feedback effects on enhancing online learners' cognitive engagement.
Cognitive engagement speaks to "the extent to which learners are able to construct and confirm meaning through sustained reflection and discourse" in a learning community [20].Early studies on online education mainly focused on how to create and maintain social interaction in online learning environment (i.e., [52,53]), gradually building a better understanding of social interaction in online learning [54].Moreover, research found that social interaction by itself is not sufficient to reflect the quality and depth of learning [44].Actively cognitive engagement that deepens the meaning of the educational experience is also very important [55,56].Zhu points out that, even in social learning situation, the quality of learning still primarily depends on the levels of cognitive engagement of individual learners [57].Thus, in this study, we focus our attention on the cognitive engagement of online learners, and use it to indicate the quality of online learning.
In the present context, the facilitating feedback is the information learners receive from the instructor/facilitator about their discussion, with which learners can "confirm, add to, overwrite, tune, or restructure" their ideas and understandings [39].In this study, the online facilitators qualitatively grade the content of learners' postings.Formative comments are given to maintain idea exchanges/improvement, keep conversations on the right track, promote deeper inquiry, and help learners to link practices to theoretical principles.By analyzing the data collected from online discussion forum, we intended to answer the question: What are the facilitating feedback effects on the level of cognitive engagement exhibited by learners in asynchronous online discussion?

Research Setting
This study was conducted in a National K-12 Schoolteacher Online Training Project in China [58], which was developed to teach K-12 teachers to integrate educational technologies into their classrooms.The online course in this study was run on Moodle [59], and was consisted of eight modules.Each module was open for a duration of three to five days.All learners were required to participate in three types of activities: reading and quiz, discussion, and hands-on practice [60].
The core of this course was the asynchronous online discussion activity.In each module, a discussion specification, including discussion topic, cases stories, orientation for preparation, and rules of participation, was provided to learners.With this specification, learners were guided to read the case stories and contribute their ideas and opinions afterwards.They were encouraged to develop their ideas based on the corresponding case stories and the module-learning theme.In addition, all learners were assigned to groups of 7-10 partners.Usually, learners with similar professional background were assigned to the same group to share teaching experience with each other.Learners were allowed to browse on all postings in the discussion forum, and required to at least respond to the postings in their own group.In order to create a learning community of being synchronous at model-level and asynchronous at posting-level [60], all learners were required to post at least one original posting, read other members' postings and reply to at least two postings in each module.

Participants
A total of 110 distance learners of two classes in this online training program participated in this study.These distance learners were all K-12 in-service teachers in China who taught various subjects ranging from Math, Science, Chinese, History, Art, to Computer technology.The demographic data shows that all learners were adults, mostly over 30 years of age, and they all had a bachelor degree or above.In addition, all learners had basic computer skills, such as browsing online, uploading/downloading materials, and searching resources, yet none of them had any prior online training experience.Learners of the first class (n = 65) served as the experimental group, and learners of the second class (n = 45) constituted the control group.The first class received the training in May 2007, and the second class in January 2008.The training in both classes lasted for one month in total.

Study Design
A quasi-experiment design was used in this study.Both groups took the same training course, used the same learning materials, and participated in the same activities in eight modules.Particularly, in the discussion activities, the two groups were provided with the same readings and the discussions were initiated by the same questions.The only difference between the experimental class and the control class was whether or not the facilitating feedback was available.
For the experimental class, one online tutor was assigned to facilitate students' online discussions by scoring and giving feedback to each posting.Such formative feedback followed these principles by encouraging learners to: (1) share their own teaching experience or use authentic information coming from their local schools or organizations; (2) revise and improve their arguments in the postings; (3) use appropriate terminologies and concepts, and to relate their ideas with principles and theories; (4) reflect on their own learning; (5) have positive motivational beliefs and self-esteem.Within 24 h after the learner submitted his/her post, the tutor graded their postings using rubrics listed in Table 1.In this course, the tutor used the comments as the personal response to the learner's discussions and contributions.At the same time, giving comments was the strategy to foster learners' deeper thinking and reflection.In the comments, the tutor pointed out the strength and weakness of learners' postings, offered guidance on discussion requirements and how to improve their current performance, and employed probing strategies to move learners to higher-level cognitive engagement and to ensure continual cognitive development.The feedback was of formative nature.Learners were encouraged to revise their old postings and improve their discussion performance based on the comments given by the tutor.In order to motivate learners to re-approach discussion in a deeper way, the tutor would review and re-grade learners' revised postings and gave bonus points to the new postings that demonstrated richer understanding and deeper thinking.The grade and the comments were provided in "score" and "teacher comment" boxes respectively as shown in Figure 1, and they were only open to the specific individual learner, and could not be viewed by others.
In addition, during the online discussion activity, the tutor employed feedback strategies, such as praising, encouraging, and appropriate interruptions, to help learners actively engage in the conversations.Postings considered as excellent would be selected by the facilitator and recommended to all learners.
The control class, by contrast, had no facilitator during the online learning process.After each module's training, learners' postings in the discussion forum were automatically graded by the learning manage system in terms of the posting's length and learners' logging times.For example, two 300-word postings were graded the same score without examining the content.In order to explicitly demonstrate the facilitating condition in the treatment group, the comparison of discussion facilitating between two groups is illustrated in Table 2.

Feedback Intervention in Two Groups Treatment Group Control Group
Grading criteria Postings were graded by the tutor based on the content of the posting.
Postings were graded automatically by the online learning system based on the length of posting.

Comments
Comments were given by the tutor within 24 h.The tutor respond by providing hints, probing questions or encouragement, reminding of overlooked steps, and directing learners' attentions.
No comments were given, no discourse development strategies are employed.

Nature of feedback
Formative.Within the three-day-length module, students were allowed/encouraged to revise/improve/reflect on their postings according to tutor's comments.
Summative.No suggestion was given to learners, and they do not have to improve or reflect on their postings.

Measurement: Cognitive Engagement Level
The Framework for Reflective Pedagogical Thinking developed by Sparks-Langer and Simmons was used as the coding scheme to assess learners' cognitive engagement level.This framework was developed to assess schoolteachers' ability to use concepts and principles to explain teaching activities and classroom events [59].It has identified seven levels of thinking, based on Gagne's hierarchy of thinking [61] and Van Manen's idea of critical reflection [62,63].
As shown in Table 3, in this study, we focus on the five thinking levels used in this framework (the levels "No descriptive language" and "Explanation with consideration of ethical, moral, I am deeply touched by the two designs of history courses with distinct styles: Firstly, I think the two courses have the identical teaching goals.However, the history course of American is designed in a novel manner and closely related with students' life.Focusing more on the cultivation of emotional engagement of students, the history courses design of China has clear three-dimensional teaching goals, and highlights cognitive development of students.
Secondly, in my opinion, there is some difference between the two course in methods used to achieve the goals.The history course of America fully embodies the constructive learning theory and the humanism learning theory, while the history course of China is dominated by cognitivism learning theory.We should reflect on and learn something from such a difference.
In Brief, in comparison of fundamental education of China and America, it is difficult to discriminate which is better.We hope all the school teaches could maintain our traditional advantages, borrow benefits from others, and conduct K-12 education down-to-earth.

Score Comments
Which traditional advantages to maintain?How to borrow from other education culture?Please specify this point using your own teaching experience or stories.
and political issues" were not included).Rubrics were also developed to assess the cognitive levels of replying postings (see Table 4).Examples of each level were also provided in the two tables.
A single posting was defined as the unit of analysis.We coded both the original and replying postings in the discussion forum, and identified the level of cognitive engagement of each one by using the indicators in coding schemas.We put too much emphasis on knowledge acquisition but ignored their critical thinking and their individual learning/thinking experience.

Level 3
Explanation with tradition or personal preference given as the rationale.
We should not simply teach our teachers to use technology, but rather we need to teach them how to integrate technology into their classrooms.Just like we always did in our own teaching, we want our students to not only acquire the knowledge, but also to apply the knowledge.

Level 4
Explanation with principle/theory given as the rationale.
This is an effective way, because the performance assessment method is used.This method use "task" to motivate students, and measures what the students can do during the process.

Level 5
Explanation with principle/theory and consideration of context factors.
Instructional media is rich in our life, but we do not have to be limited to using high-tech.All the tools, no matter multi-media or not, if appropriately used, could also improve our teaching.Recently when I taught the concepts of Angle and Edge in a 2nd grade math class in a rural area.I was told that no computers would be provided.Then I redesigned my instruction, and made an Angle model using stiff paper and nails….

Standard Examples
Level 1 Simple agreement or disagreement to the original postings.

Strongly agree with you!
Level 2 Explanation with tradition, personal preference, or principle/theory given as the rationale.
I agree with all you guys.New media is a "truck to deliver the cargo".However, we also need to have the ability to control the "truck", and make this truck's features fit into our teaching styles.Then we can have an effective instructional design.

Level 3
Explanation with tradition, personal preference, or principle/theory given as the rationale, and consideration of context factors.
I agree with you.Technology based teaching is a good way to improve students' learning.But, we need to make sure that the technology is integrated appropriately.There was a lesson competition in my school, and we were required to use technology to teach.Some teachers just simply changed their materials to electronic version but did not pay attention to the pedagogical things, the results turned out to be not good.I know a very good philosophy teacher that he did not use any technology.He just used role play games to teach students abstract philosophical concepts.His teaching was very successful.

Data Collection and Analysis
There were eight modules in total in this online course, with the online discussion activity in each module.However, the discussions in module 1, 4, and 8 were not directly associated with the learning content, but were about social interaction (i.e., Icebreaking) and course evaluation.Thus, in the data collection, we excluded the postings in these three modules and finally collected 1,540 postings from the other five modules in the discussion forum.
These threaded text-based postings were the main data source in this study.Content analysis method was applied to qualitatively analyze the content of postings aiming at providing a richer view on learners' cognitive engagement and the progression of discussion.The researchers carefully studied all the transcripts, and used the coding schemas in Tables 3 and 4 to identify learners' cognitive engagement and grouped them into different levels.To increase the inter-rater reliability, negotiated coding approach was employed.The researchers coded the discussion transcripts, and later actively discussed and revised until all researchers arrived at a final agreement [64].

Results and Discussion
Analyses of discussion transcripts of 110 learners reveal that: (1) no significant difference in the quantity of discussion postings between two groups were found; (2) in each module the cognitive engagement levels presented in original and replying postings were much higher in experimental group; (3) cognitive engagement levels of original postings was progressively increased overtime in the experimental group, while it gradually decreased in the control group.

Comparison of the Quantity of Postings
As shown in Table 5, the experimental group posted slightly more messages in the online discussion.On average, each learner in the experimental group posted 1.06 original messages per week, while the control group posted 1.02; each learner in experimental group posted 1.77 replying messages, while the control group posted 1.71.
Although the data showed that the experimental group slightly outperformed control group in the quantity of discussion, the difference was not significant.Two reasons might account for this finding: (1) the discussion rule of "at least one original posting and two replies" ensured the control group learners' participation; (2) mandatory grouping and students being told to discuss only within their own group limited their interactions with peers in other groups.
In each module, learners were required to contribute at least one original posting and reply to at least two people.However, we noticed that the actual numbers of original postings by each student per module in control group and experimental group were 1.02 and 1.05, respectively, and the replying ones were 1.76 and 1.77.This indicated that, on average, learners in this course were much more rubric-directed than self-directed.It seemed that they posted messages just to meet the minimum requirement in terms of the posting quantity.Once they reached the minimum requirement, the inquiry process ceased.To help learners recognize their ideas in discussion could be continually improved is important for driving them to a deeper level of inquiry [24].Further study is very much needed to understand how to motivate learners to make more extensive use of online discussion without being limited to the minimum requirements.

Comparison of the Level of Cognitive Engagement
The cognitive engagement in original postings.As Figure 2 illustrated, the starting points of two groups were very close to each other, which indicated that these two groups demonstrated almost equivalent cognitive level in online discussions before the facilitating feedback intervention.However, after facilitating feedback was provided to the experimental group, the difference between two groups grew as time went by.Moreover, the cognitive level of online discussions in the experimental group rose overtime, while the control group showed a trend of gradual decline.In addition, we noticed that, the experimental group's cognitive level of discussions in every module was higher than that of the control group.On average, the mean cognitive level of experimental group was 3.84 and that of control group was 2.85.An independent sample T test showed that the difference of cognitive level of discussion between two groups was significant (t = 12.16, p < 0.001).
The results indicated that, with the facilitating feedback for their online discussions, the cognitive engagement level in the experimental group was significantly, increasingly, and stably higher than that of the control group, which had no facilitating feedback.In other words, facilitating feedback had evidently improved the cognitive engagement of learners in their online discussions.Four factors might be responsible for this: (1) With the constant feedback and guidance from the facilitators, the learners in the experimental group could in a clear way develop their skills on connecting practice with theories, reflecting on cases or teaching events, and justifying their arguments.This was also shown in the raising trend of their cognitive engagement level; (2) Learners in the experimental group were given the opportunities to revise and refine their ideas/postings, which stimulates or maintains learners' continually cognitive engagement and development.(3) Facilitator's presence could serve as an external stimulus to generate more monitor pressure on learners, and they were more motivated and might perform better in the discussion; (4) The interactions between the facilitator and learners could reduce learners' feeling of loneliness and a sense of isolation, and a sense of community or belonging could make them contribute more to the online discussions.While in the control group, the feeling of freshness for online discussions might easily wane over time.Without the guidance and encouragement from facilitator, it could become harder for them to maintain their enthusiasm and to master the effective strategies for reciprocal communication and critical thinking in the online discussions.In addition, the lack of substantive feedback reduced the opportunities and motivations to rethink and further revise their ideas.
The cognitive engagement in replying postings.As shown in Figure 3, the cognitive engagement level of replying in the experimental group was consistently higher than that in the control group.Generally, the cognitive levels of most replying messages in the experimental group were above 2, while those in the control group were below 1.5.This indicated that when responding to others' postings, learners in the experimental group presented not only their statement, but also included detailed evidences to support their argument.However, most learners in the control group simply voiced their agreement or disagreement without any further analysis.The discrepancy between two replying curves might be caused by two factors: the cognitive level of the original postings and the intervention of facilitating feedback.The learners who showed deeper thinking in the original postings were more likely to demonstrate higher level of cognitive engagement in their later responses to their peers' comments.In additino, the ideas and opinions in high quality original messages were also more likely to inspire in-depth thinking and further discussion.Again, in the experimental group, the facilitator constantly reminded learners to provide evidence (cases, their own experience, etc.) to support their argument, and guided learners to enrich their ideas and link theories to their own experience.Such facilitating feedback helped them to reach level two or three.
It is interesting to find learners in the two groups posted almost the same number of messages but demonstrated significantly different cognitive engagement levels.The number of messages, in some extent, demonstrates the degree/frequency of social interaction among learners.In other words, learners' cognitive engagement performance was not necessarily related with the number of interactions.This result is consistent with the findings of Kanuka and Anderson's study [65].They coded the students' online discussion transcripts, and found that considerable interactions occurred but a low level of cognitive engagement was detected.Garrison et al. pointed out that, in online discussion, quantity of interaction is not enough, and social presence by itself is not sufficient to reflect the quality and depth of learning [44].Thus, our data offers a clue for assessing the quality of online discussion, and also shows that the posting quantity hardly reflects the quality of discussions.To pursue a meaningful online discussion directed to cognitive outcomes, qualitative dimension characterized by learners' cognitive engagement should be stressed more than quantitative measures.This point is consistent with the conclusions of several previous studies [44,[65][66][67].
In addition, we also noticed that very few original/replying postings, in both group, had reached the top level of cognitive engagement in the online discussion.Could online discussion be an appropriate activity to create higher-order cognitive engagement?Many empirical distance education studies have investigated this issue, and higher-order cognitive engagement, for example, idea integration, testing, and application, are reported in computer-mediated communication than face-to-face conversation (i.e., [27,68]).The precise and permanent nature of written communication made it a different learning experience for online learners.The ideas contributed by learners are all recorded in written form, which enable learners to reflect on their previous understanding.In addition, when expressing using deliberate and explicit written words, learners are encouraged to carefully organize and justify their argument [69].In this study, the reasons for the absence of top level cognitive engagement might be associated with the learning culture of "knowledge transmission lecture" in China.With many years of learning in such paradigm, K-12 teachers got more used to be passive information receivers rather than active contributors.When this online teachers training program was launched in 2007, many online learners-Chinese K-12 teachers-didn't know how to "speak" in online discussion, they just posted paragraphs of learning theories on the discussion boards, but seldom shared their own teaching stories and problems, or expressed their opinions in a logical and critical way.

Conclusions
This study has investigated the effects of facilitating feedback on discussions within an online program of K-12 in-service teacher training in China.By quantitatively and qualitatively analyzing the content of 1,540 postings of two classes, we have investigated the effects of facilitating feedback on learners' discussion performance in terms of cognitive engagement.The findings emerged from this study include: (1) no significant difference of the quantity of postings was found between control and experimental group; (2) facilitating feedback positively influenced the level of cognitive engagement presented in the discussion within each module; (3) with facilitating feedback, cognitive engagement levels of original postings increased overtime; while without it, the cognitive engagement levels decreased.
Based on the findings mentioned above, we drew the conclusion that such facilitating feedback has significant effects in enhancing learners' cognitive engagement in online discussions.This kind of feedback strategies include: giving encouraging information, asking thought-provoking questions to nurture in-depth and reflective discussion, and appropriately interrupting their dialog to keep discussion on track.Our results reinforced the conclusion of previous studies in that the quantity of social interactivity (i.e., postings) does not reflect the depth of online discourse (i.e., cognitive engagement).
Unfortunately more detailed demographic information about the participants was not available, for example, the years of their teaching experience, the ratio of male and female, and specific age distribution.Thus, we had no ways to depict a richer picture of our participants.Knowing about the background information is helpful in thoroughly understanding how the formative comments facilitating works in the specific context of this study.
Future research investigating effective facilitating feedback strategies to promote critical online discussion, feedback's role in changing online learning process, timing of such facilitating feedback, the feedback mechanisms in online discussion, and how feedback effects interact with learners' characteristics (e.g., motivation, knowledge base) would be promising directions.

Figure 1 .
Figure 1.Teacher score and comment interface for discussion postings.

Figure 2 .
Figure 2. The cognitive level curves of original postings over time.

Figure 3 .
Figure 3.The cognitive level curves of replying.
Comparison of Chinese and American Education: Concepts Bring Difference Published by *** Date:

Table 3 .
Cognitive coding schema for original posting.

Table 5 .
Distance learners' participation in online discussions.