Using an Implementation Trial of an ePortfolio System to Promote Student Learning through Self-Reflection: Leveraging the Success

Abstract

This paper reports on a pilot study that was conducted during a technical trial of a new ePortfolio system at a large Australian university. Students from a large (n = 325) first-year educational technology course were given the opportunity to use the new ePortfolio system weekly as part of their reflective practice at the end of the hands-on tutorial classes and also through a blogging assignment that required six posts throughout the semester. Although the students reflecting on their work and ePortfolios themselves are not new concepts, this paper reports how assessment practices can be improved using ePortfolios and how students can improve their reflective practice through simple and regular use throughout the 12-week semester that the study was conducted. From the class, 208 students responded to the survey with the results being positive. The students were able to use the system easily and did not report many problems with crashing or freezing. The lessons learnt form an important part of this study for future iterations with these reported in the paper.

1. Introduction

The ePortfolios are not new, with studies into their use being conducted for more than a decade; however, they have evolved greatly during that time and research is continuing to be conducted into various aspects of ePortfolios. One of the benefits of using ePortfolios when they are used well is that they allow students to reflect on their learning. Using an ePortfolio system to promote student self-reflection can be a powerful tool that students then have access to throughout their future studies. This paper reports on the technical trial of one ePortfolio tool that was implemented in a large first-year education course at an Australian university. The course coordinator ensured that the ePortfolio tool was used with the focus on the change in assessment tasks in this course to include student self-reflection as a weekly task, as well as the redesign of the other course assessments around the ePortfolio tool. This meant that the tool was used throughout the course in a methodical way that allowed students to gain experience in using it over time.

The research from this project demonstrates that the ePortfolio tool was an important part of course design and allowed student self-reflection, which in turn promoted student learning. This paper will describe the learning design involved in the course changes and how this maximized the use of the ePortfolio tool in the course.

2. Literature Review

One definition of ePortfolios is that it is a digitized collection of artefacts that can include various items such as demonstrations, resources, and accomplishments that represent either an individual, or a group, or even an institution [1]. This can be beneficial for student assessment tasks that require various types of artefacts.

The ePortfolios allow students to critically reflect “on one’s learning and for compiling and demonstrating evidence of learning and skill development” [2] (p. 1). This also allows students to make connections among their learning experiences, which can then enable the transfer of knowledge and skills to other contexts [3] in the future. Thus, in this instance the students are able to better apply their knowledge to their teaching situation in the future. The tool also allows the students to go back to think about what they have learnt, thus helping with their memory as well. One study to assist with critical reflection using ePorfolios reported positive results when using various classroom strategies to help students [4]. This included the innovative use of in-class questioning to allow students to reflect on why they were learning this and how this will assist the students.

2.1. Chalk & Wire ePortfolio System

Chalk & Wire (https://www.campuslabs.com/chalk-and-wire/ (accessed on 29 April 2021)) is an assessment-oriented ePortfolio system that has applicability to all disciplines and supports the users’ academic, career development and job seeking needs thanks to its wide array of tasks, for example course work presentation, external fieldwork assessment, secure resume and downloadable portfolios [5]. As a pioneer ePortfolio system, Chalk & Wire was widely used in Canada and the United States [6], especially in higher educational institutions where it was officially implemented campus-wide for reflection and assessment in various programs and courses [7,8,9]. In Australasia, however, this viable and powerful assessment ePortfolio system was under adopted. A 2017 study scanned 48 universities in Australia and New Zealand and found only one of them employed Chalk & Wire [10]. Egan et al. (2018) [11] investigated the implementation of Chalk & Wire at The University of Auckland and concluded that a program-level strategy embedding ePortfolios across the curriculum and assessment tasks is beneficial to students when a scaffolded and structured approach is adopted. These results are a positive sign for the potential implementation success of ePortfolio systems such as Chalk & Wire at other Australasian universities, and they demonstrate that this research has not been conducted in Australasia previously.

2.2. Self-Reflection and ePortfolios

Self-reflection refers to “the processes that a learner undergoes to look back on his past learning experiences and what he did to enable learning to occur (i.e., self-reflection on how learning took place), and the exploration of connections between the knowledge that was taught and the learner’s own ideas about them (i.e., self-reflection on what was learned)” [12] (p. 531). Self-reflection differs from reflection, which was defined as the “mental process of structuring or restructuring an experience, a problem or pre-existing knowledge or existing recognition” [13] (p. 193). Self-reflection was found to have a positive impact on student performance [12,14,15] and heighten student self-efficacy and resilience [16].

There was research evidence that ePortfolios could facilitate students’ reflective practice [16,17,18], promote student self-reflection [19], engagement [20], critical thinking, digital literacy, and integration of curricular experiences [21]. Previous literature also shed light on the identified significant factors influencing students’ reflective practice, utilizing ePortfolios regarding the institution, curriculum, learning process, the learner and portfolio itself [22], as well as faculty adoption, resources, training, and scaling challenges [23].

While various studies have investigated the use of ePortfolios for structured reflection activities, professional artefact presentations and growth documentation as in the work by Picardo and Sabourin (2018) [15], few have investigated ePortfolios and self-reflection for assessment tasks despite the research evidence for their efficiency as an assessment tool [19,24]. This study, therefore, aims to study how Chalk & Wire, an assessment-oriented ePortfolio system, can improve assessment practices and student reflective practice.

While Dune et al. (2018) [18] observed that the previous literature provided support for the positive role of ePortfolios in developing self-reflection and ongoing reflective practice, there remain various concerns hindering the successful implementation process of ePortfolios; for example, technological issues and increased teacher workloads [25,26,27], as well as conflicting student attitudes [28]. Although there are implementation frameworks available for use (such as in [3]), it is uncertain if any such framework was used for the technical implementation. Fortunately, recent research results have presented the best practices in implementing ePortfolios for self-reflection [29] and addressing the identified technological issues [27]. These results provide practical implications for the implementation of ePortfolios for self-reflection in Australasian universities where Chalk & Wire is under-implemented and researched [10].

2.3. Learning Design

Learning design, an area with increasing research interest over the last 15 years [30], is defined as “a new methodology to help practitioners make more pedagogically informed design decisions that make appropriate use of digital technologies” [31] (p. 1). The systematic review by Dagnino et al. (2018) [32] revealed a significant number of learning design approaches and tools; however, their adoption was limited and there were unsolved challenges regarding teachers behaviors and opinions. The identified challenges included the disconnection between the learning design and delivery [33], teachers’ lack of design skills and knowledge [34], varied pedagogical approaches, context restrictions, limited tool functionality, and lack of practitioner cooperation [35] as well as the mismatch between the teachers’ designs and students behaviors [36]. While this field of educational research and practice is now acknowledged as a complex and integrated process that demands specialized knowledge and skills, which is forward looking and globally focused [37], it was also noticed that most studies had been conducted in European and North American institutes [38].

In order to ensure the scalability and sustainability of teacher training, this study adopted the principles for practices proposed by Bennett, Lockyer, and Agostinho (2018) [39] that learning design must be an integral part and routine in teaching, teacher engagement should be a priority, and institutional arrangements must be open to changes. These principles inform the self-reflection and blog assessment tasks during the course as effective design and reflection practice were believed to be critical to high-quality learning outcomes, increased academic performance and engagement [40,41]. This learning design process is consistent with the latest research results regarding its conceptualization as both incremental explorations based on retrospective reflections and dynamic, experimental opportunities for new practices, which serve as “springboards for development” [42] (p. 111).

Following the literature review, the following research questions were designed:

• In what ways can Chalk & Wire be implemented in a first-year education course?
• How was the course learning design changed in order to implement Chalk & Wire?
• What were the students’ perceptions of their experiences about using Chalk & Wire in the course?

3. Materials and Methods

There were 325 students enrolled in the first-year education course. The course was taught by one academic who was also the course convenor and a team of eight teaching staff who each taught between one and two tutorial classes each week. There was a total of 17 tutorial classes conducted each week for ten weeks of the 12-week semester. A 50-min lecture was also conducted each week of the semester. The content for the course was based around educational technology and using technology in the secondary classroom once the students became teachers. Thus, the use of ePortfolios fitted well with the course content and allowed the students to experience first-hand another technology.

This course was chosen to participate in a university technical pilot for a new ePortfolio system, Chalk & Wire, with several learning and teaching activities being conducted in the course throughout the semester that used the system. These included an in-class weekly reflective participation journal that was completed by students at the end of each week’s tutorial class. These were completed ten times in the semester, at each of the ten tutorial classes. Students were expected to complete a minimum of eight journals throughout the semester in order to pass the 10% component of the course. The SAMR (substitution, augmentation, modification, and redefinition) model [43] was used to investigate if the product Chalk & Wire was suitable in the course in place of the previous products. The learning design for the course was then changed in light of this as Chalk & Wire was applicable at all the four hierarchical levels of learning enhancement and transformation that are proposed by the SAMR model [43].

Ethics approval was gained for the data being collected, which included a survey being conducted online in Qualtrics at the end of the semester during the final tutorial class. If students were not in the class that week they were emailed and were able to complete the survey online in the few days after the class. The survey gave the students an opportunity to have their voice heard around the technical implementation of Chalk & Wire and report on their experiences using the program. Of the 325 students enrolled in the course, 216 students responded to the anonymous online survey with 208 students completing the questions specifically regarding the ePortfolio system. This represents 64% of the students enrolled in the course completing the survey questions. From the students who completed the survey, 58% were female with over 90% of total respondents between 18–24. Students were asked how easy it was to use the platform, how often the platform froze or crashed, how easily they were able to complete the required tasks, and how useful they found the provided support documentation. Finally, students were also asked how they felt about using the software during the semester. The survey was conducted online as a reasonably quick method of gaining the data [44], while representing a wide target population [45].

Ethics approval was also obtained to analyze some of the student reflections that were written during Chalk & Wire during the semester. These included the reflections at the end of the class when the students were given further training in Chalk & Wire in week 8 of the semester. This training gave the students the skills required to use the program to complete the blog assignment and was conducted by a staff member of the library who had good working knowledge of Chalk & Wire.

Quotes used in the results were selected from open responses from the survey as well as through the analysis of one class of student reflections from the week 8 tutorial class where students were trained in more advanced features of Chalk & Wire. The reflection was part of the weekly in-class activities. Student reflections were generally positive in nature, which may represent the data shown below to be over positive; however, the students did use Chalk & Wire each week for class reflections as well as for assessment tasks. A word cloud from student comments has been created to ascertain the most used words.

Reflection, which was used as a methodology by a number of researchers [46] thanks to its ability to evaluate human experience [47] and explore the transformation of their learning [48], was adopted in the study to triangulate the research methods and boost the credibility of the current study. The triangulation of survey and students’ reflection data does not only validate the methodology but also deepens and widens understanding [49] on the Chalk & Wire implementation and students’ reflection. The methodological triangulation of different data collection methods at different times during a 12-week period helps increase the methodical rigor of the study and ensure that fundamental biases arising from the use of a single method or a single observer are overcome [50].

4. Results

The results section is presented with an explanation of the implentation in detail as well as a description of how the learning design for the course has been changed. Thus, this will answer research questions one and two, while research question three will be answered with the presentation of the results from the student survey and the analysis of the qualitative responses. From the student data, there are varied results that are mostly positive in nature.

4.1. The Implementation in Detail

Previously the students used OneNote to complete nine weekly reflections, which were not used as part of any assessment, and they also completed six blog posts in Blackboard, which were assessable. These reflection tasks allowed the students to gain reflection skills throughout the course, with Figure 1 showing the process flow chart summarizing the assignments when they were located in Blackboard, which was prior to the change in learning design.

Although this course design was considered good, there were limitations with the reflection tasks; one limitation was that the weekly reflection was not assessable, and thus not read by another audience. The blog post assignment was limited by the platform. Thus, the course was redesigned as shown in Figure 2.

In the new version of the course, in the reflection requirement of the course, which was labelled participation, the students were asked to reflect on one significant thing they have learnt and to ask one significant question in the journal space for each of the ten weeks. This was completed at the end of the 1.5 h tutorial session, with students expected to complete a minimum of eight reflections in the semester. The students were asked to write a minimum of 50 words per question. This assessment task was worth 10% of the total assessment for the course.

The other task that used the ePortfolio was an assignment that allowed the students to evaluate a range of technologies to complete the reflective blog assignment that was then placed into the ePortfolio system. The students were required to complete six posts on various set topics on technologies that were covered during the semester. This allowed the students to reflect as well as research the theory around the topics. The students were also taught about reflective writing, which included the use of Hegarty’s model [51] showing the different the levels of reflection. The students were then provided scaffolding around this, both in the lectures and the tutorial classes, which assisted in the students learning of how to critically reflect in greater ways and using more depth. Another component of the blogging assignment was for the students to include a range of multimedia in their blog posts. This ensured that the students were using the skills they had learnt throughout the course.

The students were taught how to use the system through a demonstration in the first lecture and through training in the first tutorial class. In addition, detailed instructions on how to use the platform were provided to the students in both Blackboard through the course site and in Chalk & Wire itself. The students were given assistance with the reflection submission by the class tutor each week while students were learning the process. As part of the implementation, student exemplars of the blogging assignment were created in the ePortfolio system so that the students could see the expectations around the assignment.

The course teaching staff were given training that was systematic in nature prior to the course commencing. This was via a workshop conducted for all of the course teaching staff. They were then provided access to support through a staff member involved in the pilot when needed. The support was provided to determine the best ways to support student submission methods and tutor assignment grading in the system.

4.2. Student Data Results

The students reported that Chalk & Wire was extremely easy to use (took one session to be able to accurately navigate) or very easy to use (took two to three sessions to accurately navigate), with 18% (n = 38) finding it extremely easy to use and 43% (n = 90) finding it very easy to use (Figure 3). A further 27% (n = 56) reported an average ability to use where it took some time to learn where to navigate to. Thus, 61% of the students took between one and three sessions to be able to accurately navigate the platform. The standard deviation for this question was 0.93.

A high 94% (n = 195) of the students reported that the platform either did not freeze or crash, or only did once or twice (Figure 4). This leaves only 13 of the 208 respondents who reported the platform had problems when they used it, which might also be why the standard deviation for the question is a small 0.64. Considering the number of uses per week and for the final assignment, this is a good result and suggests the program worked well throughout the semester.

The students were asked if they were able to complete the required assignments (both the reflection tasks and the blogging assignment) with ease. With 55% (n = 113) of students reporting they were able to extremely easily or very easily, the results are positive. Another 34% (n = 71) reported they were able to do so moderately easily. Only 11% stated they had some difficulty, with the standard deviation being 0.91. Unfortunately, from the data it is not clear what types of difficulty the students were reporting.

In the survey the students were asked about supporting documentation. This included videos and other useful files that were provided to assist the students. Again, the results are positive with 49% of the students reporting the documentation to be either extremely useful or quite useful (Figure 5). Another 39% of the respondents stated that it was moderately useful. Only 8% (n = 17) stated it was only slightly useful and 4% (n = 8) stated that is was not at all useful, with the standard deviation for this question being 0.91.

The students were also asked how they felt about using the software. As can be seen in Figure 6, the results are quite positive with 9% (n = 19) of the students checking “I really like it and am looking forward to seeing what I can achieve/create with this” and 43% (n = 88) of the students checking “I can see that it will be useful so I’m willing to learning how to use it”. Another 31% (n = 65) checked “I don’t have any feeling one way or another—it’s just another tool”, while 7% (n = 15) checked “Slightly stressed—it’s another thing I have to master at university”. Only 10% (n = 20) would prefer not to have to use it. This suggests that the students were either keen to use it, or okay with using it, which is a positive way of looking at the tool that was required to be used during the course. The standard deviation was a little higher for this question, with it being 1.07. This may be due to the more personal nature of the question in that it asks them how they felt about using the software, which may change depending on how they feel at the time or on the day.

Each week the students reflected on their learning at the end of class in the ePortfolio, and there was one week dedicated for students to reflect in greater detail on learning how to use the platform. This happened during the week 8 tutorial, when the students participated in a workshop conducted by a library staff member to give the students the skills needed to complete the blog assignment.

The student reflections were varied and quite detailed in what they had learnt. Some merely described what they did during the tutorial, as one reflection says, “I have learnt how to edit, format and embed multimedia into e-portfolio in order to create a complete and well displayed blog.” Some other students, however, went into more details regarding what they learnt. One student commented the following:

One thing that I have learnt is how to change the blog layout and how to embed codes like YouTube into the text block. I also learnt how to insert images, change fonts and use the system. I learnt how to submit it as well.

A word cloud was created using the 20 student comments on their reflection of learning the advanced features of Chalk & Wire (Figure 7). As expected, ‘assignment’ was the most used term, while ‘learnt’, ‘blog’ and ‘embed’ were also highly used. Two terms stand out as positive with regards to the reflections, which are ‘really’ and ‘helpful’.

The following student compared the platform and Microsoft Word: “Today I learnt how to manage the ePortfolio on which I will write my blog assignment. I also learnt that the software does not save like Word, so it is imperative to save before exiting”. This student will not have problems with saving, and such detailed reflections suggested that the students appreciated the hands-on workshop and found it useful in different ways dependending on their competence and learning experience. There were also students who not only described what they learnt but also how they felt about the learning experience, as can be seen in the following comment:

Today I learnt how to use the ePortfolio. A gentleman (feeling terrible because I have forgotten his name already—sorry!) from the library did a small presentation of how to embed videos, pictures, sounds and general ways of editing. It was really useful as now I can start to upload my blogs.

After the description of the training content, the student also shared their thoughts about the usefulness of the workshop to their assessment task completion. Another student even evaluated the platform interface and functions, saying, “Lots of ways to use the Chalk and Wire interface. It’s very user friendly, lots of ideas on how to make my blog assignment engaging and creative”.

In another reflection, the writer covered not only the procedure, but also the prospect artefact, and their feelings and intention to use technologies in their future teaching as follows:

In today’s tutorial we discovered how to use the many different gadgets and tools to bring together my blog assignment. It is nice to have an assignment which is visually pleasing to make, as it will be entertaining trying to find different types of multimedia to use in my assignment. In doing this I’m more interested in potentially using multimedia in the classroom in the future.

The above student connected the platform used and the assessment task as well as related them to their learning and future practicum, which made it an insightful self-reflection regarding a positive and meaningful learning experience. The reflection included both an evaluation of the interface and ease of use, and it was very clear about the technical difficulties one may encounter using the platform.

Despite being positive overall, there were some negative comments by a small number of students that were reported in the course evaluations (less than 1%). For example, one student pointed out a critical point about the platform, as follows: “This week we learnt about how to use the chalk and wire software properly. It is a simple system but is quite picky and you need to be careful with submitting properly”.

This is to be expected in the implementation of a new system and suggests the importance of the workshop in providing students with initial training so they can avoid the pitfalls in creating their works in an unfamiliar platform. The reflections are quite varied and specific regarding practical techniques and strategies that the students learnt during the training, which resulted in a positive learning experience for the students and smooth course implementation for the teaching staff.

5. Discussion

Through changing the learning design of the course, the assessment tasks were predominately updated and modified to allow the students to be able to reflect over their learning throughout the entire semester in the course. This provided improvements in reflective practice and allowed the students to perform well in the final blogging assignment, which also required reflection. These results confirmed the critical significance of an effective design in creating high-quality learning outcomes, and they contribute to the emerging body of empirical evidence for learning design in higher education contexts as a response to the call for further research to build a comprehensive knowledge base and progress widespread practice change raised by Bennett et al. (2018) [39].

The tool, Chalk & Wire, was implemented using good support, by training the teaching staff well in its use and also teaching the students specifically during two times in the course, in the first tutorial class and also in the class during week 8. While Agostinho and colleagues (2018) [30] found that training workshops were a kind of support accessed by teachers in their design work, they also presented mixed attitudes with some critical feedback towards the effectiveness of attending them. This study, however, revealed that training workshops contributed positively to the smooth implementation of the platform for both reflection and assessment tasks. The workshops were found particularly useful by students as they provided them with technical knowledge and skills to complete their assignments. Minimal problems were reported by the students, with most using the tool with little or no problems, which suggested the importance of training provision for students beforehand. The findings in this study are consistent with the students’ positive attitudes towards the use of ePortfolios for reflective assessments observed by Bodle, Malin and Wynhoven (2017) [24].

The weekly reflection completed each week provided evidence of learning for the students. This then enabled the students to store validated evidence of their capabilities and/or achievements and reflections on their learning. This included asking questions that may be outstanding and potentially being provided an answer to the question in either the next lecture or by the tutor who marked the weekly reflection. The results are in line with the previous literature regarding the strengths of ePortfolio systems for assessment purposes [52] (p. 5), and have furthered the understanding into the effective application of ePortfolios for integrated reflection practice, which was presented as a challenge for students in a study by Wang (2009) [6].

The assessment procedures were changed in the course and this provided greater constructive alignment between the learning outcomes, learning activities and assessment tasks, which leads to improved student learning [53]. The positive results confirmed the advantages of Chalk & Wire as pointed out in the American higher education context [7,8]. This study therefore provides practical implications for future reflection and assessment practices using ePortfolio systems, especially in Australasian universities where Chalk & Wire has not been used widely [10].

Lessons Learnt

There are several lessons learnt from the technical trial of Chalk & Wire. These include the fact that at the time of the study technical updates were required to improve the functionality of the platform, particularly for the teaching staff. This could be the solution to the challenge identified in the previous literature that the process of ePortfolios for self-reflection and reflective practice was not always smooth due to the students’ struggles in engaging, manipulating and managing technology [18,27]. However, the greatest lesson learnt is that when requiring the students to reflect on their learning, sometimes less structure or scaffolding is beneficial. The students sometimes just asked a question to fulfil the requirement of the assessment task, but they did not really have a question that needed asking. As such, they were informed this was no longer a compulsory requirement. This allowed the students to then focus on what they had learnt in class and how to apply it. The result of this meant that the tutors were no longer spending considerable time answering questions that students may not have been particularly worried about. This was an important lesson learnt as it was creating unnecessary marking for the tutors and we were spending time in lectures answering questions that may not have necessarily been thought of as important.

With future iterations, the assessment tasks can continue to be modified and adapted so that best practice is being used, as well as meeting the needs of the students. While some researchers, such as Lowenthal, White, and Cooley (2011) [54], already reported on a process for changing assessments using ePortfolios, this study contributed to the current literature on good practice and the successful implementation of e-Portfolio systems for self-reflection and assessment, which was considered still as a debated method [26,27,28].

6. Conclusions

This paper reports the implementation of a new ePortfolio system into a large education course for first-year students. It reports on how improved assessment practices and weekly use of reflection using Chalk & Wire allow students to reflect on their learning and ask questions as needed. The next iteration of this project would be to ensure that the students are fully aware of why they are reflecting on their learning and the importance of this, especially for pre-service teachers, as this will hopefully allow them to become reflective practitioners in the future. Overall, the implementation was a success with the learning design changed in beneficial ways to assist with students being able to reflect at the end of each tutorial class.

One limitation of the study is that data are only from the one course and that students were not specifically asked about their reflective practice as part of this current research. This can be corrected in future iterations of the ePortfolio system. However, the paper is an important contribution to the field as it shows that, with good training and support, an implementation can be successful for both students and staff and that changing a course design can allow for an improved course experience. This research is relevant to other ePortfolio programs, such as PebblePad, as the lessons learnt can be implemented across ePortfolios more widely.