Next Article in Journal
Modeling American Household Fluid Milk Consumption and their Resulting Greenhouse Gas Emissions
Previous Article in Journal
The Spatiotemporal Dynamics of Soybean and Cattle Production in Brazil
Article

Students’ Perception of Using Digital Badges in Blended Learning Classrooms

by 1,2,*, 3,*, 2 and 4
1
Ph.D. Program of Technology Management, Chung-Hua University, Hsinchu 30012, Taiwan
2
Huaiyin Institute of Technology, Huai’an 223003, China
3
Department of Technology Management, Chung-Hua University, Hsinchu 30012, Taiwan
4
College of Liberal Art, Huaibei Normal University, Huaibei 235000, China
*
Authors to whom correspondence should be addressed.
Sustainability 2019, 11(7), 2151; https://doi.org/10.3390/su11072151
Received: 14 January 2019 / Revised: 1 April 2019 / Accepted: 2 April 2019 / Published: 11 April 2019

Abstract

A digital badge system with a leaderboard ranking was implemented in an 18-week course offered by a Chinese mainland university during the 2018 Spring Semester. Fifty-four junior college students participated in this study, and the learning activities were based on a combination of online and offline methods using the Moodle system. The purpose of the study was to explore these college students’ perception of digital badges in blended learning classrooms as well as their perception of digital badges in general. Q methodology was used in a single case study of the students’ subjective experience of digital badges, and thirty-six Q-statements were selected after a comprehensive reflection of their views of digital badges. The participants were asked to sort the statements to model their opinion in a modified rank-order procedure. Twenty-two valid sorts were entered for the analysis, and three factors were identified to represent groups of participants with similar views. The results are discussed based on the views of students at a local university in mainland China and the way their demographic characteristics related to their perception of digital badges.
Keywords: Q methodology; digital badges; blended learning; higher education Q methodology; digital badges; blended learning; higher education

1. Introduction

With the large-scale open curriculum development of Massive Open Online Courses (MOOCs and the rise of the use of flipped classrooms, there is an urgent need to reform the educational modes in higher education. According to Liu, Huang, and Wosinski [1], “the fusion and development of information technology eliminates the barriers preventing the sharing of information and knowledge, as well as the learning boundary”. Blended learning is now adopted on a widespread scale, and there has been a significant increase in the number of blended courses in higher education in the past decade [2]. Blended learning (BL) combines the advantages of traditional face-to-face instruction with e-learning [3]. The online learning environment can provide students with information on various aspects of the curriculum, can compensate for the lack of learning in class, and can provide online learning resources for them. In view of the convenience of online resources, it is becoming increasingly important to evaluate students’ online learning outside of class and to measure their academic performance based on formal and informal learning. Many instructors believe that learning cannot be recognized without monitoring the learning process and worry that the effectiveness of learning cannot be guaranteed outside class. This has become a barrier to the promotion of blended learning.
However, blended learning and traditional face-to-face learning can be connected well by using digital badges. Mehta, Hull, Young, and Stoller [4] indicate that digital badges, as a form of microscopic certification, could provide a more detailed framework to assess students’ knowledge and skills, to supply evidence of their learning, and to demonstrate their ability to substantially change. Digital badges capture the learning performance in a way that is not recognized by the formal academic accreditation processes in higher education [5]. Even more, open badging can be an easy way to capture formal and informal learning experiences [6].
Previous research on digital badges is somewhat limited. Firstly, it is mainly based on k12 students rather than those in higher education [7,8,9]. Secondly, digital badges are usually given by institutions in programs such as teachers’ professional development [10] but fewer in college students’ learning activities. Thirdly, authors of small studies give their opinion of higher education students’ perception of digital badges. Finally, it is challenging for college students and teachers to accept digital badges as an assessment tool, and it needs to be continuously practiced in higher education to prove its effectiveness.
A digital badges system was used in an undergraduate obligatory course of multimedia animation design offered to junior students by the Department of Journalism. The purpose of the study was to explore these college students’ perception of digital badges in blended learning classrooms as well as their perception of digital badges in general. The structure of the study is as follows: Section 2 contains a review of the literature related to digital badges in education; the Q methodology and practical background, data sources, participants, and implementation process of the research curriculum are explained in Section 3, while Section 4 contains an analysis of the results. The conclusion and some suggestions for further research are presented in Section 5.

2. Theoretical Background

Digital badges (hereinafter badges) are a digital display that acknowledges, recognizes, and validates the achievement of skills and competencies through explicit evidence [11]. Abramovich [12] utilized digital badges in videogames and recommended their use as not only a credentialing mechanism but also an assessment tool in higher education. Cameron, Pierce, Banko, and Gear [13] found that achievement-based rewards during learning or testing increased participants’ intrinsic motivation. Rewards and incentives are used to improve students’ performance and motivation in educational settings. This section contains a review of the literature related to the reward mechanism in education and the use of digital badges in blended learning, as well as digital badges as reward mechanics on a Moodle learning platform.

2.1. Reward Mechanisms and Digital Badges in Education

Badges with points and leader boards can be a gamification element that enables learners to compete with themselves or others [14]. Some researchers have indicated that people have positive, negative, or mediocre perceptions of badges, and some participants have found badges to be less prestigious than certificates of completion [15]. Jones, Hope, and Adams [10] indicate that teachers have a favorable view of digital badges but that they have insufficient understanding of their purpose. Teachers who work in lower grades place greater value on badges received during the experience [10]. Digital badges can affect learners’ motivation [16,17].
However, Hanus and Fox [18] suggest that badges and leaderboards may harm educational outcomes in a competitive context and that students tend to be less intrinsically motivated in a gamified course, which is likely to cause lower final exam scores. Meanwhile, badges have no effect on students with a stronger intrinsic motivation. Some are alarmed that these external rewards represent the ultimate risk by shifting the focus from learning to badge-collecting, thereby “increasing reliance on extrinsic incentives for learning” [19].
Abramovich and Wardrip [20] consider badges as powerful, positive tools to motivate students, and Filsecker and Hickey [7] indicate that external rewards did not undermine fifth graders’ motivation and interest by the introduction of badges. Informing students of the types of digital badges before the learning activities can also increase their motivation [21]. Badges can affect learners’ motivation based on the goal achievement theory.
Badges and external reward mechanisms are used to positively reinforce and stimulate students’ behavior and self-efficacy by measuring their learning progress and learning feedback. Hew, Huang, Chu, and Chiu [22] introduced game competitive mechanisms (badges, points, and leader boards) based on the motivation theory and pointed out that positively motivated students engage with more difficult tasks. Grant [23] classified learners into three types according to their purpose of earning badges as “badge hunters, sharers, and dodgers”. Hunters care about quantity, while sharers care about displaying badges and quality participation, and dodgers are not interested in badges. Biles and Plass [24] found that competition and collaboration elicit a greater situational interest than individual play. They identified five different types of players based on different motivations: socializer, achiever, stimulation seekers, explorers, and collectors. Socializers were motivated by badges that revolved around a common platform for sharing across games. Achievers desired the badge to be linked to some sort of reward. Stimulation seekers and explorers were mainly focused on the game story or environment and considered badges to be a secondary motivation. Collectors focused on challenges and achievements. Since different types of students have different motivations, as well as different and perceptions of badges, it is necessary to explore the relationship between their cognition of badges and their motivation in order to match badges rewards with students’ motivation.

2.2. Blended Learning in Higher Education

This study used a blended learning curriculum, which included lectures to guide learning, face-to-face communication in the classroom, a general discussion of the subject of the course, exercise questions for supervising and checking students’ learning level, and activities that students can perform collaboratively outside the class [2,3,21]. By applying learning theories, Carman [25] concluded that live events, online content, collaboration, assessment, and reference materials were five important elements of a blended learning process. Riffell and Sibley [26] indicate that students in the hybrid course would have a better and more frequent interaction with the instructor in group studies than in a traditional course. Bonk [27] introduced the TEC-VARIETY Model (Tone, Encouragement, Curiosity, Variety, Autonomy, Relevance, Interactive, Engagement, Tension, and Yields Products) for online and blended learning practices to motivate and retain online learners. Based on the cognitive theory, learners’ cognitive beliefs, technological environment, and social environment have the most impact on students’ satisfaction with learning [2]. The definition of BL is as a mix of classroom instruction (i.e., face-to-face) and out-of-class online learning where online work is substituted for class time in this study [28].
As an innovative form of education certification, a digital badge is an online assessment and accreditation mechanism [23]. Digital badges testify learners’ ability more comprehensively than a traditional degree and certification. They can represent a visual record of achievement and meet learners’ needs based on the elasticity of the online learning environment. They can be created, awarded, displayed, and stored online, as well as shared on social media and professional social networking sites [15].

2.3. Digital Badges as Game Mechanics in Moodle Learning Platform

Moodle is a free and open-source learning management system (LMS) used for blended learning, distance education, flipped classroom, and other e-learning projects in schools, universities, workplaces, and other sectors [29]. As a famous learning LMS, Moodle is a diversified technical tool that supports an evaluation of blended learning. Hew, Huang, Chu, and Chiu [22] used Moodle to report the effects of game mechanics to foster users’ engagement in blended learning and found that the digital implementation of game mechanics automatically kept track of students’ activities in Moodle.
Points, badges, and leaderboards are the most-used game elements in gamification to improve students’ attitude, engagement, and performance [30]. In addition, students’ gender affects their experience of and relationship with badges. Students of different genders who obtained badges had a different incentive to learn [23], and anonymity and badge-sharing behavior were related to students’ cognitive type. The mentality of their willingness to share badges is worth exploring [31]. Even more, Yang, Benazir, and Chen [32] found that leaderboard ranking did not affect learners’ English learning outcomes because of their different attitude toward competitions. Numerous researchers have generally used interviews, focus groups, quantitative and qualitative surveys, questionnaires, and experiments to collect data to determine the effect of digital badges in blended learning [7,9,16,22].

3. Materials and Methods

3.1. Q Methodology for Analyzing Students’ Perception

The present paper used the Q methodology, which is an effective method to study people’s subjective perception [33]. The Q method is useful for systematically studying human subjectivity, especially the subjective aspects of people’s views, beliefs, attitudes, opinions, motives, needs, values, and emotions [34]. Watts and Stenner [35] indicate that this method enables the use of “a by-person factor analysis order to identify groups of participants who make sense of (and hence Q ‘sort’) a pool of items in a comparable way”. Hurley, Wehby, and Feurer [36] used the Q-sort technique to find participants who held three distinctive viewpoints about valued social competence behavioral goals. Q methodology combines the advantages of qualitative research with a quantitative analysis [37]. A small sample was used in view of the limited literature on the research method [38]. The Q methodology is used to explain the main viewpoints favored by specific participants, regardless of the number [35]. It is used to study the subjective viewpoint of the participant more effectively [39] (pp. 23–25). Wheeler and Montgomery [33] explored the views of students on learning mathematics and used the Q methodology to analyze their experience of education and their beliefs. Chen, Chen, and Chen [40] used the Q methodology to discuss students’ perception of collaboration in flipped classrooms. Ramlo [41] used it to study campus stakeholders’ different perspectives of free speech. Walker, Lin, and McCline [42] made the Q methodology more accessible by using the Q-Perspectives® Online tool in face-to-face, flipped, and online classrooms.
The Q methodology is composed of five stages in this study [39] (pp. 27–28).
Step 1: An issue domain (concourse of communication), linguistic source was the sample in this study (i.e., Q-sort items), which was collected from written narratives of students who enrolled in this course management system.
Step 2: Sample participants (i.e., person samples or P-set) were selected for the Q sorting to reflect the students’ opinion of digital badges in blended learning using a content analysis approach. Q statements marked as unclear were rephrased with the assistance of an expert cooperative until all items were rated as clear.
Step 3: The participants’ subjectivity was expressed by modeling their viewpoints through the operational medium of a Q sort. The conditions of instruction requested for agreement (+4) and disagreement (−4) were used for rank-ordering and were employed to operationalize hypothetical constructs and categories [35]. Positive scores were placed to the right of the 0 in ascending order and negative scores to the left, and items that were neutral, ambivalent, or uncertain for participants were placed in the middle.
Step 4: The Q-sort data consisting of the intercorrelation of Q sorts as variables was analyzed, and a factor analysis was conducted on the correlation matrix based on Stephenson’s original formula.
Step 5: Finally, the factors were interpreted “in terms of factor scores rather than in terms of factor loadings (as is typical in R methodology)” based on the Q methodology [43].

3.2. Context

The study took place in the context of an obligatory undergraduate course offered to junior students by the Department of Journalism in a Chinese mainland university. The course contained fifty-four class periods, including thirty-six periods of theoretical teaching and eighteen periods of computer experiments twice a week (three periods per week of one time). The duration of the course was almost 18 weeks. The plan was to arrange six subjects and five homework assignments, using a mixed online and offline teaching model. The Moodle course management platform (see Figure 1) was designed to support self-directed and collaborative learning activities and to provide a participatory platform for teachers and students to submit assignments; to grade and award badges; and to contribute, share, and give feedback. The researcher utilized some badges ranking plug-ins that Moodle supports to build a competitive mechanism.
When Moodle had been set up, each student was given an account number (i.e., ID). Teachers created criteria with modules of digital badges and uploaded their class resources, such as videos or audio flips, images, and other types of course files to the platform. The process of awarding digital badges and the blended-learning for this study consisted of three steps: Firstly, the teachers introduced the online learning system to confirm the course tasks and rules, and students were given a Welcome Badge when they first entered the system; others were awarded depending on their achievement during the course. Secondly, face-to-face lectures, discussions, group reports, and other activities were organized. The difficulty of the task was gradually increased, and the time allotted to complete them was adjusted accordingly. Thirdly, the ranking scores were published (see Figure 2).
Students who achieved a great cumulative score for thematic activities were awarded a Participation and Completion badge, and finally, all the participants submitted group and individual assignments, completed a peer review (see Figure 3), and shared their experience. Assessment badges, Knowledge talent badges, Grade badges (Novice, Adept, Apprentice, and Professor), Independent learning, Great team badges, and Communication talent badges were issued, until the end of the six topics.
The operation and badge rules were explained and posted on the website at the beginning of the class in order to eliminate the impact of technical barriers on the participants’ initial exposure to the badge concept. The badge design criteria was provided in accordance with the goal achievement theory and Malone’s theory of intrinsic motivation in educational games [44] (See Table 1), and the badge icons system was developed based on Piha’s badge component map [45]. The Mozilla Open Badges platform (https://www.openbadges.me/designer.html) was used to design the badge icon and the Credly (https://credly.com/badge-builder#badge_text) platform was used to issue and earn digital badges for social media.

3.3. Participants

The subjects in this study were 26 male and 28 female students (n = 54), who participated in blended learning activities. Based on previous enrollment information, they were expected to have experience with multimedia animation design and production. Students’ information literacy was one of the factors that influenced the effect of the badges. Since demographic variables were important factors to consider in this study, the participants were asked to provide them in the registration system.
When users have a clear goal, their performance improves [46]; hence, it was important to clarify the task content in the course planning and curriculum design. Ranking scores (see Figure 2), a badge ladder (see Figure 4), and a plug-in that Moodle supports were utilized to generate a competitive mechanism in this course to motivate the participants to take more responsibility.
Both teachers and students could see each other’s ranking on the Badge Ladder during the whole term, either the number of students owning a badge or the details of badges owned by each student (see Figure 5).

3.4. Instrument

Q statements (i.e., a Q set) form the sample in a Q methodology. The teachers collected the statements of the students who participated in the blended learning activities and selected sentences as representative of the use of various strategies in the form of a Q sample. They sought permission to collect the students’ views of the use of digital badges in this course by asking questions and establishing feedback questionnaires on the course’s online platform. The open question was: How do you feel about the use of digital badges in this course? Please state the advantages and disadvantages in 8–10 sentences.
Forty-nine feedback statements were collected, and 36 Q statements were extracted from them to ensure that the number of availing and unavailing statements was balanced. The final Q-statements were selected from a comprehensive reflection of students’ views on digital badges and divided into four themes (i.e., teaching and learning, learning evaluation, communication, and technological issues) with the 36 statements representing the main ideas of each theme as the research instrument. The participants were then asked to model their opinions with these items in a modified rank-ordering procedure (i.e., Q sort). The Q method involves Q sorting, a method of data collection and a factor analysis to assess subjective (qualitative) information [47].

3.5. Procedure

The teachers were given permission to collect the data through scheduled class meetings. One researcher briefly explained the rights of the participants, the purpose of the study, and the instructions and steps in the Q-sorting process, and any questions were answered. Learners who agreed to participate were given a set of Q cards, a score sheet (see Figure 6), and a request for demographic information including name, age, gender, grade, college, and department. They were asked to sort the statements on a nine-point forced distribution for each Q sort. They indicated their answer on a continuum from most agree (+4) to most disagree (−4) to the question, what do you think about digital badges?

3.6. Data Analysis

The PQMethod 2.35 program was downloaded from http://schmolck.org/qmethod and used to facilitate Q-sort data entry, correlation and factor analyses, and theoretical rotation of factors and to calculate the factor scores. Twenty-two valid sorts were entered for analysis. A correlation matrix was generated to represent the relationship between each pair of Q sorts before being submitted to a centroid factor analysis and judgmental (or manual) rotation for interpretation.

4. Results

A combination of auto- and hand flagging resulted in twenty-two valid sorts, and a three-factor solution was finally chosen. These factors represented three related views of using digital badges in blended learning classrooms. As can be seen from Table 2, the factor analysis indicated three broad classes of Q sorts, factors 1, 2, and 3; however, the second was bipolar [35,43], which indicated four different understandings of digital badges. Factor 1 contained 8 sorts, factor 2a (the negative pole of the factor) contained the views of Persons 15, 5, and 10; in contrast, factor 2b (the positive pole of the same factor) contained the views of Persons 17, 19, 21, 16, and 7 sorts on the third factor contained diametrically opposed views of Person 6 advocating an opposing viewpoint (factor loading of −0.48). Neutral learners, Extreme learners, and Skeptical learners represented each factor (see Table 2). The Q methodology primarily utilizes statements to define a theoretical factor for interpretation [43]. The statements that were ranked from most agree (+4) to most disagree (−4) helped to explain the characteristics of each factor with factor scores (see Table 3), while the negative pole of the bipolar factors advocated an opposing viewpoint. All groups identified the mutual evaluation in the group as an effective instructional strategy in blended learning.
Factor 1 Neutral learners (five males and three females) contained 8 sorts, 36% of the 22 students’ Q sorts. They acquired 2 Professor Badges, 4 Apprentice Badges, and 2 Novice Badges during their learning performance. This group generally believed that evaluating results with badges did not represent their true ability (+3), but they thought that digital badges had a clear objective for learning outcomes, namely, to learn better (+4). One observed that “Digital badges can better assess learners’ skills, provide an evaluation and certification at the same time as clarifying the learning process.” However, they disagreed that the ranking of badges had a negative effect (−4); rather, they described them as an innovative and future development trend compared to traditional classroom education and the credit system. One of this type of learners said “This is an innovative method of teaching that should be encouraged and promoted.”
Factor 2 Extreme learners (five males and two females) had both positive and negative significant loadings, which was a “bipolar” factor, when Factor 2a (the negative pole of the factor) contained 3 sorts (14% of the 22 students’ Q sorts) and factor 2b (the positive pole of the same factor) contained 4 sorts (18%). They acquired 2 Professor Badges, 3 Adept Badges, and one of each of the other badges during their learning performance. As can be seen from Table 2, extreme learners with a negative pole (Factor 2a) who disagreed with the item rankings had the most negative perception of badges. Extreme learners with a positive pole (Factor 2b) had the most positive perception that badges encouraged and cultivated a sense of competition more actively in blended learning. One student in this group responded to the open question, “Digital badges can well evaluate online learners’ skills and achievements, so that they can have a clearer understanding of their strengths and weaknesses.” At the same time, the mutual evaluation function provided by digital badges can make it more convenient for learners to know about the homework of other learners. These students appear to possess the characteristic of a Hunter based on their strong agreement with the statement [24], “badges are a recognition of skills and encouragement (+4) which helps to raise their competitive spirit and improve their inadequacy (+4)”. Therefore, Factor 2b agreed that badges cultivate a sense of competition and more active blended learning (+3), while the negative group, Factor 2a, agreed with the converse configuration.
Factor 3 Skeptical learners (two males and five females) were composed of 7 sorts (32% of all 22 students’ Q sorts). They acquired 1 Adept Badge and 2 of each of the other badges during their learning performance. They were deep and active participants in blended learning, although they thought that the networking and user experience were inconvenient (+4) and that the display position of personal badges was not obvious enough (+4). Further evidence could be found in the responses to the open question:
“Website page layout is not clear, and user experience is not good, although it is easy to download information. As a new attempt, digital badges are worth promoting.”
They thought that neither the convenience of online learning (−3) nor the fact that digital badges represented acquired skills and achievements were encouraged enough (−4). They also disagreed that digital badges were less satisfying than physical rewards (−2). In other words, they thought that digital badges were at least as satisfying as physical rewards. In addition, Person 6 advocated an opposing viewpoint when all the item rankings in the same configuration were reversed.
The results showed that the three groups equally agreed or disagreed with several statements. Although all three groups had different opinions of the digital badges in blended learning, there was a consensus on Statements 4, 7, 11, 12, and 34 (see Table 4).

5. Discussion and Conclusions

The Q methodology is an approach that can be employed to understand students’ perspective of the use of digital badges in a blended learning course in higher education. The Q methodology was used for the data collection and analysis in this study to explore the perspective of students with different types of characteristics of the use of digital badges in blended learning. A survey was used to collect the students’ opinions of the functions of digital badges so that they can be improved to motivate students’ learning by focusing on an instructional strategy.

5.1. Discussion

5.1.1. Student’s Perception of the Digital Badges Mechanism

The participants’ opinion of the performance of digital badges in blended learning was demonstrated using a Q-sort survey. The students were divided into groups of Neutral learners, Extreme learners, and Skeptical learners and the results indicated that they all identified the mutual evaluation of the group as an effective instructional strategy in blended learning, even though some students had a negative perception of it. All the groups equally agreed that peer assessments increased the chances of learning, which supports the observation of Riffell and Sibley [26]. Furthermore, digital badges encourage Extreme learners to actively participate in blended learning, but these learners did not all feel that digital badges were less satisfying than physical prizes, as Skeptical learners did, which supports the opinion of Abramovich and Wardrip [20] that badges are a positive tool to motivate students. Moreover, Skeptical learners thought that digital badges added fun to the interaction and communication, although they felt that the display position of personal badges was not obvious enough (+4). It is confirmed in this study that digital badges have a positive effect on blended learning, which is similar to the idea of Andrioni [48] that combining different forms of traditional and modern online learning leads to new ways of involvement and active participation in learning.
Another interesting viewpoint highlighted by the Q methodology was involved the Factor 1 and Factor 3 groups of leaners. Both groups agreed that the evaluation results of digital badges do not reflect the true level (+3). They also agreed that badges were not energizing unless teachers emphasized them in class (+2). The Q sort demonstrated the individuality of the factors. Factor 1 felt neutral or uncertain (0) about the encouragement of badges that represent acquired skills and achievements, while the Factor 3 felt dissatisfied to the degree of grading it a −4. The factor 2b groups expressed the opposite view with +4 satisfactory. The result of this part of the study is consistent with Hanus and Fox’s [18] findings, which suggest that a competitive context or other digital badge mechanisms may harm educational outcomes. This viewpoint is inconsistent, since it simultaneously provides useful information for higher education administrators on where to focus improvements to the teaching incentives.
In addition, all the groups had a positive opinion of the use of digital badges in blended learning. The factor 1 group believed that digital badges have a clear objective of enabling them to learn better. The factor 2 group was neutral about the ranking rules of digital badges in blended learning activities, and felt that digital badges were novel and unique with a bright design. For the latter, badges represent their achievements and stimulate them. The factor 3 group agreed that digital badges are easy to use but that they reduce learning supplies (+1).

5.1.2. Effect of a Digital Badges Mechanism on Blended Learning

A further analysis of the consensus statements between factor types showed that all the groups agreed that peer assessment increases the chances of learning from each other, and they were all interested in the interactive activities to obtain digital badges, which increased their enthusiasm. This result is consistent with Riffell and Sibley’s findings [26]. It was also found that the mechanism of digital badges enhanced participants’ motivation and that rewards based on achievements increased their intrinsic motivation to learn. Factor 3 expressed dissatisfaction with the online learning environment, and one of them pointed out his dislike of computer learning, but the application of digital badges in education requires teachers and students to have basic knowledge of computer technology. Therefore, it is proposed that university educators need to optimize the availability of the university network and technical support. Furthermore, almost all the participants described digital badges as authentic and innovative compared to traditional classroom education and credit systems, although some had a negative or mediocre perception of them, stating that they were less prestigious than a certificate of completion. However, digital badges, as a form of microscopic certification, could provide a more detailed framework to assess college students’ knowledge and skills, could provide evidence of their learning, and could demonstrate their ability to respond to significant change. Digital badges help to capture a learning performance that is not recognized by formal academic accreditation processes. At the same time, they enable students to obtain overall information in a more timely fashion that a final exam by being an informal way to receive feedback.
These digital badges can be a more innovative and authentic means of instruction and assessment in blended learning classrooms. Their functions are determined through this course practice, and some suggestions for teaching are proposed. The researchers also argue that the results derived from these participants in blended learning practice may be influenced by external factors in field settings [49], such as the Moodle learning platform.

5.2. Conclusions

A digital badge mechanism was designed in this study and implemented in a blended learning environment in order to determine college students’ perception of digital badges. A research framework was proposed to investigate the objective of digital badges, and a leaderboard ranking was used to examine the participants’ motivation to learn. Three groups of Neutral learners, Extreme learners, and Skeptical learners gave their opinion of the use of digital badges. Digital badges make Factor 2b groups actively participate in blended learning because they thought that badges represented their achievements stimulated them and helped to foster a sense of competition and to improve their shortcomings. The teachers of this group need to pay attention to reforming traditional approaches and blended learning classroom settings.
Almost all the participants though that digital badges were authentic and innovative, although some participants (Factor 3) had the negative perception that they were mediocre and less prestigious than a certificate of completion. Factor 3 participants suggested that the display position of personal badges was not obvious enough and that the network should be more convenient for a good user experience, especially in blended learning.
Furthermore, focusing on the design of criterion rather than on the competition based on badge ranking can be effective in motivating individuals to do their best. According to Factor 3, the acquisition of knowledge is more conducive than the acquisition of a badge to increasing their intrinsic motivation to learn.

5.3. Limitations and Future Work

One limitation of this study involves the structure of the sample Q statements, although various strategies were used in the form of a Q sample. Another limitation was that the use of Q sort forces students to express their feelings and the choice of statements that represent their real opinions. Although the Q method lacked generalizability based on its complex research process, it still provided the instructors with an abbreviated opinion of the college students through the transferability of the Q methodology. Furthermore, since these results were only based on the perspective of students at a local mainland China university, the researchers wish to consider widening the educational area in the future.

Author Contributions

All authors contributed equally to this paper.

Funding

This research was funded by the National Social Science Fund of China, grant number 17BSH132.

Acknowledgments

Technical support of this project was partially provided by Huaiyin Institute of Technology. The authors would like to thank Tung-Liang Chen and anonymous reviewers of this journal and the ECEI 2019 for their encouragement and guidance throughout this project.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Liu, D.; Huang, R.; Wosinski, M. Future Trends of Smart Learning: Chinese Perspective. In Smart Learning in Smart Cities. Lecture Notes in Educational Technology; Liu, D., Huang, R., Wosinski, M., Eds.; Springer: Singapore, 2017; pp. 185–215. [Google Scholar]
  2. Halverson, L.R.; Spring, K.J.; Huyett, S.; Henrie, C.R.; Graham, C.R. Blended learning research in higher education and K-12 settings. In Learning, Design and Technology; Michael, J.S., Barbara, B.L., Marcus, D.C., Eds.; Springer: Cham, Switzerland, 2017; pp. 1–30. [Google Scholar]
  3. Graham, C.R. Blended learning systems. In The Handbook of Blended Learning: Global Perspectives, Local Designs; Bonk, C.J., Graham, C.R., Eds.; John Wiley and Sons: San Francisco, CA, USA, 2012; pp. 3–21. [Google Scholar]
  4. Mehta, N.B.; Hull, A.L.; Young, J.B.; Stoller, J.K. Just imagine: New paradigms for medical education. Acad. Med. 2013, 88, 1418–1423. [Google Scholar] [CrossRef]
  5. Jovanovic, J.; Devedzic, V. Open badges: Novel means to motivate, scaffold and recognize learning. Technol. Knowl. Learn. 2015, 20, 115–122. [Google Scholar] [CrossRef]
  6. Ellis, L.E.; Nunn, S.G.; Avella, J.T. Digital Badges and Micro-credentials: Historical Overview, Motivational Aspects, Issues, and Challenges. In Foundation of Digital Badges and Micro-Credentials; Ifenthaler, D., Bellin-Mularski, N., Mah, D.K., Eds.; Springer: Cham, Switzerland, 2016; pp. 3–21. [Google Scholar]
  7. Filsecker, M.; Hickey, D.T. A multilevel analysis of the effects of external rewards on elementary students’ motivation, engagement and learning in an educational game. Comput. Educ. 2014, 75, 136–148. [Google Scholar] [CrossRef]
  8. Boticki, I.; Baksa, J.; Seow, P.; Looi, C.K. Usage of a mobile social learning platform with virtual badges in a primary school. Comput. Educ. 2015, 86, 120–136. [Google Scholar] [CrossRef]
  9. Davis, K.; Singh, S. Digital badges in afterschool learning: Documenting the perspectives and experiences of students and educators. Comput. Educ. 2015, 88, 72–83. [Google Scholar] [CrossRef][Green Version]
  10. Jones, W.M.; Hope, S.; Adams, B. Teachers’ perceptions of digital badges as recognition of professional development. Br. J. Educ. Technol. 2018, 49, 427–438. [Google Scholar] [CrossRef]
  11. Blackburn, R.; Porto, S.; Thompson, J. Competency-based education and the relationship to digital badge. In Digital Badges in Education: Trends, Issues and Cases; Muilenburg, L.Y., Berge, Z.L., Eds.; Routledge: New York, NY, USA, 2016; pp. 30–38. [Google Scholar]
  12. Abramovich, S. Understanding digital badges in higher education through assessment. Horizon 2016, 24, 126–131. [Google Scholar] [CrossRef]
  13. Cameron, J.; Pierce, W.D.; Banko, K.M.; Gear, A. Achievement-Based Rewards and Intrinsic Motivation: A Test of Cognitive Mediators. J. Educ. Psychol. 2005, 97, 641–655. [Google Scholar] [CrossRef]
  14. Gibson, D.; Ostashewski, N.; Flintoff, K.; Grant, S.; Knight, E. Digital badges in education. Educ. Inf. Technol. 2013, 20, 403–410. [Google Scholar] [CrossRef]
  15. Dyjur, P.; Lindstrom, G. Perceptions and uses of digital badges for professional learning development in higher education. TechTrends 2017, 61, 386–392. [Google Scholar] [CrossRef]
  16. Abramovich, S.; Schunn, C.; Higashi, R.M. Are badges useful in education? It depends upon the type of badge and expertise of learner. Educ. Technol. Res. Dev. 2013, 61, 217–232. [Google Scholar] [CrossRef]
  17. The Potential and Value of Using Badges for Adult Learners. Available online: https://www.shrm.org/about-shrm/news-about-shrm/Documents/AIR_Digital_Badge_Report_508.pdf (accessed on 10 January 2019).
  18. Hanus, M.D.; Fox, J. Assessing the effects of gamification in the classroom: A longitudinal study on intrinsic motivation, social comparison, satisfaction, effort, and academic performance. Comput. Educ. 2015, 80, 152–161. [Google Scholar] [CrossRef]
  19. Buchem, I. Digital Badges at (Parts of) Digital Portfolios: Design Patterns for Educational and Personal Learning Practice. In Foundation of Digital Badges and Micro-Credentials; Ifenthaler, D., Bellin-Mularski, N., Mah, D.K., Eds.; Springer: Cham, Switzerland, 2016; pp. 343–368. [Google Scholar]
  20. Abramovich, S.; Wardrip, P. Impact of Badges on Motivation to Learn. In Digital Badges in Education: Trends, Issues and Cases; Muilenburg, L.Y., Berge, Z.L., Eds.; Routledge: New York, NY, USA, 2016; pp. 53–61. [Google Scholar]
  21. Carey, K.L.; Stefaniak, J.E. An exploration of the utility of digital badging in higher education settings. Educ. Technol. Res. Dev. 2017, 66, 1211–1229. [Google Scholar] [CrossRef]
  22. Hew, K.F.; Huang, B.; Chu, K.W.S.; Chiu, D.K. Engaging Asian students through game mechanics: Findings from two experiment studies. Comput. Educ. 2016, 92, 221–236. [Google Scholar] [CrossRef]
  23. Grant, S.L. History and Context of Open Digital Badges. In Digital Badges in Education: Trends, Issues, and Cases; Muilenburg, L.Y., Berge, Z.L., Eds.; Routledge: New York, NY, USA, 2016; pp. 3–11. [Google Scholar]
  24. Biles, M.L.; Plass, J.L. Good Badges, Evil Badges? Impact of Badge Design on Learning from Games. In Digital Badges in Education: Trends, Issues, and Cases; Muilenburg, L.Y., Berge, Z.L., Eds.; Routledge: New York, NY, USA, 2016; pp. 39–52. [Google Scholar]
  25. Carman, J.M. Blended Learning Design: Five Key Ingredients. Available online: https://www.it.iitb.ac.in/~s1000brains/rswork/dokuwiki/media/5_ingredientsofblended_learning_design.pdf (accessed on 10 January 2019).
  26. Riffell, S.; Sibley, D. Using web-based instruction to improve large undergraduate biology courses: An evaluation of a hybrid course format. Comput. Educ. 2005, 44, 217–235. [Google Scholar] [CrossRef]
  27. Bonk, C. Best Practices for Online and Blended Learning: Introducing the R2D2 and TEC-VARIETY Models. Available online: https://mushare.marian.edu/cgi/viewcontent.cgi?article=1013andcontext=ffdc (accessed on 13 January 2019).
  28. Bernard, R.M.; Borokhovski, E.; Schmid, R.F.; Tamim, R.M.; Abrami, P.C. A meta-analysis of blended learning and technology use in higher education: From the general to the applied. J. Comput. High. Educ. 2014, 26, 87–122. [Google Scholar] [CrossRef]
  29. Costello, E. Opening up to open source: Looking at how Moodle was adopted in higher education. Open Learn. J. Open Distance e-Learn. 2013, 28, 187–200. [Google Scholar] [CrossRef]
  30. Subhash, S.; Cudney, E.A. Gamified Learning in Higher Education: A Systematic Review of the Literature. Comput. Hum. Behav. 2018, 87, 192–206. [Google Scholar] [CrossRef]
  31. Ippoliti, C.; Baeza, V.D. Using digital badges to organize student learning opportunities. J. Electron. Res. Librariansh. 2017, 29, 221–235. [Google Scholar] [CrossRef]
  32. Yang, J.C.; Quadir, B.; Chen, N.S. Effects of the badge mechanism on self-efficacy and learning performance in a game-based English learning environment. J. Educ. Comput. Res. 2016, 54, 371–394. [Google Scholar] [CrossRef]
  33. Wheeler, D.L.; Montgomery, D. Community college students’ views on learning mathematics in terms of their epistemological beliefs: A Q method study. Educ. Stud. Math. 2009, 72, 289–306. [Google Scholar] [CrossRef]
  34. Ramlo, S. Mixed method lessons learned from 80 years of Q methodology. J. Mixed Method Res. 2016, 10, 28–45. [Google Scholar] [CrossRef]
  35. Watts, S.; Stenner, P. Doing Q methodology: Theory, method and interpretation. Qual. Res. Psychol. 2005, 2, 67–91. [Google Scholar] [CrossRef]
  36. Hurley, J.J.; Wehby, J.H.; Feurer, I.D. The social validity assessment of social competence intervention behavior goals. Top. Early Child. Spec. Educ. 2010, 30, 112–124. [Google Scholar] [CrossRef]
  37. Brown, S.R. Q methodology and qualitative research. Qual. Health Res. 1996, 6, 561–567. [Google Scholar] [CrossRef]
  38. Mrtek, R.G.; Tafesse, E.; Wigger, U. Q-Methodology and Subjective. J. Soc. Adm. Pharm. 1996, 13, 54–64. [Google Scholar]
  39. McKeown, B.; Thomas, D. Q Methodology, 2nd ed.; SAGE Publications: Thousand Oaks, CA, USA, 2013; Volume 66, pp. 23–25, 27–28. [Google Scholar]
  40. Chen, L.; Chen, T.-L.; Chen, N.-S. Students’ perspectives of using cooperative learning in a flipped statistics classroom. Aust. J. Educ. Technol. 2015, 31, 621–640. [Google Scholar] [CrossRef]
  41. Ramlo, S. Free speech on US university campuses: Differentiating perspectives using Q methodology. Stud. High. Educ. 2018. [Google Scholar] [CrossRef]
  42. Walker, B.B.; Lin, Y.; McCline, R.M. Q Methodology and Q-Perspectives® Online: Innovative Research Methodology and Instructional Technology. TechTrends 2018, 62, 450–461. [Google Scholar] [CrossRef]
  43. Brown, S. Political Subjectivity; Yale University Press: London, UK, 1980; Available online: https://qmethodblog.files.wordpress.com/2016/01/brown-1980-politicalsubjectivity.pdf (accessed on 13 January 2019).
  44. Malone, T.W. Toward a theory of intrinsically motivating instruction. Cogn. Sci. 1981, 5, 333–369. [Google Scholar] [CrossRef]
  45. Piha, S. Afterschool and Digital Badges: Recognizing Learning Where It Happens. In Digital Badges in Education: Trends, Issues and Cases; Muilenburg, L.Y., Berge, Z.L., Eds.; Routledge: New York, NY, USA, 2016; pp. 135–144. [Google Scholar]
  46. Jung, J.H.; Schneider, C.; Valacich, J. Enhancing the motivational affordance of information systems: The effects of real-time performance feedback and goal setting in group collaboration environments. Manag. Sci. 2010, 56, 724–742. [Google Scholar] [CrossRef]
  47. Dziopa, F.; Ahern, K. A systematic literature review of the applications of Q-technique and its methodology. Method Eur. J. Res. Method Behav. Soc. Sci. 2011, 7, 39–55. [Google Scholar] [CrossRef]
  48. Andrioni, F. Cross-European Perspective in Social Work Education: A Good Blended Learning Model of Practice. Sustainability 2018, 10, 1539. [Google Scholar] [CrossRef][Green Version]
  49. Gillard, L.A.; Lewis, V.N.; Mrtek, R.; Jarosz, C. Q Methodology to Measure Physician Satisfaction with Hospital Pathology Laboratory Services at a Midwest Academic Health Center Hospital. Lab. Med. 2005, 36, 361–365. [Google Scholar] [CrossRef]
Figure 1. The course management systems.
Figure 1. The course management systems.
Sustainability 11 02151 g001
Figure 2. A ladder for ranking scores.
Figure 2. A ladder for ranking scores.
Sustainability 11 02151 g002
Figure 3. A peer assessment activity.
Figure 3. A peer assessment activity.
Sustainability 11 02151 g003
Figure 4. A Moodle Badge Ladder plug-in.
Figure 4. A Moodle Badge Ladder plug-in.
Sustainability 11 02151 g004
Figure 5. A participant’s badges on the user details webpage.
Figure 5. A participant’s badges on the user details webpage.
Sustainability 11 02151 g005
Figure 6. A Q-sort response grid and answer sheet.
Figure 6. A Q-sort response grid and answer sheet.
Sustainability 11 02151 g006
Table 1. A list of the badge criteria in the course.
Table 1. A list of the badge criteria in the course.
BadgesNameCriteria and RulesI/GFQIssuerA
Sustainability 11 02151 i001Great team BadgeComplete teamwork: Two people in a group must submit Flash homework; total score of 80 or above.G1More than 5 badges in totalIrNo
Sustainability 11 02151 i002Completion BadgeSubmit work on time; complete the course activities.I3 3 badges at most IrNo
Sustainability 11 02151 i003Welcome BadgesLog into the system, register, and provide the required information.IIrgr1 badge per personIrNo
Sustainability 11 02151 i004Participation BadgeIssued to top 20 in course activities ranking. Rules: 5 points for log-in and 20 for each specified activity completed (including replies and comments in forum, evaluation, peer-assessment, feedback, download course resources, and other learning activities).I220 badges per timeIrNo
Sustainability 11 02151 i005Independent learning badgesComplete independent learning activities on time, including viewing course materials before class; complete pre-class test; and complete after-school test.I11 badge per personPNo
Sustainability 11 02151 i006Communication talent BadgeIssued to students who receive a peer review medalI11 badge per personPS
Sustainability 11 02151 i007Knowledge talent BadgeIssued to students who achieve more than 80 points in knowledge test.I11 badge per personIrNo
Sustainability 11 02151 i008Assessment BadgeComplete peer review.I11 badge per personIrS
Sustainability 11 02151 i009Grade Badge: NoviceFinal examination badges.
Novice Grade Badge awarded for less than 70%.
Adept Grade Badge awarded for between 71% and 79%.
Apprentice Grade Badge awarded for between 80% and 89%.
Professor Grade Badge awarded for more than 90%.
IOnce at the end1 badge per personIrNo
Sustainability 11 02151 i010Grade Badge: Adept
Sustainability 11 02151 i011Grade Badge: Apprentice
Sustainability 11 02151 i012Grade Badge: Professor
* I: Individuals; G: Groups; F: Frequency; Irgr: Irregular; Q: Quantity; Ir: Instructor; P: Peers; A: Anonymous; S: Self-Selection.
Table 2. The factor loading and demographic variables for each type of learner *.
Table 2. The factor loading and demographic variables for each type of learner *.
Person No.Factor 1Factor 2Factor 3Gender
1369 Male
2066 Male
1463 Female
963 Female
158 Female
358 Male
754 Male
2247 Male
15 −64 Female
5 −58 Male
10 −85 Male
17 49 Female
19 80 Male
21 37 Male
16 83 Male
6 −48Female
11 87Female
12 87Female
18 39Male
2 76Male
8 40Female
4 53Female
* Only significant loadings shown (p 0.01); decimals omitted.
Table 3. Students’ statement scores by factor types.
Table 3. Students’ statement scores by factor types.
Factor StatementsFactorRankings (*)
123
Factor 1 Neutral learners
33Digital badges have a clear objective for learning outcomes; I can learn better.4−1−1
35Digital badges do not represent a person’s true ability.3−10
28Digital badges and the websites have insufficient attention.3−30
1Digital badges make me actively participate in blended learning.13−2
24The network is not convenient, and user experience is not good.1−34
2Digital badges are less satisfying than physical rewards0−4−2
31Digital badges add to the fun of interaction and communication.02−1
21Badges represent acquired skills and achievements, which encourages me.04−4
10Obtaining badges is conducive to communication with classmates−132
19Digital badges improve my learning.−221
6Digital badges are novel and unique with a bright design.−33−1
25Ranking of badges has a negative effect on me.−4−1−1
Factor 2 Extreme learners
21Badges represent my achievements and stimulate me.04−4
15Badges help to foster a sense of competition and improve my shortcomings.−14−3
1Digital badges make me actively participate in blended learning.13−2
6Digital badges are novel and unique with a bright design.−33−1
31Digital badges add fun to interaction and communication.02−1
36Blended learning tends to be compared to traditional education.423
27Digital badges improve interaction and teaching efficiency.−21−1
29Digital badges refer to grades.−30−4
3There is unfair phenomenon of low scoring when students evaluate each other.0−10
20The evaluation results of digital badges do not reflect the true level.3−23
30Digital badges are not famous or widely used.1−20
14Badges are not motivating unless teachers emphasize them in class.2−22
28Digital badges and the website have insufficient attention.3−30
24The network is not convenient, and user experience is not good.1−34
2Digital badges are less satisfying than physical rewards0−4−2
Factor 3 Skeptical learners
24The network is not convenient, and user experience is not good.1−34
5The display position of a personal badge is not obvious enough−2−14
16Acquiring a badge rather than acquiring knowledge is not conducive to an intrinsic motivation to learn.−202
18Computer study is troublesome. I don’t like learning on the internet.−4−31
22Digital badge icons are drab and imperfect.−1−21
28Digital badges and the website have insufficient attention.3−30
6Digital badges are novel and unique with a bright design.−33−1
31Digital badges add fun to interaction and communication.02−1
1Digital badges make me actively participate in blended learning.13−2
2Digital badges are less satisfying than physical rewards.0−4−2
13Digital badges give timely feedback on learning.20−2
17Online learning is convenient and conducive to independent learning.11−3
21Badges represent acquired skills and achievements, which encourages me.04−4
* Item rankings: +4, most agree; 0, neutral or uncertain; −4, most disagree in this sample.
Table 4. The statements on factor types that achieved a consensus.
Table 4. The statements on factor types that achieved a consensus.
Factor StatementsFactorRankings (*)
123
4Peer assessment increases the chances of learning from each other.233
7The ranking rules of digital badges in blended learning activities are reasonable and clear.000
11Digital badges, like Red Flowers for primary school students, are not suitable for college students.−3−4−3
12Digital badges are easy to use, but they reduce learning supplies.111
34I am not interested in the interactive activities of obtaining digital badges and have little enthusiasm for them.−3−3−3
* Item rankings: +4, most agree; 0, neutral or uncertain; −4, most disagree in this sample.
Back to TopTop