Digital Skills Framework in Higher Education ^†

Abstract

Digital skills should be incorporated in the curriculum as the skills are essential attributes graduates should acquire before joining the workforce. Understanding the digital skills framework and abilities among learners may help educators create suitable learning outcomes, activities, and assessments to enable the learners to acquire these skills. This study shares expert judgements on a digital skills instrument (DSI) on content validation framework according to sufficiency, clarity, coherence, and relevance. A pre-test involving 218 students from various disciplines was conducted and Cronbach’s Alpha was used to measure the validity and reliability of the DSI. The overall result indicated a strong reliability. Clarity was the characteristic garnering the highest value, followed by coherence, relevance, and sufficiency. The DSI was revised and 28 items encompassing five components were proposed following the feedback of: information literacy, computer and technology literacy, digital communication/collaboration skill, digital identity and well-being, and digital ethic. Based on the pre-test, all the components indicated high reliability, ranging from 0.847 to 0.958, and all of the items were retained to measure digital skills. Overall, the experts’ evaluations and pre-tests suggested that the DSI and its five components as the digital skills framework can be used for measuring digital skills.

1. Introduction

1.1. Digital Skills Proficiency: Influencing Factors

Having proficient digital skills is important in this era of modernity, as technology has become a necessity to keep the world functioning smoothly. Because of this, digital literacy has been an issue of concern regarding whether upcoming generations will be adequately equipped with digital knowledge. The ability to locate, evaluate, and produce clear information across a range of media platforms is referred to as digital literacy [1]. There are many factors that influence one’s digital skills proficiency. Technical skills, critical understanding, and communicative abilities are some of the factors listed [1], with critical understanding being the most important. On the other hand, students’ level of education also has an impact on the development of their digital skills [2]. Additionally, individuals who have a great critical understanding are also able to undertake a more in-depth analysis and evaluation of the information that they find in the media [1]. Another factor is technical skills, defined by Dewi et al. [1] as “the ability to access and operate media”. As the factor implies, individuals possessing this skill are good at computer and internet skills, the use of media, and even advanced use of the internet. Another factor is one’s ability to communicate. It has been suggested that individuals who have mastered the skill are able to interact with others and participate online, while also being able to produce media content. The other factor that affects digital skills proficiency that was discussed by Van Deursen et al. [3] is age. It has been postulated that a person’s age may have an impact on their digital competency. On the other hand, digital skills have facilitated the growth of big data analysis. Because decision-making based on big data analysis is now widespread, there is a compelling need to improve digital skills [4].

1.2. Digital Skills as Graduate Attributes in the Curriculum Framework

Digital skills are among the future skill sets required of all graduates in order to embrace challenges and stay relevant [5]. They are known to provide the younger generation with the skills required in the 21st century, ensuring their survival in an unfamiliar environment. Therefore, those who have these skills are more employable, productive, and creative, and they will remain safe and secure in the new digital economy landscape. In other words, digital skills are needed as they encompass the ability to acquire and manage information through digital devices, communication applications, and networks [6]. The new skillset and the change in the education landscape also prompted the Malaysian Qualification Agency (MQA) to revise the 2010 qualification framework [7] so that it would emphasize the digital attribute. This was done to ensure that students at higher education institutions would be taught and trained in the skills, in addition to other skills such as numeracy, communication, interpersonal, and entrepreneurial skills. This new trajectory is aligned with the global agenda of preparing graduates to thrive in a volatile, uncertain, complex, and ambiguous (VUCA) environment. For this reason, since April 2019, the Malaysian Qualification Agency (MQA) has required all universities in Malaysia to comply with this new framework for their new curricula.

To meet this requirement, educators must not only expose their students to digital skills, but they must also determine the levels of their students’ digital skills in the first place in order to help the former to design materials that can be used to equip the latter with the skills. This is important, as it can help them to develop appropriate assessments and approaches to achieve intended learning outcomes according to different disciplines or courses. Most importantly, the practice also helps them to understand their students better and to design a curriculum that meets their needs.

The Malaysian Qualification Framework (MQF) has established a general definition and attributes as the minimal criteria for higher education providers to develop their curricula. The MQF defines digital skills as the ability to use information/digital technology to support work and studies. The skills include gathering and storing information, processing data, using applications for problem solving and communication, and using digital skills ethically or responsibly. Four components were extracted from the general definition: information literacy, computer and technology literacy, visual literacy, and digital communication/collaboration skills.

The focus of this paper is to seek validation from experts based on their views on the digital skills instrument that contains the four components, and to establish the reliability of final items based on the pilot study. The aim is that the research outcome could guide curriculum developers and course designers to design appropriate content knowledge, delivery activities, and assessments that could instill and equip students with both skills that are aligned with the MQA description and the aspiration of the Malaysian Government who wishes to produce graduates with 21st-century skills.

2. Materials and Methods

This study is a cross-sectional study that uses the quantitative method to construct a digital skills framework. This study involves the development of questionnaire items, validation of the questionnaire process and testing the reliability of the questionnaire. The validation of the questionnaire involves expert content validation. For determining the reliability of the questionnaire, internal reliability was conducted and analyzed using Cronbach’s Alpha.

2.1. Digital Skills Instrument (DSI)

There are 22 items that are divided according to the four components in the digital skills instrument (DSI), namely, information literacy, computer and technology literacy, visual literacy, and digital communication/collaboration skills.

Table 1. Initial component of digital skills instrument (DSI).

Component	Number of Items
Information literacy	11
Computer and technology literacy	4
Visual literacy	3
Digital communication/collaboration skills	4
Total	22

shows a breakdown of the number of items for each of the components.

2.2. Expert Judgment Content Validation

Evaluation through expert judgements involves asking several individuals to make a judgement on an instrument or to express their opinions on a particular aspect (refer to

Table 2. Judges and field of expertise.

Expert	Field of Expertise	Research/Teaching Experience (Years)
1	Language testing (social sciences)	18
2	Measurement and evaluation (social sciences)	20
3	Digital literacy (science and technology)	21
4	Artificial intelligence (science and technology)	15
5	Instructional communication and new media (social sciences)	16
6	Psychometric assessment (social sciences)	25
7	Pharmacy (health science)	20
8	Law and criminal justice (social sciences)	21

for the list of experts). The experts play a crucial role in explaining, adding, and/or modifying the required components. Initially, two experts in measurement and evaluation from a comprehensive and a research university were selected to evaluate the digital instrument. Then, feedback from eight experts was also taken into account in the refinement of the DSI. Overall, two digital content experts, three senior lecturers from science and technology, and three experts from social sciences were involved to ensure the validity of the DSI before its final version was administered to the actual sample.

The volunteer experts came from a variety of disciplines, including science, technology, and social sciences (non-science and technology). The criteria for selection were based on either their experiences in curriculum development, in holding any academic post, or in having more than 10 years research or teaching experience. According to the standard and MQF, digital skills must be measured across disciplines and at all qualification levels. Therefore, experts from different fields should be more than able to provide better descriptions of appropriate digital attributes for different curriculum or academic programs.

All of the items were divided into the categories of sufficiency, clarity, coherence, and relevance (refer to

Table 3. Categories, indicators, and scale.

Category	Scale	Indicators
Sufficiency The items within the same dimension should suffice to measure this dimension.	4	The item is sufficient to measure the dimension
	3	The item measures some aspects of the dimension but does not represent the full dimension
	2	A few items must be added to fully assess the dimension
	1	The item is insufficient to measure the dimension
Clarity The items can be understood easily. (i.e., syntax and semantics are appropriate).	4	The item is sufficient to measure the dimension
	3	The item measures some aspects of the dimension but does not represent the full dimension
	2	A few items must be added to fully assess the dimension
	1	The item is insufficient to measure the dimension
Coherence The items are logically related to the dimension or indicator they are measuring.	4	The item is sufficient to measure the dimension
	3	The item measures some aspects of the dimension but does not represent the full dimension
	2	A few items must be added to fully assess the dimension
	1	The item is insufficient to measure the dimension
Relevance The items are essential or important. (i.e., they must be included).	4	The item is sufficient to measure the dimension
	3	The item measures some aspects of the dimension but does not represent the full dimension
	2	A few items must be added to fully assess the dimension
	1	The item is insufficient to measure the dimension

Source: adapted from Escobar-Pérez and Cuervo-Martínez [8].

for the scales and indicators of the categories). All of the experts were required to rate each item using a scale ranging from 4 to 1. The experts were also encouraged to give comments in the open space in the instrument.

2.3. Pilot Study

After the questionnaire was assessed by the experts, the resulting instrument was pilot tested with potential respondents to determine the reliability of each component and item in the final instrument. An online survey was done and randomly distributed to students from various faculties and fields.

3. Results

3.1. Content Validation by Expert Judgment

Cronbach’s Alpha values were measured to check the data reliability and internal consistency, and values higher than 0.6 were considered to be acceptable. Values ranging from 0 to 1 with greater values indicate higher internal consistency measures on the opinions and perceptions among the respondents. The reliability of the instrument focusing on the experts’ evaluations on the four constructs, namely sufficiency, clarity, coherence, and the relevance of the instrument were analyzed using Cronbach’s Alpha. The overall result indicated strong reliability (0.915 ≤ Cronbach’s α ≤ 0.95) for the 22 items.

Table 4. Cronbach’s Alpha and descriptive statistics for each of the categories.

Category	Cronbach’s Alpha	Mean	Standard Deviation
Sufficiency	0.915	3.56	0.446
Clarity	0.944	3.40	0.532
Coherence	0.941	3.55	0.530
Relevance	0.934	3.52	0.529

presents the descriptive statistics (means and standard deviations) of the categories of the perceptions of digital skills. The means ranged from 3.40 to 3.56.

Table 5. Cronbach’s Alpha values for each of the components and characteristics of digital skills.

Construct	Characteristics	Characteristics	Cronbach’s Alpha	Average
Digital Skills	C1—Information Literacy	Sufficiency	0.924
		Clarity	0.961
		Coherence	0.949	0.94
		Relevance	0.926
	C2—Computer and Technology Literacy	Sufficiency	0.966
		Clarity	0.860
		Coherence	0.934	0.93
		Relevance	0.950
	C3—Visual Literacy	Sufficiency	0.855
		Clarity	0.732
		Coherence	0.812	0.82
		Relevance	0.902
	C4–Digital Communication/ Collaboration Skill	Sufficiency	0.736
		Clarity	0.650
		Coherence	0.823	0.76
		Relevance	0.837

indicates the Cronbach’s Alpha values for each component and characteristic based on the judgments by the experts.

It can be seen that the Cronbach’s Alpha values for information literacy ranged from 0.924 to 0.961 and the Cronbach’s Alpha values for computer and technology literacy ranged from 0.860 to 0.966, indicating strong internal consistencies compared with the other two components, namely visual literacy, whose values ranged from 0.732 to 0.902, and digital communication/collaboration skills, whose values ranged from 0.650 to 0.837. It could be concluded that the experts’ scores on their perceptions showed consistency towards digital skills on information literacy and computer and technology literacy. On the other hand, the respondents’ perceptions of visual literacy showed a moderate level of consistency compared with digital communication/collaboration skills.

Further analysis on the enhancement of the questionnaires was conducted by considering the input given by the experts. For example, the questionnaire was improved by restructuring the order of the items, the sentence structure, and by adding two components, as well as removing the visual literacy component. Some of the items in the component were also merged with the other components. Consequently, 28 items representing five components in the DSI, namely information literacy, computer and technology literacy, digital communication/collaboration skills, digital identity and well-being, and digital ethics were developed.

3.2. Pilot Study

Distribution of Respondents

Figure 1 below shows the distribution of respondents based on college and faculties. Overall, 218 students from various academic programs participated in the online survey. Specifically, the highest percentage of the respondents came from the Faculty of Accountancy (44.04%), followed by students from the Faculty of Computer and Mathematical Sciences (26.61%) and from the Faculty of Film Theatre and Animation (12.84%). The results of the pilot study were analyzed using Cronbach’s Alpha.

Table 6. Item-total statistics: item and values in five components of perceptions on digital skills.

Component	Item	Scale Mean If Item Deleted	Scale Variance If Item Deleted	Corrected Item-Total Correlation	Cronbach’s Alpha If Item Deleted	Cronbach’s Alpha Overall
C1-Information Literacy	C1.1	110.49	235.726	0.687	0.966	0.957
	C1.2	110.52	235.264	0.738	0.966
	C1.3	110.42	235.904	0.754	0.966
	C1.4	110.60	232.683	0.804	0.965
	C1.5	110.57	232.689	0.794	0.965
	C1.6	110.71	234.522	0.736	0.966
	C1.7	110.76	232.793	0.758	0.965
	C1.8	110.69	233.006	0.790	0.965
	C1.9	110.80	231.848	0.780	0.965
	C1.10	110.59	234.732	0.717	0.966
	C1.11	110.72	232.590	0.749	0.966
	C1.12	111.07	235.193	0.607	0.967
C2-Computer and Technology Literacy	C2.1	110.51	233.928	0.747	0.966	0.895
	C2.2	110.99	232.968	0.620	0.967
	C2.3	110.69	233.790	0.741	0.966
	C2.4	110.76	233.300	0.736	0.966
	C2.5	11.13	234.324	0.572	0.967
C3-Digital Communication/Collaboration Skill	C3.1	110.69	232.702	0.746	0.966	0.919
	C3.2	110.67	233.016	0.718	0.966
	C3.3	110.89	231.938	0.692	0.966
	C3.4	110.61	234.082	0.701	0.966
C4-Digital Identity and Well-Being	C4.1	110.31	236.879	0.616	0.966	0.847
	C4.2	110.63	233.745	0.654	0.966
	C4.3	110.40	235.882	0.629	0.966
C5-Digital Ethics	C5.1	110.39	234.846	0.688	0.966	0.942
	C5.2	110.59	233.691	0.714	0.966
	C5.3	110.44	235.408	0.674	0.966
	C5.4	110.43	235.167	0.665	0.966

shows the proposed number of items by components, as well as the item-total statistics, which include the scale mean if an item is deleted, scale variance if an item is deleted, and corrected item-total correlation and Cronbach’s Alpha if an item is deleted. As eliminating items from each component reduces the respective Cronbach’s Alpha, most items were retained. Moreover, the overall Cronbach’s Alpha analysis revealed a strong internal consistency among respondents for each component. The overall analysis and the value of Cronbach’s Alpha if an item was deleted showed very high values with alpha values greater than 0.8.

4. Discussion

The findings of this study led to the development of a digital skills framework. The construction of the digital skills framework in this study was subjected to a process of recurrent validation. There were two stages in the process. In the first stage, two experts provided extensive feedback on the content for four constructs (information literacy, computer and technology literacy, visual literacy, and digital communication/collaboration skills). The result of the first stage validation shows that two components, “visual literacy” and “digital communication/collaboration skills”, had a moderate level of consistency. Only “visual literacy” was removed from the list of components. “Digital communication/collaboration skills” remained as a component as they are the skills that are currently needed. In the digital world, all learners or users must be able to collaborate with others in order to achieve the purpose and advantages of using digital technology. These qualities, notably communication and collaboration skills, are seen as crucial in the 21st century [9]. The second stage of validation involved eight experts. In the second stage, two important components were added based on the relevant items and comments from the experts. Their input recognized the need for practicing integrity in the digital world. Many recent studies have focused on the influence of digital technology use on well-being, particularly among adolescents [10]. Hence, components of the framework’s five components are information literacy, computer and technology literacy, digital communication and collaboration skills, digital identity and well-being, and digital ethics. The importance of these primary components in the development of digital skills has been emphasized. Generally, these components incorporate basic awareness (such as managing information, full participation in the digital society, and showing responsibilities and appreciation), use (such as technical knowledge and the utilization of knowledge resources), and the creation of content with an effort towards adaptation to the digital society [11]. Figure 2 illustrates the analyzed components alongside the incorporated components, which have been highlighted by [11].

As shown in Figure 2, anyone who wants to develop their digital skills must first learn how to obtain information and use computers or other digital devices, as shown in the incorporated component of “use”. A healthy society emphasizes not only the application of digital technology and knowledge (which leads to information literacy, computer literacy, and technology literacy), but also the development of new knowledge through communication and collaboration. Additionally, everyone is drawn closer together via engagement and communication, as depicted in the incorporated component of “create”. However, when there are several parties or individuals involved, communication becomes more complex. Thus, for ethical and well-being considerations, the use of knowledge exploration and social interaction in communication should be monitored. Hence, “understand” entails a socialization process, i.e., the individuals’ responsibilities in interaction and communication. A guide on how to use digital tools wisely, such as social presence and emotional presence in social networks, is required for consideration in order for the development of digital skills. Additionally, a proper guide can assist individuals to develop their digital identities [12]. Eventually, the five components of the digital skills framework have been well-elucidated in the three aspects of “use”, “create”, and “understand” with basic awareness, as shown in Figure 2.

5. Conclusions

The aim of this study was to obtain the validity and reliability of the digital skills framework through expert judgements and a pilot test study. Consequently, information literacy, computer and technology literacy, digital communication and collaboration skills, digital identity and well-being, and digital ethics are the five agreed-upon components of the digital skills. Understanding the components of the digital skills is important to all educators before they can design their lessons, learning activities, and assessments during curriculum development. Another critical aspect is that educators must identify their students’ digital literacy or ability in order to create significant and meaningful digital learning experiences in the classroom. The digital skills instrument (DSI) can be used as a tool to gain better insights into digital literacy among students. Finally, this tool can also serve as a digital needs analysis tool to investigate the strengths and weaknesses, and provide continuous quality improvement in curriculum development.