Digitalization of Educational Organizations: Evaluation and Improvement Based on DigCompOrg Model

Abstract

The digitalization of educational organizations is a political and social priority at European level and the model which is the basis for the analysis is DigCompOrg as part of the European Framework of Competences. This article summarizes the results of a longitudinal evaluative research (from 2018 until 2022) around the digitalization process of a compulsory secondary education center. We have applied a mixed method and an evaluative research design based on the use of questionnaires, focus groups and a research diary. This article is focused on data from the last evaluation (2021–2022) where the participants are 26 members of the management team, 46 teachers and 374 students. Our results show that progress has been made in the digitalization process, especially in some areas such as leadership, infrastructure/equipment and pedagogy/support/resources, which have obtained high scores. On the other hand, the data also show other areas where there is more scope for improvement, such as collaboration, digital networks and also innovative assessment practices. This research can be valuable as an example of a good practice around the digitalization of institutions of formal education.

1. Introduction

Digital technologies have transformed the way we live in the world and our learning processes, which has very relevant educational implications. Consequently, schools as complex organizations need to adapt to these new ways of learning and acquiring knowledge. That is why the European Commission, in the last fifteen years, has developed frameworks and tools to support the improvement of digital skills in the education system. At the same time, the Ministries of Education of European countries, and many other countries in the international context, have increasingly supported the teaching of digital competence in formal school education, the need for which has become more evident since the COVID-19 pandemic [1,2]. The European Commission recognizes this and reinforces the movement in this direction through the Digital Education Action Plan 2021–2027 which proposes a vision of a high-quality, inclusive and accessible digital education in Europe that entails improving digital skills and capacities for a digital transformation [3].

One of the fundamental aspects for this digital transformation of education is the improvement of digital competence education that not only includes the competence of the main agents of the educational process (students and teachers), but also guides their actions in the educational center, regarded as an enabling and dynamic agent [4] as well as an organizational learning environment from the perspective of organizational psychology [5,6].

In this way, digital competence is addressed in the analysis of organizations and at the same time it arises as an essential skill for citizens of the 21st century [7] that guarantees the continuity of education in any social situation at a global level [8]. Initially defined as “the confident and critical use of Information Society Technology (IST) for work, leisure and communication” [9] (p. 394/15), since then there have been many definitions and approaches that try to narrow down its meaning [10]. In this research we rely on the concept of digital competence based on the model “DigComp: The Digital Competence Framework for Citizens” [11] reviewed in “DigComp 2.0: The Digital Competence Framework for Citizens” [12]. Five dimensions are distinguished in this competency model: information, communication, content creation, safety and problem solving and can be broadly defined as the safe, critical and creative use of digital technologies to achieve goals related to work, employability, learning, leisure, inclusion and participation in society [12].

Other key contributions to facilitate the digital transformation of education, training and the acquisition of digital skills are the “European Framework for the Digital Competence of Educators” (DigCompEdu) [13] and the “European Framework for Digitally Competent Educational Organisations” (DigCompOrg) [14,15]. The concept of digitally competent educational organization refers to the effective use of digital technology by the educational organization that makes it possible to provide a compelling student experience and ensure a good return on the investment made in digital technology [15]. In this line, different studies point out the convenience of this model to evaluate educational actions within a digital action plan [16,17,18,19]. Based on these frameworks, the European Commission, through the Joint Research Center (JRC), developed in 2018 the online self-reflection tool on digital skills for schools called “SELFIE” (“Self-reflection on Effective Learning by Fostering the use of Innovative Educational technologies”) [20] and in 2021, the “SELFIE for TEACHERS” tool [21] to help teachers reflect on how they are using digital technologies in their professional practice.

As a consequence of the Council Recommendation to adapt the education and training system to the digital age, the different member states of the European Union have begun to define official curricula and review study plans in accordance with these strategic lines. According to a recent report by the European Commission/EACEA/Eurydice [22], more than half of the European countries regard the treatment of digital competence as a cross-cutting theme in primary education, but this percentage decreases in secondary and non-compulsory education. In the Spanish regulatory context, the Organic Law 3/2020, of 29 December, which modifies Organic Law 2/2006, of 3 May, on Education (LOMLOE) [23] describes the obligation to contemplate the digital change that is taking place in society. The preamble of the law specifies that the educational system must cater to a new social reality with a broader and more modern approach to digital competence in accordance with the European recommendations regarding competences for lifelong learning and gender digital divide [24].

Similarly, the different Spanish autonomous communities have begun to establish digital transformation plans at all levels of compulsory education and training, promoting the evaluation of skills based on instruments based on European reference models [25,26,27]. The response of the autonomous community of the Region of Murcia (Spain)—which is the geographical context of our research—is the implementation of the educational program called “Digital Centers” [28] offered for the compulsory primary and secondary education stages and in whose institutional context this research emerges. With a duration of four academic years, this program was born with the aim of advancing the widespread use of digital technologies in the educational centers that take advantage of the program. Among its main purposes are to increase the digital competence of students, promote active and participatory methodologies thanks to the use of technologies or improve teaching digital competence through training. It is relevant to note that “Digital Centers” has been replaced in the 2021–2022 academic year by the “Prodigi-e Plan (2021–2025)” program [29]. This program continues along the path of the previous one, but goes further by proposing an educational digital transformation in educational centers that provide non-university education. This digital transformation is understood as the set of actions that guide the improvement and modernization of the processes, habits and behavior of educational organizations and their staff using digital technologies. In addition, the use of SELFIE is required as a tool that allows establishing the grounds for the elaboration of a digital plan in the center, understood as the instrument that adapts and facilitates the use of digital media in the teaching and learning process for a development integral of the student body [30].

The validity of the SELFIE tool is supported by numerous studies that opt for it in order to assess the digital maturity of the educational institution [31,32,33,34,35,36,37]. From these works, it is concluded that SELFIE represents the practical implementation of DigCompOrg in terms of operationalizing and evaluating the digital capacity of schools, it offers valid information that can be used to prepare a school digital strategy regarding potential areas for improvement and also helps to monitor digital maturity progress over time.

2. Research Problem and Objectives

This research is proposed in the field of scientific studies aimed at educational improvement based on precise data related to awareness of the context. Thus, the research problem is: how to evaluate the digitalization process of an educational organization to promote improvement processes? For this, the DigCompOrg model has been taken as a basis and the following research objectives have been specified:

• Evaluating the digital capacity of the educational institution based on the perception of the main agents involved (school leaders, teachers and students).
• Analyzing the improvement processes through a longitudinal analysis that allows us to observe the evolution of the study over time.

The main final outcome of these research objectives is the design of the Digitalization Plan for the future of our secondary education school trying to reach the top of the model in every indicator and dimension. This research and the consequent improvement plan can serve as models for future research and also for other educational organizations interested in the digitalization process as measured by the indicators of the European model DigCompOrg.

3. Materials and Methods

A non-experimental, change-oriented research and evaluative design was developed. In relation to the temporal dimension, it is a longitudinal research design. And depending on the participants, it is a single case design [38]. The reason for the choice of the case study was convenience, since the educational center in which the research is carried out has developed, during the last four academic years, an educational innovation program to improve the digital capacity of the center called “Digital Centers” [28] whose main purpose is to promote the progressive incorporation of digital resources and media in the teaching and learning process.

A mixed method has been used with quantitative techniques (SELFIE questionnaire) and qualitative techniques (focus groups and researcher’s diary) that have allowed the development of participatory strategies in order to build co-design processes for the proposal of improvement strategies [39]. This type of mixed design fits the definition of concurrent mixed method, based on the use of both quantitative and qualitative data “in order to provide a comprehensive analysis of the research problem” [40] (p. 31), thus integrating the information to improve the interpretation of the results.

For the successive implementations of the improvement processes, the instructional design is based on the ADDIE model [41,42], which allows integrating the basic aspects of the educational innovation program in the traditional sequence that goes from the analysis of the context to the evaluation. The data from the first phases of the research have been published in other previous works [43,44,45]. This article focuses mainly on the third phase of the evaluation, but emphasizing the comparative analysis with data from previous cycles.

3.1. Participants

For the application of the SELFIE questionnaire, all compulsory secondary education students (ESO in Spain), who were the direct recipients of the educational program to improve digital skills, were invited, as well as all teachers and the management team. The total number of participants in the 2021–2022 academic year was 446, with 374 students (ST in the tables), 46 teachers (T) and 26 school leaders (SL) or members of the management team. The actual data of participants in the different cycles of the study since the 2018–2019 academic year are shown in

Table 1. Participants in the academic years 2018–2019, 2020–2021 and 2021–2022.

Education Agent	Invited Sample			Participant Sample
				SELFIE			Focus Group
	2018–19	2020–21	2021–22	2018–19	2020–21	2021–22	2018–19	2020–21	2021–22
School Leaders (SL)	30	27	27	25	25	26	5	4	9
Teachers (T)	75	49	48	61	43	46	7	6	8
Students (ST)	450	542	490	440	393	374	9	8	6
TOTAL	555	618	565	526	461	446	21	18	23

.

For the creation of the focus groups, the entire population that had previously participated in the questionnaire was invited. Based on criteria of representativeness, three discussion groups were formed, consisting of key informants who volunteered. The criterion of representativeness among the students was the selection of students of different educational levels and gender, seeking parity among the members. In relation to the teachers and school leaders, specialists of different ages and work experience were selected, always trying to respect gender balance. The number of participants for each of the focus groups was between 5 to 10 members in order to establish regular and effective communication [46].

Finally, the third instrument was a research diary (RD) which was filled by one member of the management team who is also a member of the research group, so he was involved both in the improvement process and in the research process based on a social approach to educational research.

3.2. Phases

The first phase of the research began during the 2018–2019 academic year, where the SELFIE (school leaders, teachers and students) questionnaire was applied, leading to the detection of the first needs that allowed the design of improvement actions in relation to the use and exploitation of digital technologies that are made in the educational center [43,44]. The next application phase was scheduled for the 2019 and 2020 academic years, but one course had to be postponed due to the lockdown which ensued from the COVID-19 pandemic. During the 2020–2021 academic year, the questionnaire was applied again and the analysis of the results made it possible to design a second digital improvement action plan [45]. These actions were implemented and the results were evaluated in the 2021–2022 academic year, and said results are the ones presented in this article (Figure 1).

3.3. Instruments to Collect Data

For the collection of quantitative data, the SELFIE questionnaire design based on the DigCompOrg model was selected to provide schools with information on the effective use they make of digital technologies [20]. This tool guarantees the anonymous and secure collection of the opinions of students, teachers and school leaders on digital technologies for education through brief statements scored on a Likert scale from 1 (lowest score) to 5 (highest score). The questionnaire is structured around sixty-two items about different aspects grouped into eight dimensions of analysis: (A) leadership; (B) collaboration and networking; (C) infrastructure and equipment; (D) continuing professional development; (E) pedagogy: supports and resources; (F) pedagogy: implementation in the classroom; (G) assessment practices and (H) student digital competence.

For the collection of qualitative information, the discussion group technique was chosen. To do this, an ad hoc semi-structured script was developed, organized around a battery of open-response questions based on the same dimensions of analysis as SELFIE [44]. It was also complemented with a research diary (RD), an instrument in which the main researcher’s observations were collected in an organized and systematic way in his role as moderator of the focus groups and also coordinator of the digitalization process at the educational center. It consisted of a systematic record of information carried out by a single key informant and using the chronology of the annotations as the main criterion.

These three techniques have made it possible to triangulate the information obtained with the aim of giving the study greater validity, in addition to the value of qualitative information itself as a strategy for approaching the real context of educational research [40].

3.4. Procedure

In the beginning of this research, both students and teachers were informed and advised by an expert who guided them throughout the implementation process and during the days on which the information was collected. The questionnaire was filled out electronically at the center (mainly via mobile phone) or at home. The data obtained was anonymous, aggregated and stored on the server of the European Commission. Once the data was analyzed, we worked with the descriptive statistics offered by SELFIE.

For the collection of qualitative data, different discussion groups were formed, made up of key informants who had previously filled out the questionnaire. The talks took place at the center and were conducted by the lead investigator. Once the informed consent of the participants was obtained, the sessions were recorded and the verbal content of the session was transcribed for subsequent analysis. For the analysis of these data, different tables were made in which the verbatim citations were categorized according to the thematic areas and competency descriptors contemplated in SELFIE. This analysis was completed with the observations collected in the research diary during the three phases of the investigation.

4. Results

We have jointly analyzed the quantitative and qualitative results, since, as explained, the qualitative information was used to triangulate and contrast the quantitative data obtained with the SELFIE questionnaires for school leaders, teachers and students. We have presented the results in relation to every dimension of the SELFIE model considering that not all of them are in the students’ questionnaire.

4.1. Leadership

This dimension does not appear in the student questionnaire. Of the total of five indicators of this dimension, in the 2021–2022 academic year, the item with the highest score was “copyright and licensing rules” (A5), with an average value of 4.1, being the faculty who rated it the highest. On the contrary, the descriptor “time to explore digital teaching” (A4) was the one that obtained the lowest mean score (2.4) with similar values to the previous course (

Table 2. Descriptive statistics on Leadership.

A. Leadership	2018–2019		2020–2021		2021–2022
	SL	T	SL	T	SL	T
“Copyright and licensing rules” (A4)	3	3.4	4	4.3	3.9	4.3
“Time to explore digital teaching” (A5)	No data	No data	2.1	2.6	2.2	2.6

Note: Items are measured on a 5-point Likert scale, from 1 = strongly disagree to 5 = strongly agree. There are items that were not included in the 2018–2019 SELFIE questionnaire, so no data is available.

).

These scores are consistent with what was expressed in the discussion groups, since both the management staff and the teaching staff have generally appreciated the existence of a digital strategy at the school. The majority of the participants also stated that the center usually respects copyright and licensing rules when using digital technologies. However, most teachers pointed out the lack of time to explore how to improve their methodology with digital technologies being one of the main drawbacks: “Since three years ago, we have been following a route in our center that allows us to continue with a digital plan” (SL1). “I would like to have more time to educate myself properly and put what I learn into practice in the classroom” (T2).

Most teachers still point at the lack of time as one of the main barriers that limit the use of technologies to explore. This belief could justify the “scarce presence of teachers in autonomous training related to new technologies” that the educational center has scheduled the last four academic years in the afternoon (RD, 13 October 2021).

Therefore, in this area we can observe that there is a coincidence between the quantitative and qualitative data. In addition, the qualitative techniques highlight that, according to the teachers’ vision, the lack of time is one of the main reasons that hinder the use and exploration of digital technologies in the classroom.

4.2. Collaboration and Networking

The students were not asked any questions in relation to this topic. In the 2021–2022 academic year, out of a total of four indicators, the descriptor that obtained the highest average score is “progress review” (B1) with a 3.5, while “synergies for blended learning” (B4) obtained a 3.1, which shows that little progress has been made in the collaboration with other centers or organizations to support the use of digital technologies over time (

Table 3. Descriptive statistics on Collaboration and Networking.

B. Collaboration and Networking	2018–2019		2020–2021		2021–2022
	SL	T	SL	T	SL	T
“Progress review” (B1)	2.8	3	3.1	3.7	3.4	3.7
“Synergies for blended learning” (B4)	2	2.2	2.8	3.1	3	3.3

).

In the discussion groups, most of the participants reported that the progress in teaching and learning with digital technologies made in the educational center has been assessed, although they did not specify how. All the members of the management team recognized that they have hardly collaborated with other organizations to support the use of digital technologies: “Since the pandemic we use more digital platforms and tools to work with students” (T3). “We barely have contact with other ‘digital centers’ to see what they are doing within the program (Digital Centers)” (SL2).

It is remarkable that the descriptor that has obtained the worst rating in this area is “synergies for blended learning” since, according to the management team, during the 2020–2021 academic year the center collaborated with other international educational centers through the eTwinning program that is part of Erasmus +, the European Union program for education, training, youth and sport that promotes the collaboration and twinning of schools from different European countries through the use of technology. A possible explanation would be the “lack of publicity of this type of project among the entire educational community” (RD, 10 March 2022).

The contrast between quantitative and qualitative data in this area shows that the point of view of the management team differs from the point of view of teachers, especially in those items related to collaboration in projects with other schools or organizations to support the use of digital technologies. The discussion groups show that the teachers have a more negative view than the school leaders despite the fact that the teachers score this item more in the questionnaire.

4.3. Infrastructure and Equipment

School leaders, teachers and students participated in this area. In the 2021–2022 academic year, out of a total of sixteen indicators, the descriptor with the highest average score was “bring your own device” (C13) with a 4.1. It was closely followed by the descriptor “digital devices for learning” (C8) with an average of 4 points, a very notable increase compared to the 2018–2019 academic year. The descriptor that again obtained the lowest score was “online libraries/repositories” (C16) with an average of 3.1, with management staff being the group that awarded the fewest points (

Table 4. Descriptive statistics on Infrastructure and Equipment.

C. Infrastructure and Equipment	2018–2019			2020–2021			2021–2022
	SL	T	ST	SL	T	ST	SL	T	ST
“Bring your own device” (C13)	No data	No data	No data	4.2	3.9	3.6	4.3	4.2	3.9
“Digital devices for learning” (C8)	2.8	2.9	3.3	4	4	3.4	4	4.2	3.8
“Online libraries/repositories” (C16)	No data	No data	No data	2.6	3	3.1	2.5	3	3.7

).

In the discussion groups, all the participants agreed that they brought their own devices (mainly mobile phones) to the center, although the teachers thought that the use of the mobile phone was excessive in the students and, frequently, they used it inappropriately. In addition, the management team stated that there were digital devices available for students if they needed them, although most students said they were unaware of this information. On the other hand, most of the participants were not aware that there were online libraries with learning material in the center: “In this center we have many computers to be able to work with the students (…), but the use they make of their mobile phones, or the amount of time they used them should be limited” (T5). “I did not know I could ask for a computer, and take it with me if I need it” (ST2).

This area was one of the most highly valued, obtaining high average scores in all its items. This perception may have been motivated by the investment in electronic devices and infrastructure that has been made in the center mainly from the 2019–2020 academic year to the current one. The educational center has increased the charging points in several classrooms and has invested in the “purchase of 48 laptops and 10 mobile data SIM cards” which can be borrowed, with the aim of reducing the digital divide among students (RD, 1 November 2021).

In this area we can see that there is a coincidence between the quantitative and qualitative data, obtaining similar data. In addition, qualitative techniques have allowed us to highlight a potential source of conflict regarding students’ use of mobile devices in the classroom. Teachers are calling for internal regulation on the use of mobile phones at school.

4.4. Continuing Professional Development

School leaders and teachers participated in this area. In the 2021–2022 academic year, out of a total of three descriptors, the descriptor with the highest average score was “participation in CPD” (D2) with 3.7, with the teaching staff giving it the highest score. It was followed with a slightly lower value by “CPD needs” (D1) and “sharing experiences” (D3) with 3.4 points with similar values to previous courses (

Table 5. Descriptive statistics on continuing professional development.

D. Continuing Professional Development	2018–2019		2020–2021		2021–2022
	SL	T	SL	T	SL	T
“Participation in CPD” (D2)	3.7	3.2	4	3.8	3.6	3.8
“CPD needs” (D1)	3.2	2.9	3.5	3.6	3.2	3.5
“Sharing experiences” (D3)	3.2	2.8	3.4	3.7	3	3.7

).

In the discussion groups, most of the participants agreed that they have had many training opportunities, both internally and externally through the Center for Teachers and Resources (CPR) of the Region of Murcia: “There are many courses to train in new technologies (…) many of which are very interesting, although what we lack is time to do all the ones I would like” (T6).

Teachers and school leaders have been generally satisfied with the training offer related to digital technologies available both through the Center for Teachers and Resources and through the autonomous training offered by the center: “autonomous training seminars in the afternoon” and “short-term training for teachers that are scheduled weekly during recess and are taught by the teachers themselves” (RD 9 December 2021).

Therefore, in this area we can see that there is a coincidence between the quantitative and qualitative data, but qualitative techniques have been able to highlight that teachers often do not find the opportunity to put into practice what they have learned through internal and/or external training.

4.5. Pedagogy: Supports and Resources

School leaders and teachers participated in this area. In the 2021–2022 academic year, out of a total of five descriptors, “online educational resources” (E1) together with “communicating with the school community” (E4) were the descriptors that obtained the highest average score with 4.3 and 4.2, respectively. They were followed by “using virtual learning environments” (E3) and “open educational resources” (E5) with 3.8 points, which indicates a fairly positive assessment of the support and resources available at the center. On the other hand, “creating digital resources” (E2) obtained the lowest average score with 3.6. Similar values to last year were observed, but there was a very notable increase with respect to the 2018–2019 academic year (

Table 6. Descriptive statistics on pedagogy: supports and resources.

E. Pedagogy: Supports and Resources	2018–2019		2020–2021		2021–2022
	SL	T	SL	T	SL	T
“Online educational resources” (E1)	3.8	4.1	4.1	4.6	4	4.5
“Communicating with the school community” (E4)	3.6	4.1	4.5	4.3	3.9	4.4
“Using virtual learning environments” (E3)	3.2	3.2	3.8	3.6	3.7	3.9
“Open educational resources” (E5)	No data	No data	3.8	4	3.6	3.9
“Creating digital resources” (E2)	3.1	3.3	3.6	3.9	3.3	4

).

This appreciation was confirmed in the discussion groups, where most of the participants considered that they used online digital educational resources, as well as virtual learning environments and, to a lesser extent, open educational resources: “In all the thematic units, digital activities are proposed to achieve the objectives (…) we also use Google Classroom to communicate and upload the tasks or to offer other type of information” (T2).

The widespread use of digital platforms by teachers (mainly Google Classroom), as well as the use of online educational resources had increased mainly from the 2019–2020 academic year due to the outbreak of the COVID-19 pandemic. However, “the creation of online repositories with open educational resources or own digital resources is yet to be done”, something which corroborates the negative perception that the educational community has in this regard (RD, 12 June 2022).

Also, in this area we found similar results between the quantitative and qualitative data, this area being better valued globally both in the questionnaire and in the discussion groups.

4.6. Pedagogy: Implementation in the Classroom

School leaders, teachers and students participated in this area. In the last course, out of a total of six descriptors, the highest scored item was “engaging students” (F4) with an average of 3.8, with similar scores among the three groups of participants. In general, teachers are acknowledged to carry out digital learning activities that involve students. On the other end, the descriptor with the lowest average score, with 3.2 points, was “cross-curricular projects” (F6) and it was the members of the management team who once again gave it a lower score (

Table 7. Descriptive statistics on Pedagogy: Implementation in the classroom.

F. Pedagogy: Implementation in the Classroom	2018–2019			2020–2021			2021–2022
	SL	T	ST	SL	T	ST	SL	T	ST
“Engaging students” (F4)	3.2	3.6	3.4	3.9	4	3.4	3.8	3.9	3.6
“Cross-curricular projects” (F6)	23	2.6	3.2	2.9	3.2	3	3	3.2	3.3

).

These results are consistent with the information obtained from the discussion groups, where the majority of the informants positively valued the digital learning activities that they carried out with the students (3.6). Likewise, the majority of the teachers and students were satisfied with the type of activities they proposed for individual and group work. However, they hardly took into consideration technology to work with projects that combine different subjects: “This course in Language we are doing an ‘interactive panel’ where the whole class is uploading our material (…) and that way, we can also see what our classmates are doing” (ST2).

The descriptor “cross-curricular projects” was the one with the worst average score. However, according to the management team, “at least one interdisciplinary project has been carried out during the Cultural Week that is held every year at the center”. The negative assessment can be explained because there is still little use of digital technologies as a means of involving students (RD, 11 May 2021).

The quantitative and qualitative results are coincident. However, in the items referring to cross-curricular projects, the student’s interviews tended to show a more negative perception compared with the scores they gave in the questionnaire.

4.7. Assessment Practices

School leaders, teachers and students participated in this area, although the latter did not participate in all items. In the 2021–2022 academic year, out of a total of ten descriptors, “using data to improve learning” (G9) was the descriptor that obtained the highest average score with 3.6 followed by “assessing skills” (G1) and “digital assessment” (G7) with 3.5 points. These data indicate that assessment practices have been implemented in the center through digital technologies, mainly with respect to the 2018–2019 academic year, although there is still plenty of room for improvement. The descriptor “feedback to other students” (G6) continued to be the one with the lowest rating, with a 2.9, something that indicates that technology is hardly used to make observations about the work of classmates (

Table 8. Descriptive statistics on Assessment Practices.

G. Assessment Practices	2018–2019			2020–2021			2021–2022
	SL	T	ST	SL	T	ST	SL	T	ST
“Using data to improve learning” (G9)	No data	No data	Does not apply	3.2	3.7	Does not apply	3.4	3.8	Does not apply
“Assessing skills” (G1)	2.5	2.9	Does not apply	3.5	3.5	Does not apply	3.3	3.7	Does not apply
“Digital assessment” (G7)	2.8	2.9	Does not apply	3.6	3.5	Does not apply	3.4	3.5	Does not apply
“Feedback to other students” (G6)	23	23	2.6	2.8	2.9	23	2.9	2.9	2.8

).

In the discussion groups, all the participants agreed that teachers usually use technology to assess students’ skills, although the skills that students develop outside the school are valued little or not at all: “I think that the number of us who use technologies to carry out evaluations of our students is increasing (…)” (T1). “Many times, the teacher sends us homework through Classroom, but he does not explain how to do it” (ST4).

This area was one of the lowest rated by the educational community. Despite the fact that the majority of the teaching staff affirmed that they use digital technologies to evaluate the skills of their students, among the institutional documents to which the researcher has had access (Didactic Programs, department minutes, evaluation minutes), they have found little evidence that this practice is fully incorporated into the teaching methodology (RD, 10 June 2021).

In this area, both quantitative and qualitative data coincide: the use of digital technologies to improve student assessment practices is deficient.

4.8. Student Digital Competence

School leaders, teachers and students participated in this area. In the last course, out of a total of thirteen descriptors, the descriptor “learning to communicate” (H8) was the one which obtained the highest score, with an average of 3.9, which indicates that the center provides many communication opportunities using digital technologies. They also obtained a high score for “safe behavior” (H1) and “responsible behavior” (H3) with 3.8 and 3.7 points respectively, with values similar to the previous year. At the other end of the spectrum, we find “solving technical problems” (H13) and “learning coding or programming” (H11) with a 3 and a 2.8 respectively, which shows that little progress has been made regarding these skills compared to the previous course (

Table 9. Descriptive statistics on Student Digital Competence.

H. Student Digital Competence	2018–2019			2020–2021			2021–2022
	SL	T	ST	SL	T	ST	SL	T	ST
“Learning to communicate” (H8)	3.4	3.3	3.2	4.2	4	3.2	4	4.2	3.5
“Safe behavior” (H1)	3	3	3.7	3.8	3.7	3.5	3.8	3.8	3.8
“Responsible behavior” (H3)	3	3.1	3.2	3.6	3.7	3.6	3.7	3.6	3.8
“Solving technical problems” (H13)	No data	No data	No data	3	3.3	2.7	2.8	3.1	3.2
“Learning coding or programming” (H11)	No data	No data	No data	2.7	2.7	2.5	2.6	2.8	2.9

).

In the discussion groups, most of the participants agreed that technologies are used in the center to communicate, as well as learning to act safely and responsibly on the internet, although this is not always achieved. On the other hand, the majority of the students believed that they barely learn to solve the technical problems that arise when using the technologies, nor to program and/or code: “I have become accustomed to using a virtual platform to set up my courses at the beginning of the course (…) I give them the keys and show them (the students) how it works” (T7). “When I have a problem with the computer or with some homework, we go down to the Aula Plumier (computer room) during recess so that a teacher can help us” (ST1).

In this area, “solving technical problems” and “learning coding or programming” were the descriptors that obtained a lower mean score. However, since the 2019–2020 academic year, the management team has set up what has been called “ICT tutorials”. During four recesses a week, students can go if they need to a Plumier classroom (with computers) where an expert teacher helps them solve technical problems that may arise in their daily practice. Therefore, it would be necessary to investigate the reason for the low rating of this descriptor among the students. On the other hand, the management team acknowledged that they still have to design “some educational project for the center in which learning related to computational thinking (coding and programming) is enhanced” (RD, 11 May 2021).

In this area, although the three participating groups give a similar score in the questionnaire, the students’ interviews tend to show a more negative point of view regarding the work that is carried out in the school to improve their own digital competence.

The following table (

Table 10. Average score of the answers of all the participants for each one of the areas of SELFIE.

SELFIE Areas	2018–2019	2020–2021	2021–2022
A. Leadership	3	3.7	3.6
B. Collaboration and Networking	*	3.2	3.3
C. Infrastructure and Equipment	3.2	3.5	3.6
D. Continuing Professional Development	3.1	3.7	3.5
E. Pedagogy: Support and Resources	*	4	3.9
F. Pedagogy: Implementation in the classroom	*	3.3	3.5
G. Assessment Practices	2.6	3.2	3.3
H. Student Digital Competence	3.1	3.4	3.5

Note: The areas “collaboration and networking” are introduced in the 2020–2021 questionnaire. The areas “pedagogy: support and resources” and “pedagogy: support and resources” are introduced in the 2020–2021 questionnaire. In the 2018–20219 questionnaire they are included in the area called “teaching and learning”.

) reflects the average score of all the participants by areas during the three academic years in which the study is applied. We can see that the mean values are similar with one or two tenths of a difference. It is noteworthy that area E, “pedagogy: support and resources” continues to be the most favorably valued, with an average score of 3.9, while areas B, “collaboration and networks” and G, “assessment practices” continue to be the areas that obtain fewer points with respect to the rest of the areas, rising only one tenth on average in the 2021–2022 academic year. However, we want to highlight a decrease in the average score in area D, “continuing professional development” with two tenths less in the 2021–2022 academic year compared to the previous academic year, despite the positive assessment that most of the subjects interviewed had in this area during the last academic year.

If we compare the averages in each of the areas, a notable increase is observed in the score obtained in the last analysis with respect to that obtained in the 2018–2019 questionnaire. However, there are no significant changes between the academic years 2020–2021 and 2021–2022, obtaining the same total average score of 3.5 in both phases and with a maximum difference of one or two tenths between areas during the 2020–2021 and 2021–2022 academic years (Figure 2). Therefore, the data presented indicates that work must continue in these areas to detect possible inconsistencies, as well as the causes that explain a stagnation or setback in the improvement process.

5. Discussion and Conclusions

The recent global public health crisis has accelerated the integration of digital technologies in education and training systems. In this regard, the European Union recognizes the potential of digital technologies for inclusive and high-quality education, and this is reflected in its new Digital Education Action Plan 2021–2027 [3]. But for the development of effective digital skills, the education system needs the support of tools and processes to assess, plan and develop its needs [47]. In this context, the present work reflects the results of the evaluation carried out at a Spanish compulsory secondary school in relation to the development of its digital competence according to the areas contemplated in the European DigCompOrg model. This model helps to evaluate the actions undertaken to improve the digital capacity of an educational institution [19,48], enabling, at the same time, the design of instruments that allow such evaluation. In this sense, the choice of the SELFIE tool has favored the development of a practice of joint reflection in the educational center and informed decision-making about its digital strategy. Different impact studies carried out in Italy [33] and in Spain [34] collected evidence similar to that presented here on the adoption of the SELFIE tool to assess the digital capacity of schools.

When considering the quantitative results of the last analysis of the SELFIE report corresponding to the 2021–2022 academic year in the center under study and the total average score obtained from the opinions of the entire school community (school leaders, teachers and students), the area with the highest score is E, “pedagogy: supports and resources” with 3.9 points, followed by the areas of A, “leadership” and C, “infrastructure and equipment” with an average of 3.6. This indicates that the center has invested a great deal of effort to improve competence areas included in the European model, such as “infrastructure” or “leadership and governance practices”, normally relegated to the background when considering the development of digital competence [49,50]. Therefore, the incorporation of technology in schools requires a redefinition of the organizational culture of schools [51]. In this way, greater planning, management and leadership are observed over time as key dimensions for the achievement of competencies in an organizational system.

Thus, it can be assessed that the use of SELFIE—based on the DigCompOrg model— has been an excellent evaluation instrument, helping to focus on the organization as a whole, rather than on the individual digital competence of educational agents [31,32,33,34,35,36,37]. This approach is in line with the theoretical current of learning organizations that learn [52,53], and provides added value to the institution and educational leadership as key elements of improvement and innovation in schools.

On the opposite side, among the worst considered areas are B, “collaboration and networking” and G, “assessment practices”, with an average of 3.3. The data shows that these areas tend to obtain lower scores than the rest, according to other studies [35]. Although the score for these areas has experienced a small improvement in the 2020–2021 academic year, the results indicate that we must keep on implementing the practices that have an impact on increasing the use of digital technologies in collaboration with other institutions. Also, it is advisable to encourage assessment activities supported by digital technologies to improve the perception of both effectiveness for student learning and adequate performance assessment [54].

It is worth highlighting that if we compare the average scores obtained over time, it is observed that there is a notable improvement in the perception that the educational community has regarding the use it makes of digital technologies in its center; however, very similar results (only two tenths of difference between them) are obtained during the last two phases of the study, which suggests that the digital improvement process could have stalled. Even slightly lower average values are obtained in the 2020–2021 academic year compared to the previous academic year in the areas A, “leadership” (−0.1), D, “continuing professional development” (−0.2) and E, “pedagogy: support and resources” (−0.1).

In this regard, the information obtained through the discussion groups, the researcher’s diary and the observation, shows a growing discontent on the part of the teaching staff with respect to the little time they have available to explore with the new technologies in the classroom despite the fact that social demand is greater. This perception has a negative impact on both the school’s digital strategy and continuous professional development. Likewise, the majority of the students thought that, although they consider working with digital resources motivating [55], they lack content, tasks and various resources. That is why certain actions must be reviewed and reoriented to allow progress in the digitization process that is being carried out in the educational center (

Table 11. 2021–2022 digital improvement goals for each SELFIE area.

Area SELFIE	Descriptor SELFIE	Average	Objective
A. Leadership	A.4. Time to explore digital teaching	2.4	Design a schedule to explore how to improve the teaching with digital technologies.
B. Collaboration and Networking	B.4. Synergies for Blended Learning	3.1	Establish collaborative networks with other schools or organizations to support the use of digital technologies.
C. Infrastructure and Equipment	C.16. Online libraries/repositories	3.1	Create online libraries or repositories with teaching and learning materials.
D. Continuing Professional Development	D.3. Sharing experiences	3.4	Support teachers to share experiences within the school community about teaching with digital technologies.
E. Pedagogy: Supports and Resources	E.5. Open educational resources”	3.8	Promote the use of open educational resources in teachers and students.
F. Pedagogy: Implementation in the classroom	F.6. Cross-curricular projects	3.2	Engage students in using digital technologies for cross-curricular projects.
G. Assessment Practices	G.6. Feedback to other students	2.9	Use digital technologies to enable students to provide feedback on other students’ work.
H. Student Digital Competence	H.11. Learning coding or programming	2.8	Promote the learning of coding or programming in students.

).

Regarding the qualitative results, which show variations in the perception of the school leaders and teachers on the one hand, and the students, on the other, about the different levels of use of digital technology; similar conclusions are found in other studies [33]. Among the main needs that can be highlighted, we find the lack of time that teachers have to try and explore teaching methods using digital methodologies, as pointed out in other works [35,56]. Also, the majority of those surveyed demand greater participation in interdisciplinary projects since, as previous research suggests [57,58], the methodology that contributes the most to the development of digital competence is based on multidisciplinary problem solving and project work. However, in the context of the teaching practice these types of learning activities are still underused [58].

The analysis shows other deficient aspects such as the absence of online libraries or repositories or the design of activities that teach students to program, code or solve technical problems. In this sense, other studies also indicate that students feel more confident about their digital competence in communication and collaboration activities than in areas of digital content creation and problem solving using digital technologies [59,60]. All these aspects are consistent with the scores obtained in the questionnaire and have been included in the improvement proposal (Table 11).

On the other hand, the majority of the teaching staff stated that they are accustomed to using technologies in the evaluation of students’ abilities; however, this statement is inconsistent with the average score given by the students, as well as with the data extracted from the documents to which the principal investigator had access. In this same sense, the students pointed out that their teachers do not acknowledge the digital skills that they develop outside the educational center. Similarly, there is no consensus between teachers and students regarding the most appropriate time of use of the mobile phone, whether a safe and responsible behavior on the internet exists [61,62] or whether an adaptation of the teaching methodology is made to suit the individual needs of the students using digital technologies.

Among the main limitations of the study, we can point out that when presenting a single case study, the results do not allow to establish generalizations. However, this type of study in the educational field encourages practical strategies for the improvement of its reality in the particular context where the research is carried out [63,64]. There may also be limitations in the use of self-reflection data, since the opinions do not necessarily always reflect the reality of the center, while on the other hand, the value of the perception of the main agents involved in the evolution of their institution should be taken into account. In addition, these biases associated with self-perception do not invalidate the usefulness of this type of evaluation because they promote a deliberative process in which the educational community reflects on its knowledge and use of digital technologies and, by doing this, can identify practices that must be improved [58]. Similarly, the results would have been more plausible if statistics went beyond the average scores as only the analysis offered by the SELFIE platform had been used. Therefore, this aspect needs to be tackled in future lines of research.

In conclusion, the results presented in this report follow the path of the current educational policies which put the focus on the establishment of periodic evaluation mechanisms as a fundamental means for a digital transformation [65]. In our case, the research carried out has allowed the educational center to engage in a longitudinal process of reflection and self-diagnosis of its digital capacity as an educational organization, which has served as the basis for the design of a Digitization Plan adapted to its needs. Finally, we can highlight that the number of studies such as this offer an overview of the entire educational system, which should guide educational policymakers in decision making [66] that help develop capabilities of schools within the formal education system in a more efficient way.