Patrimonializarte: A Heritage Education Program Based on New Technologies and Local Heritage

Abstract

This paper presents the design of the Patrimonializarte program, which has been implemented in six classes and three different educational levels in two schools in Galicia (Spain). It involves working with elements of heritage in proximity to the schools’ pupils by employing Information and Communication Technology (ICT) tools. The objectives are (a) to find out if the design is coherent and relevant according to expert judgement and (b) to discover whether the activities related with new technologies are effective according to the evaluation of those involved in the program. This is evaluation research employing mixed methods in which a collective study of cases is carried out. The main results show (1) the evaluation of the design and the tools of the program achieve an optimal degree of agreement according to the panel of experts as far as the variables measured are concerned, and (2) the schoolchildren and teaching staff provide a positive evaluation of the use of ICT tools for working with heritage. E-learning and m-learning make it possible to motivate pupils in the learning and teaching process. Working with ICT tools acquires importance with regard to the possibilities they offer to disseminate the heritage. An integral evaluation of programs is considered relevant in order to understand their multi-dimensionality.

1. Introduction

This paper originates from a prior exploratory study, which identified the needs of infant, primary, and secondary teachers in relation to working with heritage in the classroom [1]. The results obtained reflected the interest of teachers in innovative activities including the use of ICT tools, such as the creation of videos and the gathering of information regarding different heritage elements in order to put it at the disposal of the general public. The teachers identified the need to have at their disposal education programs that help them to work with heritage, as they recognise that their training was deficient in this area.

With the aim of responding to these needs, the heritage education program Patrimonializarte was designed and adapted to the context and the circumstances in which it was to be carried out. The teachers stated that their pupils did not recognise their local heritage and that they demonstrated a lack of motivation in working on it. The program focuses on working with the heritage in the pupils’ local area and employs innovative methodologies, including information and communications technology. The aim is to enable pupils to identify their heritage, to value it, and to contribute towards its dissemination and preservation. In this way, an impact on the community can be achieved, which helps to create individual and shared identities, thereby contributing towards the safeguarding of the local heritage.

1.1. The Evaluation of Heritage Education Programs

Heritage education has acquired increasing importance in the education system in Spain. Two of the institutional points of reference in heritage education in Spain are the Plan Nacional de Educación y Patrimonio (National Plan for Education and Heritage) (PNEyP), focused on research, innovation, and training in heritage, managed by the Instituto del Patrimonio Cultural de España (Cultural Heritage Institute of Spain) and dependent on the Spanish Ministry of Education in collaboration with the regional governments, and the Observatorio de Educación Patrimonial en España (OEPE) (Observatory of Heritage Education in Spain), directed by Olaia Fontal (University of Valladolid). The OEPE manages a database which inventories, analyses, and evaluates heritage education programs [2,3]. Since 2010, it has included more than 1200 references. However, only 1.9% of these programs envisage some kind of evaluation tool [3]. This is a significant limitation or deficiency, given that evaluation should be an essential part of any education program. The OEPE has carried out analyses and categorisations of these heritage education programs [4,5,6,7] and has, recently, created a tool for the evaluation of the quality of heritage education programs; the Q-Edutage scale based on evaluation by standards [8]. However, the evaluation of heritage education continues to be present in a limited number of education programs [9]. Thus, there is a need to broaden research in this field in order to be able to establish guidelines and models that contribute towards an improvement in the quality and effectiveness of these programs.

1.2. An Evaluation of New Technologies in Heritage Education

The acquisition of digital competence is one of the capacities that are established, and commonly accepted, in education around the world. To a greater or lesser degree, most schools include work with ICT in their classrooms [10].

Pupils can go from being mere consumers of information via the Internet to also being producers of content and active subjects in our information society. There are many software programs such as Eduloc, Map My Tracks, and Google Earth, among others, which make it possible to georeference and geolocate contents and points of interest in order to share information with the rest of the digital community. The contents generated can be viewed on mobile devices with free applications. These resources facilitate the introduction of mobile technology into learning environments, which is known as mobile learning [11]. One advantage of this type of software is that students can establish any element as a point of interest. Not only can material or monumental elements be referenced, but other immaterial elements can also be included, which can prove extremely useful in promoting the conservation and dissemination of intangible heritage.

There is already a broad literature on the use of ICT (e-learning and m-learning) in heritage education. The majority is in agreement regarding the great potential of these tools for working with heritage [12,13]. There are also many references to program evaluation in heritage education with a technological basis, both in the field of education and in museums [14,15,16,17]. However, program evaluation in heritage education is a field that has not yet been developed as it should [18]. There is a lack of integral evaluations considering program evaluation from the beginning (analysis of needs, design, initial evaluation), during its implementation (formative evaluation), and at its end (summative evaluation and meta-evaluation). Therefore, the development of evaluation models including these aspects is considered a necessity. However, what type of model would be appropriate for heritage education evaluation programs?

1.3. An Approach to Evaluation Research in Education

There is a certain degree of consensus on the establishment of different stages in education research and, therefore, in program evaluation.

At present, program evaluation has moved away from the phase prior to tests and measurement, to which Lukas and Santiago [19] make reference, and from the stage of tests or effectiveness, which took place at the beginning of the 20th century, which Guba and Lincoln call The First Generation, or the measurement generation [20].

Beginning in 1942, what is considered to be the first systematic and rigorous study of program evaluation was carried out. Ralph Tyler studied 30 schools over a period of eight years (the Eight-Year Study), during which he carried out a program evaluation in order to analyse the effectiveness of the curricula. Tyler proposed a model of educational evaluation that emphasised the achievement of objectives, the organisation of the curriculum based on these objectives and providing value judgements on the information gathered, thereby moving away from measurement [21].

During the 1960s, the learners were the subject of evaluation. This became known as the age of description, what Guba and Lincoln call The Second Generation [20]. At the end of the 1960s, there was a stage known as Accountability when the evaluation was oriented towards the individual. A period of theoretical production began regarding the multiple dimensions of the evaluation process, which would come to be known as evaluation research [22]. Two papers by Cronbach and Scriven changed the direction of this field and focused attention on evaluation oriented towards decision-making [23,24]. In accordance with this, in the 1970s, there was a proliferation of evaluation models. There was a move to what Guba and Lincoln call The Third Generation: Judgment [20], including models such as the decision-oriented model: Context, Input, Process and Product (CIPP) by Stufflebeam et al. [25] and Alkin’s Center for the Study of Evaluation (CSE) [26].

From 1973 onwards, evaluation became consolidated as a specific field of research (the stage of professionalisation). Alternative models began to appear: Scriven’s Goal Free Model [27]; the responsive evaluation of Stake [28]; MacDonald’s democratic evaluation [29]; Parlett and Hamilton’s model of illuminative evaluation [30]; and Eisner’s evaluation as artistic criticism [31]. Journals specialising in evaluation began to be published and specific subjects on evaluation began to be taught in the great universities of the United States. Professional associations were established, such as the Joint Committee on Standards for Educational Evaluation (1975), which publishes the guidelines for the evaluation of programs, projects and education materials [32].

From the 1980s, the participation of stakeholders gained relevance, with its origins in the constructivist paradigm. This stage is what Guba and Lincoln take to be the last generation within the Fourth Generation Evaluation, which would be closer to present-day program evaluation [20].

Pérez Juste states that educational evaluation is the evaluation, based on pre-specified criteria and references, of technically designed and systematically gathered and organised information, regarding the number of relevant factors, which integrate educational processes in order to facilitate the taking of decisions for improvement [33].

1.4. The Present Study

This study is based on the new perspective presented by Escudero regarding evaluation research in the 21st century [34]. Evaluation is situated in a sphere of social change which aims to resolve problems in real contexts. Mixed methods are used, and the agents involved in the evaluation acquire prominence. Specific justification is made use of for the realities of the object being evaluated.

Taking this new perspective of program evaluation as a reference point, our research is based on a real context in which specific needs have been identified and are sought to be resolved. The agents involved gain relevance in the evaluation of the program itself, as their evaluations are gathered on the methodology employed (in the case of the pupils), or of the program itself (in the case of the teachers).

This research deals with a new dimension within heritage education research, which conceives program evaluation as a cyclical process that includes planning, action, and evaluation (

Table 1. Outline of the planning, design, and evaluation of the program.

Needs assessment (Detection, prioritisation and selection of primary needs)	Planning	Contact with schools	Meta-evaluation
		Meeting with participating teachers
		Design of the program based on needs
	Initial evaluation	Coherence and relevance of the program
		Tools
	Implementation (one course September-June)	Infant education
		Primary education
		Secondary education
	Formative evaluation	Assessment of student knowledge
		Evaluation of partial achievement of the program by teachers and families
		Changes, adjustments and improvements to the program
	Summative evaluation	Assessment of student knowledge
		Evaluation of the final results and effects of the program by teachers, families and local authorities

).

It is a new contribution to this field of study based on a review of the model created by Pérez Juste, who proposes integral, integrated and integrating evaluation [33], taking as a starting point an evaluation of needs in order to formulate objectives which contribute towards resolving the shortcomings or needs found. The program is designed in accordance with the objectives and the initial evaluation (of the design and tools) is carried out, which makes it possible to verify the intrinsic quality of the program (content, technical quality) and its suitability to the context. The next step is the implementation of the program, along with the formative evaluation (the analysis of the fulfilment of the planning, the suitability and effectiveness of the activities and the degree of satisfaction of those involved). There then follows the summative evaluation, which makes it possible to discover the degree of achievement obtained according to the learning outcomes and the impact of the program in the family or the surrounding area. Finally, the meta-evaluation of the whole process, that is, the evaluation of the evaluation itself, is carried out. Introduced by Scriven and developed in depth by Stufflebeam, this is considered to be the last step of program evaluation and makes it possible to take decisions in terms of improvement [35,36].

This paper presents the design and implementation procedure of the program and its initial evaluation, both of the design and of the tools and the formative and summative evaluation as far as the methodology used by the pupils is concerned.

This study aims to provide answers to the following questions:

• An evaluation of the design of a heritage education program based on the use of ICT and local heritage. Is the design of the program coherent and relevant?
• The effectiveness of the use of ICT in the development of heritage education programs. What are students’ assessments of the methodology used?

2. Materials and Methods

We present an evaluative model of programs based on the design and implementation of the specifically created program Patrimonializarte. Based on Stake’s premises, a collective study of cases by way of research-action has been designed in which the evaluator constitutes an intrinsic part of the program [37]. The study is supported by mixed methods, which are commonly used in program evaluation. As defended by Cook and Reichardt, the principle of methodological complementarity has been chosen, with the aim of creating a critical interpretative setting and catering for the evaluative complexity of the program (the diversity of objectives, of agents involved, and recipients of the program) [38]. Particular importance is given to the participation of the agents involved in the program in their own evaluation, as defended by authors such as Tejedor and Fetterman, Kaftarian and Wandersman [39,40]. On the basis of these theoretical considerations, the education program and its evaluation have been designed in accordance with Pérez Juste’s model, which defends an evaluation that is integral (providing for conceptual, procedural, and behavioural objectives), integrated (with the educational activity), and integrating (constituting an active medium, which involves teachers, pupils, and the context in which the evaluation is carried out), always with the ultimate aim of improvement [33].

In accordance with this, once the needs of the teachers had been detected and, based on the comments of a discussion group organised with the participating teachers before the implementation, the program was designed in two phases. Phase 1 focused on activities to raise awareness of heritage for its symbolic appropriation. Phase 2 included activities regarding dissemination and enhancement, which allow the pupils to play an active role in the safeguarding of heritage. In order to evaluate the pupils’ learning, a series of different tools were designed, depending on the pupils’ ages, ranging from interviews, graphic projective tests for infant education (IE), open and closed response questionnaires, and reflective texts to the recording of videos in primary (PE) and secondary (SE) education. At the same time, the evaluation of the program was planned based on different moments: The initial, formative and summative evaluation, in addition to the meta-evaluation of the whole process. In order to achieve this, tools for the evaluation of the program and its impact were designed with the aim of collecting data from the teachers, families and local agents. Both parts (the design of the program and the design of the evaluation) were subjected to the evaluation of external experts prior to being put into action in order to ensure the quality of the program. This is presented in detail in the following sections.

2.1. Context and Participants

The design has contemplated the application of the program in the three levels of compulsory education (infant education (IE), primary education (PE), and secondary education (SE)) in two different schools with the aim of identifying dissonances and affinities between different realities (

Table 2. Participants in the Patrimonializarte program.

Agents Directly Involved in the Education Process										External Agents
				Total of pupils per level			Total of teachers per level
	Year and level	Pupils	Teachers	Infants	Primary	Secondary	Infants	Primary	Secondary	Fathers, mothers, legal guardians	22	Local authorities	2
School 1	6th infants	15	1	39	60	12	2	3	7
	5th primary	10	1
	2nd secondary	12	7
School 2	6th infants	24	1								21		2
	2nd primary	25	1
	3rd primary	25	1

). We have used an intentional sample chosen according to criteria of viability, practicality, and profitability of the program. The first school (School 1) is located in a rural area of the interior of the province of A Coruña, which has approximately 3000 inhabitants. The second (School 2) is located in a town in close proximity to the city of A Coruña, with a population of approximately 30,000.

Two groups can be identified among the participants: a) Agents involved in the education process: Pupils, N = 111 (in the first phase) and N = 86 (in the second phase, due to the fact that 25 participants from School 2, corresponding to the 3rd year of primary, were lost) and teachers, N = 12; b) agents outside of the program: Families, N= 43, and local authorities, N = 2, the distribution of which can be observed in Table 2. The teachers taught different subjects (physical education, information technology and communication, technology, Galician language and literature, geography, and history). Participation in the program was encouraged from different knowledge areas in order to obtain an interdisciplinary perspective of heritage education.

2.2. Procedure Prior to Implementation

An initial informative meeting regarding the program was held at the schools with the aim of involving the participating teachers. Once the number of participants had been reached, a discussion group was organised in which the implementation process, with the consensus of the teachers, was structured in the following way:

• Information and planning: The teachers inform the pupils of the planning and methodology which will be carried out throughout the academic year.
• Performance of classroom tasks: The pupils carry out the planned activities in order to achieve the aims proposed in the program.
• Carrying out of joint tasks: Activities in which the three levels converge (transmitting results from one year group to another, school celebrations, exhibitions open to the school and local community).
• Evaluation: Evaluation of the knowledge acquired by the pupils and of the program by both pupils and teachers.
• Knowledge transfer: The final products of the program are presented to both the school and the local community via digital resources.

A second discussion group was held with the teachers and different organisational roles were agreed upon (

Table 3. Organisational chart.

	Roles of the Agents Involved in the Program
Teaching staff	To review the content guide, activities, competences, teaching aims, criteria and standards of evaluation provided by the external evaluator in order to propose changes and improvements.
	To establish the number of work sessions and the annual calendar, to review the tasks and grouping of pupils for the work of the program.
	To monitor the degree to which the objectives of their part of the program are achieved.
	To guide the pupils towards independent learning, motivating them to work autonomously and offering all the information and resources needed during the implementation of the program.
	To evaluate the program according to the criteria and standards set out in the evaluation tools.
Coordinator of the program in the school	To call the participating teaching staff to the preparatory and monitoring meetings of the program and advise of any possible changes in the schedule.
	To collect the evaluation tools from the teachers.
	To resolve any doubts which may arise during the course of the program in collaboration with the external evaluator.
External evaluator	To draw up a guide to the program, with contents, activities, competences, teaching aims, criteria and standards of evaluation for each year group.
	To carry out participant observation during the implementation in order to check the evolution of the program, its problems and possible improvements.
	To provide all necessary help at each stage of the application of the education program and evaluate the entire process.

).

2.3. The Design of the Program

A perspective of integral, integrated, and inclusive evaluation was assumed, based on the new lines of action in education research [33,34]. Education intervention is understood within a process that combines the planning, execution, and evaluation of the program in parallel. Bearing this in mind, the research is structured according to different moments:

• The evaluation of necessities (detection, prioritisation, and selection of primary needs and the design of the program based on these aspects);
• The initial evaluation (of the design and the tools);
• The implementation of the program;
• The formative evaluation (changes, adjustments and improvements to the program);
• The summative evaluation (analysis of the results and effects of the program);
• Meta-evaluation.

In accordance with this outline, the objectives of the program have been established and the design has been adapted to the diagnosis of necessities:

• To design activities relating to heritage elements in proximity to the students in order to stimulate their motivation and identification.
• To include activities that contemplate the use of digital resources in order to encourage the pupils’ motivation.

The program is structured in two phases (

Table 4. The distribution of the program and monitoring of evaluation.

Phase 1			Phase 2
Agents	Objectives	Methodology	Agents	Objectives	Methodology
Students	Know Respect Preserve Understand Value Enjoy	Activities to raise awareness and processes of identification Opening up to the family and education community	Students	Transmit Act Disseminate	Activities for taking action regarding heritage and dissemination General opening up
Teachers			Teachers
Family			Family
Education community			Education community
			Local community
			Heritage managers
Evaluation of needs and initial evaluation		Formative evaluation			Summative evaluation and meta-evaluation

).

Taking local heritage as a starting point, it is related to the territory in which it is located in order to achieve a greater level of awareness and feeling of identification on the part of the pupils [41]. The objective is for them to recognise their everyday ordinary space in order to make it easier for them to understand the heritage in an integrated and contextualised way. Activities employing technology (e-learning and m-learning) are included in order to increase the students’ motivation (

Table 5. Information and Comunication Tecnology (ICT) activities.

	Activities Phase 1		Activities Phase 2
Infant Education (IE)	A1	Classroom work with digitalised old photographs: School, the passing of time	No activities were carried out relating to ICT
Primary Education (PE)	A2	Classroom work with digitalised old photographs: The local area and traditional professions	A6	Field trip to record audio-visual material: forgotten heritage
	A3	Field trip to work with photography: Perspectives and shots, changes and continuities in the local area	A7	Editing of audio-visual materials for dissemination with the CyberLink PowerDirector program
Secondary Education (SE)	A4	Classroom work with digitalised old photographs: Changes and continuities in the local area	A8	Work with the Geoaumentaty program: geolocation and geo-referencing of heritage elements researched by all the groups
	A5	Editing of audio-visual materials with the Ciberlink Power Director program: Changes and continuities in the local area	A9	Work with the BlocksCAD program and Ultimaker Cura for the editing and design of a 3D dolmen

). In the evaluation of their needs, the teachers considered that innovative methodologies were those that best captured the attention of the children. New technologies are useful tools for the study of heritage and form the basis for introducing new methodologies, thus leading to digital literacy [18].

The design of the program incorporates contents, competences, and objectives extracted and adapted from the official curricula of infant, primary, and secondary education in the Autonomous Community of Galicia.

The aim of phase 1 was to enable the pupils to identify with their local area by way of working with digitalised old photographs of their surrounding area (school, trades, places). The SE pupils worked with the CyberLink PowerDirector program (Figure 1) to edit and put together a documentary video on changes and continuities in their local area. The work they produced was openly presented to the whole school and included information via QR codes, which redirect to the school’s webpage.

In phase 2, the objective was for the schoolchildren to recognise the forgotten heritage of their local area in order to be able to perform tasks relating to it. Activities were designed to disseminate this heritage and to attempt to turn it into living heritage of the community. In addition to the audio-visual activities, the students were encouraged to create a final product with the geolocation software Geoaumentaty, so that all of the groups could have a point of convergence in the program. The knowledge acquired was transferred to the local community with the aim of building learning networks and promoting service-learning.

This geolocation augmented reality (AR) tool makes it possible to see the points of interest (POI) created by the pupils in real time. By using the physical coordinates and the compass on the mobile device, they are able to see, in the actual place, virtual information about each site. This tool has a content management system (Figure 2 and Figure 3) in which the pupils can input, edit, and publish multiple contents (videos, images, links to URLs of interest, text descriptors) and create routes with different POI.

The application associated with the program can be used to search for itineraries and to visualise the contents, comment, mark favourite routes, and do the route following the information shown on the map (Figure 4). In the final presentation, open to the whole local community, the pupils showed the instructions to be followed for the download and visualisation of the information so that those attending could access it.

In addition to working with geolocation and augmented reality applications, the pupils also designed a dolmen in 3D (Figure 5), which they used to carry out research and which is georeferenced in Geoaumentaty. With their model and a 3D printer, after studying the scale plans of the dolmen, they generated a design via the BlocksCAD program and prepared the piece to be printed with the Ultimaker Cura software. 3D printing, a tool included in ICT-supported learning, facilitates the development of creative and motivating digital skills [42].

2.4. Tools

In order to evaluate the knowledge acquired by the students and the program itself, the following tools were designed with the aim of gathering the perceptions and evaluations of the students regarding heritage before and after the implementation of the program; their awareness of the passing of time; their affective and emotional capacities and their opinions of heritage; the significant learning they have achieved; and their own evaluation of the program.

For phase 1 (the initial evaluation and the formative evaluation), two semi-structured group interviews were carried out in infant education (pretest-posttest). For the pupils in primary and secondary education, two multiple choice questionnaires were designed (pretest-posttest). For the analysis, the system of categorisation summarised in

Table 6. System of categorisation of Phase 1.

Tool 1-Pretest	Tool 2-Posttest
C1. Perceptions regarding heritage	C1. Perceptions regarding heritage
C2. The passing of time	C2. The passing of time
C3. Evaluation of the heritage	C3. Evaluation of the heritage
	C4. Significant learning
	C5. Evaluation of the methodology

was used. Although the categories of analysis are the same for the different year groups, their variables and items are adapted to the educational level of each group and they are formulated with different questions.

In Phase 2 (summative evaluation), two evaluation tools were designed for infant education. The first was a graphic projective test to identify the children’s perception regarding the preservation of the heritage in their immediate surroundings. The tool was applied after carrying out activities focusing on knowledge and the symbolic appropriation of a dolmen located in the proximity of their school, and its preservation and dissemination. The second tool was a semi-structured group interview composed of 10 items.

Table 7. System of categorisation for Phase 2.

Tool 1	Tool 2 (10 Items)
C1. Signage	C1.	Knowledge acquired regarding the heritage
C2. Preservation	C2.	Awareness of preservation
C3. Human action	C3.	Affective and emotional capacities
C4. Additional information	C4.	Evaluation of the methodology

shows the categories of analysis.

For primary education, a data collection tool (written text) was designed for the later analysis of content regarding the children’s perception regarding heritage. The second evaluation tool was a questionnaire made up of 16 items with open-ended questions.

For evaluation in secondary education, a data collection tool was designed, which consisted of a recording, made by the students, of a video for the later analysis of the content. The second instrument designed was a questionnaire with 18 items with open-ended questions. For the analysis, the categories summarised in Table 6 for Tool 2 were followed.

In order to gather information regarding the teachers’ evaluation of the knowledge acquired by the pupils and their opinions of the program, two tools were designed. For the formative evaluation, there was an online questionnaire made up of 20 open questions. The main objective was to carry out a monitoring evaluation and to proceed, if necessary, to make improvements or changes. In the summative evaluation, an online questionnaire made up of 21 open-ended questions was used. The aim was to carry out a final process evaluation of the program according to the categories shown in

Table 8. System of categorisation for the formative and summative evaluation by the teachers.

Phase 1	Phase 2
C1. Partial achievements of the program	C1. Efficiency. Final achievements
C2. Relevance of the development of the contents	C2. Effectiveness. Achievements of the process
C3. Usefulness of the activities	C3. Efficiency. Achievements in the improvement of the teaching
C4. Suitability of the resources	C4. Impact: overall results
C5. Suitability of the methodology	C5. Degree of satisfaction
C6. Degree of satisfaction

.

Although the tools were the same for the two schools in terms of their construction, they differed in certain contents, due to the fact that they were adapted to the different contexts. In addition to the aforementioned tools, the technique of participant observation was employed throughout the whole process [37]. The tools for collecting information in the observation were the field notes and the audio recordings.

3. Results

3.1. Analysis

The data extracted from the evaluation of the design and the tools created by the panel of external experts and the initial, formative, and summative evaluations by the pupils and the teachers’ formative and summative evaluations were analysed. In order to analyse the agreement between the judges, the Bangdiwala’s weighted agreement coefficients (B^W_N) [43] were calculated. In order to analyse the agreement between the pupils and teachers, attention was given to the categories related with the methodology of the program in which the agents involved evaluated the activities related with the use of ICT. For the analysis of the data, the R programs 3.6.3 and SPSS (analyses of frequencies and descriptive statistics of quantitative data) and the Atlas.ti program (content analysis of the qualitative data) were used.

3.2. Evaluation of the Design

After designing the guides to the program, a third focus group was organised with the participating teaching staff for the initial evaluation and, then, the design of the program [39]. Corrections were made and the guidelines to be followed during the implementation were established. Having concluded the redesign, the program was sent for evaluation by the external expert judges.

The documental composition (the structure of the program as a document from a technical point of view) and its internal and external coherence (the relationship between the elements of the program and their links with external elements) were analysed via expert judgement. For this, the items were submitted for review to six judges, who evaluated on a scale of 1 to 4 in terms of coherence and documental composition, applying the indexes of agreement of Bangdiwala’s weighted agreement. The six external judges were selected based on criteria of affinity with the topic matter of the program. Three of them are doctors and lecturers in different Spanish universities whose main line of research is the field of heritage education. The other three are teachers of the three levels of education included in the program (IE, PE, SE). They were selected intentionally as they are the ones who are in the classroom every day with their pupils and are best able to evaluate the methodology employed within the real context.

The index of Bangdiwala’s agreement B^W_N makes it possible to graphically represent the degree of agreement and provides a measurement of its strength [43]. In this representation (Figure 6), the black squares show the agreement observed, whereas the grey squares represent partial agreement. The white area in each rectangle is the graphic representation of the disagreement.

The Bangdiwala’s weighted agreement coefficients obtained were: B^W_N = 0.917 (coherence of the design) and B^W_N = 0.883 (set of documents). In order to interpret the agreement coefficients, Muñoz and Bangdiwala propose the following criteria: Values of between 0.000 and 0.200 indicate a poor level of agreement; between 0.201 and 0.400, a weak agreement; between 0.401 and 0.600, moderate; between 0.601 and 0.800, good; and higher than 0.801, an excellent level of agreement [44]. In accordance with these criteria, the level of inter-judge agreement as far as the design of the Patrimonializarte program is concerned, can be considered excellent in terms of its coherence and documental composition.

In Phase 1, the initial set of items was submitted to review by six judges, who evaluated each one on a scale of 1 to 4, in terms of relevance (43 items) and clarity (68 items). In order to analyse the level of inter-judge agreement, the Bangdiwala’s weighted agreement coefficients (B^W_N) were calculated. As can be observed in Figure 7, the Bangdiwala’s weighted agreement coefficients obtained were: B^W_N = 0.973 (Relevance Phase 1) and B^W_N = 0.914 (Clarity Phase 1).

In Phase 2, the set of items was submitted to review by 4 judges, who again evaluated each one on a 4-point scale, in terms of their relevance (105 items) and clarity (105 items). The Bangdiwala’s weighted agreement coefficients obtained were: B^W_N = 0.94 (Relevance Phase 2) and B^W_N = 0.846 (Clarity Phase 2), which can be interpreted as excellent in accordance with the criteria proposed by Muñoz and Bangdiwala [44].

3.3. Evaluation of Digital Resources by the Students

First of all, the results of the formative evaluation are shown (Phase 1, activities A1-A5, see Table 5) by the IE, PE, and SE pupils.

In IE, within category 4 “Evaluation of the methodology”, all of the pupils evaluated activity A1 (working with digital photographs) in a positive way. The schoolchildren recognised their usefulness, although, due to their age, it proved difficult for them to provide judgements of evaluation regarding the activities.

In PE, we analysed category 4 “Significant learning”. With regard to “discovery via photography”, 46.6% of the pupils recognised that, thanks to the work carried out with digital photographs, they have identified changes and continuities that have taken place in the school/trades/local area, through activities A2 and A3. In SE, 63.6% mentioned the same for activities A4 and A5. 51.7% of PE and 27.3% of SE stated that they recognise that the photographs enabled them to learn about the past. Further, 1.7% of PE and 9.1% of SE did not answer this question.

In category 5 “Evaluation of the methodology”, in relation to the “evaluation of the work with photographs”, in PE (A2-A3), 58.6% of the pupils gave a positive evaluation of the field trips for the photographic work, due to the fact that it enabled them to evaluate the changes that had taken place in their area. Furthermore, 29.3% stated that, in addition to being fun, they learned new things and 12.1% said that it was fun, and they did not have class. There were no negative evaluations. As far as the “evaluation of the creation of videos” is concerned, in SE (A5), 83.3% of the pupils considered that creating audio-visual documents was relevant with the justification that it is a good way of showing the work to the rest of the community. Furthermore, 16.7% stated that carrying out this work gave them satisfaction. There were no negative evaluations.

With regard to the results of the summative evaluation (Phase 2, activities A6-A9) in PE, for category C4, regarding “the evaluation of the activity of audio-visual creation”, 85.7% of the pupils gave a positive evaluation of the creation of the documentary video. They made reference to the entertainment factor, as they found it enjoyable to record videos, but also to the usefulness of this method in terms of dissemination, providing answers such as “so that people can know everything (about the dolmen)”; “because then it can be seen that it [the dolmen] is not looked after properly”; and “because in this way we can make more people contribute towards the maintenance of the dolmen”.

In SE, category C4 included items that aimed to reveal the importance given by the pupils to the dissemination of the heritage with which they had worked. The pupils unanimously agreed on the importance of disseminating the knowledge they had acquired among the community as a whole. These items are related with that which evaluates the geolocation activity carried out with the Geoaumentaty program (A8) as it is the means which was used to disseminate information regarding the heritage elements studied. In category C5, the pupils gave an unanimously favourable opinion of the usefulness of the georeferencing activity. Among the reasons given were: “because then you can know exactly where each place is”; because it allows people to know about our area”; and “because in this way we can tell people everything about heritage”. As far as the evaluation of the use of ICT tools for the dissemination of their work is concerned, 90.9% stated that these were the ideal means for disseminating information. Also, 9.1% gave no opinion in this regard.

3.4. Evaluation of Digital Resources by Teaching Staff

With regard to the formative evaluation, in category C3, the usefulness of the activities was analysed. All of the teachers considered these activities to be suitable in terms of the degree of motivation of the pupils. They were of the opinion that other types of more traditional or less dynamic activities would not attract the attention of the pupils in the same way.

With regard to the results of the summative evaluation, in category C1, the teaching staff evaluated the effectiveness of the program in terms of the acquisition of competences. Ten of the 12 teachers who participated awarded the program a high score in terms of effectiveness, stating that it had contributed towards improving the acquisition of key competences such as digital competence. For category C4, referring to the impact of the program, 9 of the 12 positively evaluated the repercussion of the program in the press, exhibitions, and the digital media, making mention of its benefits in the dissemination of the heritage in question. The remaining three teachers did not provide an answer to this item.

4. Discussion and Conclusions

The main objectives of this study were, first of all, to evaluate the coherence and documental composition of the design of the Patrimonializarte program and the relevance and clarity of the tools used based on the judgement of experts. Secondly, it aimed to evaluate the effectiveness and usefulness of the digital resources used in the program based on the judgement of the agents involved (pupils and teaching staff).

4.1. Program Design

Following the analysis of the results of the evaluation of the design of the program and the calculation of the Bangdiwala’s weighted agreement coefficients [43], an optimal degree of agreement was shown in terms of its coherence and documentary composition. In the variable “documental composition”, a weighted agreement coefficient B^W_N = 0.883 was obtained, slightly less than that of the variable “coherence of the design”. Although this result is excellent, according to the criteria of Muñoz and Bangdiwala [44], its slight decrease is due to the fact that the judges considered that the chronological planning should have been more specific. However, the nature of the fieldwork itself made it necessary to carry out a more general planning given that it should be adapted to the temporal realities of the schools [34]. Following the same criteria, the Bangdiwala’s weighted agreement coefficients demonstrate excellence in the relevance and clarity of the tools employed in the program.

A program that is solid and structured in terms of its documental composition, its organisation, and the monitoring of its evaluation has been designed. Thus, the tools designed for the evaluation also had an extremely high degree of agreement. For this reason, they can be considered to be relevant and coherent in relation to the program. However, the diversity of the tools employed constitutes both a strength and a weakness at the same time. As a collateral effect, the systematisation and later analysis of the information proves to be complex. On the other hand, its strength lies in the fact that this variety of tools makes it possible to measure the results of the application of the program in the different groups in a more individualised manner.

4.2. Digital Resources

The results extracted from the evaluations of the digital resources in the program by the pupils and teaching staff indicate that there are no significant differences between the two agents, given that both consider that a methodology supported by ICT tools for working with heritage is positive.

One strength in common lies in the variable “dissemination”, as the results from Phase 2 (categories C4 and C5) show that the pupils lend importance to the activities carried out largely thanks to the fact that they enable them to disseminate their work. This may be due to the fact that, via working with ICT tools, the pupils themselves become the creators of the contents and the protagonists of their own learning, thereby motivating them to carry out the activities [45]. In this regard, the teaching staff remarked that a positive point of the use of ICT tools was the motivation that this methodology offers to the pupils and that, via their use, the pupils acquired competences, among them digital competence [11].

As far as category C4 is concerned, relating to the significant learning acquired by PE pupils, there is a certain degree of incongruence in the results. When the schoolchildren were asked what discoveries they had made through their work with digital photographs, in PE, 46.6% stated that they had identified certain changes and continuities which had taken place in their local area. In SE, this figure was 63.6%. The majority of PE pupils stated that working with digital photographs had enabled them to learn about the past, in addition to being able to identify changes and continuities. However, only 27.3% in SE mentioned that this activity had helped them to learn about the past.

The SE pupils stated that they had noticed changes and continuities thanks to the activities, but they claimed to have obtained a lesser degree of knowledge about the past. This may be due to the typology and methodology used in the different activities. In PE, a field trip was organised, in which the pupils (A3) could verify in situ the changes and continuities that had taken place in their local area via the taking of photographs from the same perspective as that of the old photographs. In line with this, the PE pupils positively evaluated the field trip for the work with digital resources in category C5.

This activity may be more effective than the activity carried out in SE (A5), in which an audio-visual document was created counterposing old photographs and the songs of a local author, which relate what life there was like in the past. Perhaps this activity requires the pupils to think with a higher level of abstraction in order to visualise changes and continuities. Thus, they did not obtain so much knowledge of the past as their counterparts in PE. However, 83.3% of the pupils considered that the making of audio-visual documents was relevant, with the justification, again, of the importance of dissemination.

4.3. Main Conclusions

Many studies on the use of new technologies in heritage education and on the evaluation of heritage education programs have already been carried out. However, there is a shortcoming in this area that must be remedied; an integrated, integral, and integrating evaluation of the programs [33]. For this reason, a design has been specifically created, which has been evaluated and implemented in order to then be able to evaluate its results in an active manner.

The study has achieved its proposed objectives. The reliability of the program and of its evaluation tools has been measured and proven and the evaluations made by the pupils and teachers regarding the use of new technologies in heritage education have been analysed.

The results show that employing elements of heritage that lie in the learners’ immediate proximity generates a high level of interest. E-learning and m-learning are motivating methodologies that allow pupils to acquire significant knowledge by learning to do things. The pupils give importance to the use of ICT tools due to their great potential in the dissemination of heritage.

The teachers believe that their students feel more involved in a methodology in which they have a leading role and generate contents themselves, seeing their work reflected in a webpage or a mobile application. One limitation detected is that the design of activity A5 for SE did not fully achieve the proposed objectives. The pupils have a better understanding of the passing of time and acquire more knowledge about the past by seeing the changes which have taken place in situ, rather than doing work in the classroom, as has been proven in PE.

This study is a contribution to program evaluation research in heritage education, given that it deals with evaluation as a cycle that encompasses the time before, during, and after its implementation. The usefulness of ICT tools is corroborated in the study of heritage.

We believe that future lines of research should approach evaluation at all moments of the program, not only at its end, in order to respond to the demands of the agents involved, to make it possible to carry out improvements or adjustments to the program and to emit value judgements in order to improve heritage education. In this way, contributions can be made towards improvements in the field of heritage education.