Cómo Entrenar tu Dragón: A European Credit Transfer System Module to Develop Critical Artificial Intelligence Literacy in a PGCERT Programme for New Higher Education Lecturers ^†

Abstract

This paper summarizes the findings and main conclusions from the first presentation of the module “CETD23: Cómo entrenar a tu dragón: la inteligencia artificial generativa como herramienta para mejorar el aprendizaje en entornos online e híbridos”. This is an optional module accredited through the ECTS (European Credit Transfer System) and delivered as part of the “Plan de Formación de Docencia y Personal Investigador 2021–2025” of the Universidad de Las Palmas de Gran Canaria (ULPGC). The Plan de Formación is a development programme offered by Spanish universities to new and existing teaching staff, aimed at improving the quality of their teaching practises in line with Aneca’s Docencia regulations (like the PGCERT and PGCAPT programmes in the UK). The aim of module CETD23 is to explore the use of Generative AI (GenAI) to enhance learning and teaching and to build the AI literacy of ULPGC’s teaching staff. The module received high student satisfaction, with an average score of 4.84 on the Likert Scale, and achieved a 100% completion rate for the final summative project. The final conclusions highlight the need for universities to establish reglamentos (policies and guidance) on how to use GenAI to enhance learning and assessment, as well as to involve students as equal partners in the design and assessment of methods that use AI.

1. Introduction

The popularization of large language models (LLMs) and Generative AI (GenAI) software based on those models (such as ChatGPT, Gemini, or Claude) by the end of 2022 forced higher education (HE) institutions around the world to engage in a debate regarding how to use GenAI to enhance learning and teaching in a university context, as well as how to set up a basis for the ethical use of GenAI conversational agents in assessment. By the summer of 2023, prestigious associations of universities in the English-speaking world, such as the Russell Group in the UK, had already launched a statement with five principles for the use of AI in universities. This statement was taken as a starting point by many universities in the UK to publish their own institutional policies and recommendations on how to use GenAI to enhance learning and teaching, as well as to design more authentic assessments around AI.

In Spain, the debate regarding the use of Generative AI (GenAI) in HE has been present in the HE sector since 2021. However, it was in 2023 when CRUE Universidades Españolas (the biggest association in the HE sector in Spain, comprising 77 leading universities) established the need to equip teaching staff with the capabilities, skills, and knowledge required for the ethical use of AI in teaching and assessment in HE. This implied that new and existing docentes universitarios (teaching staff) should receive appropriate and comprehensive training on GenAI and how to develop their own AI literacy through the relevant Planes de Formación (Development Plans).

2. An Alternative Vision for the Introduction of Generative AI in HE

It shall be highlighted that, when the first presentation of the module was written (November 2023), the use of Generative AI (GenAI) to enhance learning and teaching was still a novelty, and most universities worldwide were writing their own institutional policies for the use of GenAI. Universities were also planning how to train lecturers and students on the ethical use of AI to support learning and teaching. In this context, Bearman and Ajjawi [1] established that when it comes to AI in HE, both lecturers and students are learners, and any training course aimed at teaching staff should also reflect that.

It is also significant that the academic literature on GenAI often identifies Gen Z users as “students” while Gen X and Gen Y users are identified as “teachers” [2]. However, Generation Z users are also starting their academic careers as new teaching staff and researchers, so it is very likely that the accreditation programmes for new teaching staff will be undertaken more and more widely by Gen Z users. Any module or course covering AI in those accreditation programmes should consider this demographic group and whether they have a different approach to or understanding of AI compared to other generations.

The module leader endorsed a vision of higher education (HE) aimed at fostering good and compassion within society, drawing on the principles articulated by Czerniewicz and Cronin [3]. The objective was to develop a module that critically examined the concept of ‘AI literacy’ and the application of large language models (LLMs) through a lens of compassion within HE. This approach aligns with the mission statement and core values of the ULPGC.

3. Development of Module CETD23

3.1. Context

The module “CETD23 Cómo entrenar a tu dragón: la inteligencia artificial generativa como herramienta para mejorar el aprendizaje en entornos online e híbridos” (How to train your dragon: Generative AI to improve learning in online and hybrid environments) is taught as part of the development plan Plan de Formación de Docentes y Personal Investigador (PFDI) 2021–2025 of the Universidad de Las Palmas de Gran Canaria (ULPGC). This plan is aimed at both teaching staff and research staff, in order to support new lecturers in their accreditation programmes and experienced lecturers in the application process for academic promotions.

The ULPGC is a public university which was founded in 1979. It is, at present, a medium-sized university, with 1640 staff members, 19,876 students, six campuses located on three different islands, and 43 international projects in the tertiary sector.

The module counts as 1 credit in the European Credit Transfer and Accumulation System (ECTS), which is equivalent, roughly, to 50 h of study in the UK CATS credit system.

3.2. Rationale for Module CETD23

Module CETD23 was conceived in accordance with the values of the ULPGC, which supports a vision of HE that promotes social justice, solidarity among nations, and the defence of freedom, equality, and transparency, among other values.

The module’s purpose is threefold:

• Exploring the use of Generative AI (GenAI) to enhance learning and teaching.
• Building up AI literacy for university teaching staff.
• Designing new assessment methods that promote the ethical use of GenAI among students.

Building up the AI literacy of teaching staff implies first clarifying the concept of “AI literacy” and making this consistent with a vision of HE founded on compassion and justice. AI literacy can be defined as a critical awareness of the potentialities, the limitations, and the technological, social, and ethical challenges that the use of Generative AI models brings to teaching, assessing, and researching in HE. This is to form highly aware graduates who put the values of compassion, care, transparency, and justice at the heart of any professional activity that they perform.

This conception of AI literacy was complemented by a literature review of the concept itself in order to identify the skills and capabilities encompassed under this umbrella term:

• Knowledge, understanding, and use of AI multimodal agents [4].
• AI ethics and human-centred considerations, such as fairness, accountability, safety, etc. [5].
• Higher-order thinking skills [4].
• Digital literacy skills, such as data literacy, information literacy, etc. [6].

By the time the literature review took place, it was not clear whether a new AI pedagogy to foster AI literacy was needed, or if it was just a case of adapting current digital pedagogies and teaching methods. Bearman and Ajjawi’s article [1] “Learning to work with the black box: Pedagogy for a world with artificial intelligence” (2023) was a source of inspiration for writing the module. The authors state that HE institutions should adapt their pedagogical strategies to help students navigate and critically engage with AI technologies in real-world contexts. They suggest that AI should be understood within its social and technological context, emphasizing the importance of teaching students how to work with AI’s inherent ambiguities and uncertainties.

In addition to these sources, module CETD23 was deeply influenced by the series of online events “PGCert and Educational Developers”, organized by Virna Rossi (Ravensbourne University London). The online events are aimed at teaching staff and academic developers working in the delivery of PGCert/PGCAP programmes in the UK, and are dedicated to different pedagogical practises related to those programmes. In May 2023, a specific event was dedicated to the introduction of AI in PGCert programmes. The module leader of CETD23 attended that event, linking the content on AI and assessment and the risks and limitations of GenAI described in the module with the main conclusions reached by attendees of the event, and showcased different approaches to helping new teachers to familiarize themselves with AI literacy and the skills and knowledge the term comprises: by embedding those skills in existing modules, or by designing a specific module to cover GenAI.

3.3. Learning Design and Delivery Mode

The first presentation of module CETD23 was launched in January 2023, with an initial cohort of 20 students, primarily belonging to Generations X and Y (75%) and Z (only five participants).

The module was delivered following a blended learning approach that mixed taught sessions at the university’s campus of Tafira with online sessions delivered through MS Teams. Blended learning was chosen to give participants more flexibility to complete the module, since the university has a double-insularity identity, with teaching staff located across different islands. For the first cohort, one of the participants was in another country due to work commitments.

The synchronous sessions, both on campus and online, were designed to support the online component of the module developed in Moodle (the university’s VLE platform). Participants had the choice to complete the module as a self-study online course through the VLE, or to follow a cohort-based approach by joining the synchronous sessions. These sessions were designed to build up the concepts introduced in the online module in Moodle, as well as to foster a learning community of practice, keeping participants motivated to complete the module.

The online module comprised 10 self-contained units addressing the different skills and knowledge included under the umbrella term “AI literacy” that are listed in the previous sections.

The 10 self-contained units comprising the module included a welcome/induction section and a section covering the final project assignment. The topics addressed in the units included the following:

• A basic understanding of how AI and machine learning work, including ethical considerations regarding how those models are trained and the environmental impacts of large computational centres.
• Familiarization with the most common GenAI conversational multimodal agents (Copilot, Claude, Gemini, ChatGPT, etc.), building digital skills to obtain the most efficient results; prompting techniques such as chain-of-thought and ethical jailbreaking, and discussing how to formulate complex problems for AI chatbots, were also covered as part of this familiarization.
• Understanding GenAI’s risks and limitations in HE contexts, like hallucinations, bias, lack of transparency, and inequalities in access.
• Acquiring a sound knowledge of ethical and regulatory frameworks for the use of AI in society, so that teaching staff can foster critical conversations on ethical uses of AI with their own students. In the European Education Area (EEA), the regulatory framework is shaped by the EU AI Act and the provisions that guarantee existing legislation of data protection and copyright within the EU area.
• A critical examination of the concepts of assessment, academic integrity, intellectual ownership, and copyright to support GenAI in enhancing assessment. Module participants can explore the concepts of authentic assessment, project-based learning, or assessment as/for learning as alternative approaches that support the use of GenAI to build both the AI and assessment literacy of undergraduate and postgraduate students, instead of using AI chatbots to just write down the assignment.
• A review of the recommendations and guidance from international and national bodies such as Jisc, UNESCO, QAA, etc.
• Hands-on learning activities in which module participants can try GenAI as students, and from which they can derive inspiration to design learning activities for their own students:
  • Customizing GenAI chatbots as digital assistants and tutors.
  • Assessing the terms of use and privacy policies of the most popular LLM providers (OpenAI, Anthropic, Google) in light of the provisions included in the EU AI Act and EU GDPR.
  • Course debates that highlight the ethical and social dilemmas of LLMs: for example, the impact of AI on labour rights and civil liberties, the use of copyrighted data to train AI multimedia chatbots, or how the providers behind LLMs process and store confidential data passed on by users in responses to questions.
  • Designing assignments based on real-life projects that students would undertake in their professional areas.
  • Using GenAI chatbots to customize rubrics and individual feedback.

4. Assessment

The module concluded with a final summative assessment in which participants had to design and develop a learning activity or assessment method that fosters an ethical and innovative use of Generative AI to support learning. Participants could then apply this to their own courses and trial it with their own students.

Participants were asked to submit a 2500-word assignment providing a description of the learning activity or assessment method they had designed for their own courses. They were told that this assignment should include the following information:

• A detailed description of the learning activity or assessment method to be implemented with students.
• The academic module, programme of study, and demographic profile of the users targeted by the learning activity: undergraduate and/or postgraduate students, part-time or full-time students, whether there are neurodivergent students or students with special needs, et cetera.
• The Generative AI multimodal agent(s) supporting the learning activity/assessment method and the rationale for the chosen GenAI software.
• A reflective account of how the proposed learning activity and GenAI software choice relate to the module’s learning outcomes, considering potential risks and challenges for participants’ students and weighing these against the benefits of using the GenAI tool.

Participants could derive inspiration for designing their own activity from the learning activities they had experienced as students throughout the course, such as the following:

• Customizing a GenAI chatbot to act as a digital assistant and tutor.
• Analyzing the safety and privacy risks of GenAI chatbots: evaluating the terms of use and privacy policies of LLM providers.
• Activities that reflect on ethical, economic, and social dilemmas surrounding LLM models: environmental impact, employment legislation, copyright, ownership, intellectual property, etc.
• Designing a more authentic assessment: project-based learning, assessment as learning.
• Using GenAI to support feedback: the design of rubrics and personalisation of student feedback.
• Writing customized guides and recommendations on how to use Generative AI to support learning for their own students.

5. Student Feedback for the Module

The information provided in this section has ethical approval from the Director of the Vicerrectorado de Innovación Educativa de la ULPGC, in accordance with institutional policy.

16 participants out of 20 completed the final survey, giving the module a total score of 4.813 out of 5 on a Likert Scale. All participants who submitted the survey agreed that the module content met their expectations (4.87 out of 5), with 81% of respondents awarding the maximum score. The course completion rate was 100%.

The ice-breaking activities included in the introductory unit of the module showed that the first cohort of students were “early adopters of new technologies,” with 100% of the participants reporting that they were accustomed to using approved learning technologies in their classes. The fact that participants were confident with technology explained their positive attitudes towards the use of GenAI in HE, recognizing the potential benefits for enhancing student engagement and personalized learning experiences.

However, there was also a level of scepticism regarding the reliability of LLMs as a source of information for very specific academic subjects. Technical glitches such as hallucinations and inconsistencies were also reported by participants while working with different LLMs to create multimedia content for their own modules. In this respect, several participants reported that, with very specific academic sources, it took them more time to ensure that the results created by the GenAI model did not contain any inconsistencies or fabrications than it would have taken to create the learning materials themselves. This view was supported by the rest of the class.

Debates were a popular learning activity within the module, focusing on key aspects such as ethical frameworks and copyright issues. At a time when there was no specific institutional policy on the ethics of AI and the EU AI Act had not yet been approved, participants had the opportunity to explore the final drafts of the Act and discuss ethical codes, such as the Russell Group’s “Principles on the use of AI in education” [7], discussing their pros and cons.

Participants particularly valued the module’s international approach, including the diversity of sources in the reading lists and guest speakers from the UK HE sector. They also appreciated the opportunity to familiarize themselves with common AI multimodal LLMs and engage in hands-on activities, inspiring them to design their own learning activities using GenAI with their students.

Participants’ feedback also included suggestions for improvement, which are provided in the Section 7.

6. Analysis of Final Assignment Submitted by Participants

The most popular learning activity chosen by participants (15 out of 20 submissions) involved customizing a GenAI chatbot to act as a digital assistant/tutor for their students. Thirteen submissions focused on customizing an AI chatbot for undergraduate students, while two submissions described using a customized AI chatbot as an assistant for new PhD students. Participants chose either ChatGPT (individual subscription plan) or POE (free version) to customize their chatbots.

Two participants designed learning activities fostering ethical debates around GenAI, such as the risks of using institutional data, which are confidential, when formulating questions for GenAI agents like ChatGPT or Copilot.

Three out of 20 participants designed learning activities promoting the use of GenAI chatbots (ChatGPT and Copilot) for customizing rubrics in formative assessment and providing guidance to their students on the ethical use of GenAI chatbots in assessment.

It is particularly significant that none of the participants chose to design an assessment method that promoted the ethical use of GenAI. It is worth noting that the module covered the impact of GenAI on assessment in one of its units, encouraging participants to reflect on concepts such as academic integrity, authorship, and plagiarism. Authentic assessment was introduced within that unit as an approach to designing more intelligent assessment methods, including project-based learning, case studies, and inquiries. However, none of the participants applied the principles of authentic assessment to design an assessment method activity. During class discussions in the assessment unit, participants expressed that they would not feel confident applying assessment methods to make mindful use of GenAI in their own teaching practises until there was a clear institutional policy and guidance from the ULPGC on how use GenAI with students.

Due to the module’s timeline, participants were not able to try the learning activities proposed as a final project for their students, although some participants reported to the module leader that they were able to use their customized AI chatbots with undergraduate students during the summer of Semester 2. Participants found the chatbots useful for addressing generic questions and basic queries from large numbers of undergraduate students, but those chatbots could not answer more complex queries from undergraduate students in their final years (for example, queries about final dissertations or marking criteria).

7. Lessons Learnt

The key messages extracted from participants’ feedback after running the first presentation of the course were as follows:

• There is a need to simplify the current module’s syllabus by moving some advanced topics (e.g., authentic assessment and GenAI) to an advanced module, while keeping the current module foundational.
• There is a need to increase the number of practical sessions in the module, both on campus and online. This includes one-to-one tutorials focused on achieving effective results with GenAI tools through prompting engineering and problem formulation.
• Some participants suggested follow-up modules that focus on embedding AI literacy in curriculum design and utilizing Generative AI to innovate assessment methods.

8. Conclusions

As the first cohort of participants were confident with technology, no differences among participants belonging to different generations were observed in their ability to use prompting techniques to obtain the best results for Generative AI tools. Participants agreed that any training for new lecturers/teaching staff in higher education should include pedagogical uses of Generative AI and how to build AI literacy among teaching staff and students.

Participants consistently emphasized the need for institutional guidance on how to use GenAI to support learning, which pedagogical approaches to adopt, and how to guide students in using GenAI ethically. They also expressed a need for institutional direction on the new pedagogical approaches that should be adopted to make the most of GenAI.

The module exposed gaps in participants’ awareness of the risks and limitations of GenAI, highlighting the need for informed discussions on these topics. At the end of the module, participants shared the view that AI Literacy should not be limited to technical and digital skills: the module revealed that participants found critical awareness and reflection on GenAI to be just as valuable as learning how to provide effective prompts for these large language models.

The positive feedback and high score received for module CETD23 suggest that the module’s content, structure, and activities were well received and effective in achieving its learning objectives. This positive reception is a strong indicator that the module’s core components can be scaled and adapted to other educational settings. The module’s structure (based on self-contained units) also benefits its scalability, as it can be easily adjusted for shorter or longer courses, or even integrated into existing curricula.

Although the module was delivered following a blended learning approach, the combination of self-study learning activities and live online tutorials makes the module equally viable for delivery entirely online (with group activities adapted to individual assignments) or as a taught module, in which participants can engage with additional online learning activities to build up new knowledge.

The module’s focus on AI literacy and critical thinking is broadly applicable across disciplines, but the specific tools and examples used may need to be adjusted. For instance, a humanities course might emphasize the ethical implications of AI in society, while a STEM course might focus more on prompt-engineering techniques.

While CETD23 includes an international perspective, adapting the module to different educational contexts will require further localization, adapting the module’s aims and objectives to specific institutional guidance and policy on the use of GenAI. The units on regulatory frameworks, ethics, and data protection implications should also be adapted to cover relevant national and international legislation.

9. Further Recommendations

In June 2024, the ULPGC published an institutional policy on the use of Generative AI (GenAI). Since the aims of module CETD23 are closely aligned with this policy, further research into authentic assessment and the integration of AI tools like Microsoft Copilot is now encouraged.