Developing and Validating an AI-TPACK Assessment Framework: Enhancing Teacher Educators’ Professional Practice Through Authentic Artifacts

Abstract

In today’s digital era, teachers are expected to incorporate artificial intelligence (AI) into the classroom. Teacher educators must therefore model its use while evaluating their own AI-related knowledge to guide future teachers effectively. Existing assessments often rely on self-reporting questionnaires, which may introduce bias, and the TPACK (Technological-Pedagogical-Content-Knowledge) framework, which overlooks distinctive AI characteristics. This study develops and validates an AI-TPACK assessment tool for teacher educators, grounded in authentic pedagogy and systematically designed through the ADDIE model (Analysis, Design, Development, Implementation, and Evaluation). The study aims to identify AI-relevant TPACK components and add new ones; test the tool’s validity; and analyze teacher-educator competency patterns. The development involved dual literature reviews (22 TPACK studies; 34 AI studies) and empirical analysis of 60 authentic instructional artifacts. Five experts confirmed their content validity (CVR = 0.86, CVI = 0.91) and the inter-rater reliability (ICC = 0.84, range 0.76–0.88). The tool comprises 4 components—AIK, AIPK, AICK, and Integration—14 criteria, and 65 indicators, and reveals four competency patterns: technological innovator; pedagogical integrator; content developer; and beginner. The strong correlation (r = 0.78) between AIPK and integration underscores the importance of synergy. The tool contributes theoretically and practically to advancing teacher-educators’ AI knowledge and competency assessments.

1. Introduction

The ongoing revolution of artificial intelligence in education (AIEd) has reshaped society, presenting both opportunities and challenges to education (Organisation for Economic Co-operation and Development [OECD], 2023; United Nations Educational, Scientific and Cultural Organization [UNESCO], 2023). Among its advantages, the growing availability of relatively low-cost large language models (LLM) supports writing, knowledge generation (Pesovski et al., 2024), visualization of abstract concepts and intricate subject matter (Küchemann et al., 2024), and fosters more interactive learning among students (Shen, 2024). While the benefits of incorporating AI tools into teaching are evident, simply integrating them into the classroom does not guarantee the quality of education (Holmes et al., 2019). To fully harness the potential of AI, educators must purposefully embed it in their instruction, in line with specific learning objectives (Chiu, 2023a). Those responsible for shaping the next generation of educators play a pivotal role in this process. Effective teacher training requires deliberate and systematic design that fosters AI-oriented technological pedagogical content knowledge (AI-TPACK) to enable future teachers to integrate AI tools in their classrooms in an ethical and effective manner (Celik, 2023).

Preparing educators in this way is particularly urgent, as the AI era introduces challenges that are both novel and increasingly complex. Unlike traditional digital technologies, AI tools possess autonomous capabilities, machine learning, and dynamic adaptability (Russell & Norvig, 2020). Addressing these demands requires not only technical proficiency, but also a profound understanding of how AI can fundamentally alter both teaching and learning processes (Ng et al., 2021). Within this context, teacher educators serve as primary change agents in the educational ecosystem, thereby requiring a robust framework for understanding and evaluating their personal AI-integration competency. Their role extends beyond modeling effective practices; it also includes preparing future teachers to successfully navigate AI-driven environments, thereby contributing to systemic change across educational levels (Admiraal et al., 2017; Tondeur et al., 2017).

The present review seeks to advance the development and validation of a comprehensive AI-TPACK measurement tool that can be used by teacher educators. Grounded in the lesson planning transformation model (Stender et al., 2017) and relevant recommendations for AI-TPACK research (Mourlam et al., 2021), this paper focuses on the construction of an applicable instrument for facilitating comparative evaluation across the planning, implementation, and reflection stages of AI-integrated teaching. Given the paucity of clear implementations and descriptions of current AI-TPACK assessment tools in practice (Celik, 2023), the proposed instrument is designed for varied AI applications and aligned pedagogical approaches.

1.1. From TPACK to AI-TPACK

The TPACK model builds on Shulman’s (1986) seminal concept of pedagogical content knowledge (PCK), which highlights the specialized knowledge that teachers require in order to transform subject matter into comprehensible forms for students. Mishra and Koehler (2006) extended this framework by adding a technological dimension, yielding seven interrelated domains: (1) technological knowledge (TK)—the ability to understand and apply technology; (2) pedagogical knowledge (PK)—encompassing teaching methods and processes; (3) content knowledge (CK)—representing subject matter expertise; (4) technological content knowledge (TCK)—the interaction between technology and content; (5) technological pedagogical knowledge (TPK)—the manner in which technology supports pedagogy; (6) pedagogical content knowledge (PCK)—transforming content for teaching; and (7) the integrative TPACK—for the effective integration of technology.

Critiques of TPACK have prompted refinements, such as emphasizing contextual factors (e.g., school culture and resources) that shape its application (Voogt et al., 2013). Traditional evaluation tools include self-reporting questionnaires, notably Schmidt et al.’s (2009) TPACK instrument, which assesses perceptions across the seven domains, alongside qualitative approaches, such as reflections and observations (Koehler et al., 2013). However, reliance on these self-assessment methods has been criticized for subjectivity, overemphasis on self-perception, and limited ability to capture authentic classroom practices (Chai et al., 2013; Willermark, 2018). Although psychometric evaluations tend to reveal adequate reliability (e.g., Cronbach’s alpha > 0.80), they raise concerns about construct validity across diverse contexts (Valtonen et al., 2015).

While TPACK provides a solid foundation for digital integration, it falls short with regard to AI, given the latter’s distinct attributes—such as autonomy, adaptability, and ethical complexity (Holmes et al., 2019; Selwyn, 2019). As such, broad models may risk obscuring these idiosyncrasies, underscoring the need for tailored extensions, such as the AI-TPACK (Celik, 2023; Thyssen et al., 2023).

1.2. AI in Education: Current Trends

Recent syntheses and high-impact studies converge on core pedagogical trends and system priorities in education and teacher preparation, linking classroom practice to institutional readiness and policy needs. The dominant trajectory frames AI as an augmenting force that reshapes learning goals, assessment, and professional development rather than as an outright teacher replacement. Recent systematic and influential papers report a shift toward learning-with-AI that foregrounds higher-order thinking, reconfigured assessment, and integrated AI-ethical literacy in curricula, with growing emphasis on personalized instruction through educational chatbots for interactive support and large-scale adaptive learning systems (e.g., Bower et al., 2024; Chiu, 2023a; Walter, 2024). Reviews also document accelerating institutional pressure to embed pre-service and in-service training on AI literacy and prompt engineering while noting limited empirical evaluation and uneven institutional readiness, particularly in preparing educators to navigate intelligent tutoring systems and recommendation systems for resource curation (Alshahrani et al., 2024; López-Regalado et al., 2024). At the system level, scholars highlight governance concerns—equity, bias, data protection—and recommend new professional standards, assessment models, and co-designed pilots to preserve relational teaching functions while scaling personalization through adaptive analytics for data-informed decision making and supporting multimodal learning environments enabled by large language models (LLMs) and teacher productivity (Alshahrani et al., 2024; Chiu, 2023b; López-Regalado et al., 2024; Walter, 2024).

1.3. Teacher Education in the AI Age

Teacher education in the AI Age encounters multifaceted challenges, particularly with the rapid integration of AI technologies. As noted by Tan et al. (2025), a significant research imbalance favors AI applications in teaching over professional development, alongside ethical concerns such as privacy breaches and biases. While previous research has advocated for professional development to ensure seamless integration (Chiu et al., 2023; Ng et al., 2023) and highlighted the need for innovative pedagogies (Zawacki-Richter et al., 2019), these models often address technology in a general sense.

A critical gap persists within existing inquiries. While models of professional growth have emphasized TPACK-based approaches (Kim, 2024; Sun et al., 2023) and self-efficacy enhancements (Yang et al., 2024), they overwhelmingly overlook the unique demands of AI—such as algorithmic literacy and ethical integration (Celik, 2023). This is particularly evident in the current state of technology integration in teacher education, which remains oriented towards general tools (Tondeur et al., 2017).

Teacher educators, as pivotal change leaders (Tondeur et al., 2018), need specialized frameworks to navigate these new challenges. This underscores the urgent need for a framework like AI-TPACK, which is designed to equip them with the tools for cultivating adaptive expertise in AI-enhanced education. Such professional growth greatly depends on transforming academic knowledge into practical scripts through iterative cycles of planning, action, and reflection (Stender et al., 2017)

1.4. Extending and Refining the TPACK Model

Beyond the TPACK, several models have sought to capture evolving digital practices. Puentedura’s (2006) SAMR model (substitution, augmentation, modification, and redefinition) and Hughes’s (2005) RAT framework (replacement, amplification, transformation) represent early attempts. Technology-specific adaptations include Ebner et al.’s (2010) incorporation of Web 2.0 for social collaborations and Radianti et al.’s (2020) VR-TPACK for immersive experiences. Finally, contextual expansions encompass Young’s (2016) cultural TPACK and Willermark’s (2018) leadership-oriented variant.

However, despite these advances, prior extensions have rarely addressed the unique demands of AI. Models such as SAMR and RAT emphasize integration levels, but overlook autonomous learning and ethical implications, often resulting in superficial applications (Selwyn, 2019). Broader frameworks, such as DPACK, account for digital culture—yet risk downplaying AI-specific features, such as algorithmic bias and adaptability (Thyssen et al., 2023). These gaps highlight the need for a dedicated AI-TPACK extension, grounded in authentic pedagogical artifacts, to capture nuanced integrations beyond self-reports (Harris et al., 2010; Hofer et al., 2011).

1.5. Early Conceptualizations and Empirical Advances

Initial attempts to formulate an AI-TPACK highlight the educational potential of AI. Celik (2023) proposes a competency framework that emphasizes ethical and pedagogical integration, while Ng et al. (2021) outline AI literacy for teachers, and Bobula (2024) presents a systematic review of AI implications. Core components emerging from these efforts include AI Knowledge (AIK) for understanding capabilities and limitations, AI Ethics for ensuring fairness and transparency, and AI Pedagogy for effective instructional integration. Empirical studies provide further insights: Chatterjee and Bhattacharjee (2020), for example, examine teachers’ sentiments towards AI, Chiu and Chai (2020) explore future readiness, and Kim (2024) investigates skill-building. Collectively, these studies reveal only nascent patterns and underscore the scarcity of validated tools.

In order to refine AI-TPACK both conceptually and empirically, Ning et al. (2024) introduced a seven-element framework, including AI-technological knowledge (AI-TK) and its intersections (Figure 1), yielding seven interrelated domains: AI-technological knowledge (AI-TK), which involves understanding the core principles and functions of AI technologies; pedagogical knowledge (PK), which encompasses teaching methods and processes, particularly how to integrate AI tools effectively; content knowledge (CK), which represents subject matter expertise and how it intersects with AI applications; AI-technological content knowledge (AI-TCK), which is the ability to integrate specific AI tools to represent particular subject matter; AI-technological pedagogical knowledge (AI-TPK), which is the knowledge of how to use AI tools to support specific teaching methods; pedagogical content knowledge (PCK), which involves the transformation of subject matter for teaching with an emphasis on how AI can assist in this process; and finally, AI-TPACK, the overarching integrative knowledge required to effectively use AI to enhance both teaching and learning of specific content.

Figure 1. AI-TPACK structural diagram by (Ning et al., 2024).

The researchers found predictive relationships among the components, with technical knowledge exerting the strongest influence. Similarly, Mishra et al. (2023) adapted the TPACK for generative AI, underscoring its highly adaptable and social dimensions. Empirical validations have further strengthened the AI-TPACK, though notable methodological gaps remain. Al-Abdullatif (2024) developed the Intelligent TPACK framework and validated it through structural equation modeling with 312 faculty members. Their analysis shows that AI literacy significantly predicts AI acceptance (β = 0.43, p < 0.001) with the model explaining 67% of the variance. Finally, Ng et al. (2023) validated a five-factor AI competency model (teacher professional engagement, instructional design, content choices, learning competencies, and facilitating learners’ AI competency) using confirmatory factor analysis (χ² = 234.56, df = 125, CFI = 0.95). Their findings established strong convergent validity with 21st-century skills (r = 0.72, p < 0.001).

However, these empirical advances predominantly rely on self-reporting methodologies, exposing a critical gap in the field of study-unit assessments. Choudhury et al. (2024) validated the AI-TPACK scale in Indian settings, showing that cultural factors necessitate adaptation (RMSEA = 0.068, CFI = 0.93). Building on this, Ng et al. (2025) demonstrate that AI competency positively predicts TPACK, which in turn mediates teaching performance. Yet a fundamental limitation persists: the absence of validated instruments grounded in authentic pedagogical artifacts. This gap underscores the need for artifact-based assessment tools that can evaluate the AI-TPACK as it emerges in real educational settings, such as lesson plans, instructional materials, and reflective practices.

1.6. Assessing and Developing AI-TPACK

To measure TPACK, self-reporting surveys are often used (Archambault & Crippen, 2009; Schmidt et al., 2009). However, despite their benefits, such methods are prone to biases, such as overestimations within emerging domains (Drummond & Sweeney, 2017) and a failure to differentiate between technologies, which inadequately addresses the dynamic nature of AI (Valtonen et al., 2015). Critiques also emphasize the discrepancies between self-reports and actual practices (Willermark, 2018), highlighting the need for more authentic, artifact-based assessments that draw on lesson plans and instructional materials (Harris et al., 2010; Hofer et al., 2011).

Building on these concerns, psychometric principles remain essential, including content validity, construct validity, and reliability (Seifert & Lindmeier, 2024). For AI-TPACK, artifact-based tools could offer a more authentic evaluation that captures ethical, contextual, and adaptive dimensions (Mourlam et al., 2021). Although recent validations, such as Al-Abdullatif’s (2024) model (CFI = 0.94), underscore the mediating role of trust, instruments that are grounded in authentic artifacts remain scarce. This highlights the need for hybrid approaches that combine psychometric rigor with real-world evidence.

Beyond measurement, the cultivation of AI-TPACK is equally significant. Professional development initiatives draw on established models such as Guskey’s (2002) attitudinal change model, addressing barriers through sustained training (Garet et al., 2001) and professional communities (Wenger et al., 2002). In AI contexts, competencies range from basic tool use to advanced ethical integration (Celik et al., 2024). Clustering analyses reveal a range of profiles, from low to high integrators, shaped by self-efficacy and pedagogical beliefs (Shrestha et al., 2025).

Emerging evidence also reveals discipline-related variations; among math teachers in China, AI-TPACK remains at the preliminary stage, with self-efficacy supporting progress but traditional beliefs hindering it (Xie & Luo, 2025). Studies on generative AI integration highlight four user types—from cautious adapters to pedagogical innovators—and propose the PSAMR (including prohibition) as an extension of the SAMR (Li et al., 2025). Systematic reviews of AI-related professional development demonstrate that integrating TPACK into workshop designs and other settings enhances innovation but requires contextual adaptation (Ng et al., 2025). Finally, institutional support and ethical considerations further shape these patterns, emphasizing the need for tailored programs for diverse profiles.

To examine these developments rigorously, a range of research methodologies have been employed. Quantitative methodologies, including surveys for tool validation, longitudinal tracking of development, and comparative analyses across groups, provide measurable insights into AI-TPACK evolution. Qualitative approaches, such as ethnographic observations in training settings, action research that involves teachers as investigators, and case studies of instructional units, provide contextual depth (Mourlam et al., 2021). Finally, mixed methods, such as McKenney and Reeves’s (2018) design-based research, content analysis of curricula, and social network examinations of collaborations, combine these strengths to enable holistic inquiry. These methodologies all play a key role in capturing the complexities of the AI-TPACK, enabling rigorous exploration of knowledge interplay, while informing evidence-based extensions. For tool validation, structural equation modeling ensures psychometric robustness (Al-Abdullatif, 2024), while artifact analysis supports authenticity (Harris et al., 2010).

In conclusion, research has established the TPACK model as a widely recognized framework for technology integration (Mishra & Koehler, 2006) while highlighting the educational promise of AI (Holmes et al., 2019) and stimulating discourse on teacher training in the AI era (Ng et al., 2021). Yet theoretical gaps persist, including the absence of a fully articulated AI-TPACK model that is grounded in authentic output. Methodologically, validated artifact-based instruments remain scarce, while practical shortcomings continue to be identified, following the neglecting of disciplinary and cultural variations (Choudhury et al., 2024). Cross-cultural validation is also limited, with research largely confined to Western settings, underscoring the need for context-sensitive adaptations.

2. Research Aims and Questions

This study advances an empirically grounded AI-TPACK framework through the development and validation of a comprehensive AI-TPACK assessment instrument based on authentic pedagogical artifacts from teacher educators. The objectives are threefold: to define and map the AI-TPACK components, establish a reliable measurement tool, and identify competency patterns that can guide professional development in teacher education programs.

The following three research questions were defined:

What components of AI-TPACK emerge within the authentic pedagogical outputs provided by teacher educators, and how can these elements be defined and mapped?

To what extent does the developed tool provide a valid and reliable measure of the theoretical AI-TPACK construct?

What levels of AI-TPACK can be identified among teacher educators’ artifacts, and what characterizes each level?

Addressing these questions could provide a novel framework for refining curricula, enhancing assessment practices, and informing policies for AI integration in teacher education programs, ultimately fostering systemic improvements across educational settings.

3. Methodology

This study employed a systematic multi-phase approach based on the ADDIE (Analysis, Design, Development, Implementation, Evaluation) model to develop and validate a comprehensive AI-TPACK assessment instrument. This model, widely used in instructional design (Molenda, 2015), provides a structured methodology for creating research-based educational tools. The ADDIE cycle, particularly the continuous feedback loop between Development and Evaluation, provides the robust methodological rigor necessary to iteratively build, refine, and empirically validate a complex, multi-dimensional assessment framework. The following are details of the five-phase process: conceptual grounding, literature review, instrument development, implementation, and evaluation.

3.1. Phase 1: Conceptual Grounding—Defining Purpose and Theoretical Foundation

The first phase focused on establishing the instrument’s purpose and theoretical foundation. The application context was defined, and appropriate theoretical models were selected to ensure both conceptual and practical relevance. Grounded in the TPACK framework, the instrument was designed around three core dimensions: AI knowledge (AIK), AI pedagogical knowledge (AIPK), and AI content knowledge (AICK). These dimensions were elaborated into distinct AI-TPACK components to capture comprehensive and precise insights into the educational context.

3.2. Phase 2: Literature Integration—Building the Evidence Base

The second phase of the process empirically justified the theoretical choices and established an empirical basis for the instrument. Two systematic reviews were conducted. First, existing TPACK measurement instruments were examined. Studies were included if they used, described, or developed instruments for measuring TPACK among pre-service or in-service teacher educators. The search strategy combined AI-powered platforms, such as SciSpace, with established academic databases (e.g., Web of Science, Scopus, ERIC, Google Scholar) using the primary search query “TPACK” AND (“instrument” OR “rubric” OR “assessment”) AND (“validation” OR “development”) AND “teacher”. Complementary searches of gray literature and targeted themes (e.g., educational technology and teacher training) expanded the coverage. This process led to the identification of 505 records. Following systematic screening and eligibility assessments, 25 studies were included in the qualitative synthesis—comprising 16 journal articles, 6 conference papers, and 3 book chapters (Figure 2).

Figure 2. PRISMA flow diagram: TPACK measurement instruments literature review.

Next, as the second part of phase 2, a targeted systematic literature review was conducted with the aim of identifying methodologically rigorous studies on quality frameworks, pedagogical effectiveness, teacher competency development, and implementation assessments—all in the context of AI in teacher education. The search strategy was designed to capture studies that employ validated frameworks or instruments. This search yielded 294 studies, all published between 2020 and 2024, addressing topics such as framework validation, pedagogical effectiveness, longitudinal and multi-institutional assessment, and teacher competence development. Additional targeted search sets identified 111 studies on the impact of AI on assessment methodologies and pedagogical evaluation and 32 studies on innovative assessment strategies in AI-enhanced teacher education. Finally, 73 records (55 citations and 18 references) complemented this process, bringing the total to 510.

Strict eligibility criteria were applied. Studies were required to demonstrate methodological rigor, employ validated frameworks or instruments, and address at least one of the following topics: development and validation of quality frameworks, assessment of pedagogical effectiveness, evaluation of teacher competencies, or implementation supported by empirical evidence.

Following the removal of duplicate items, 396 records were screened by title and abstract. Of these, 282 were excluded for insufficient methodological or empirical quality. The remaining 114 full-text articles were then reviewed in detail by the authors, resulting in the exclusion of another 96 studies, due to the lack of a validated framework (n = 35), insufficient empirical assessment (n = 30), and inadequate quality criteria (n = 31). The final synthesis comprised 18 studies that met all inclusion criteria, representing rigorous and empirically grounded research on frameworks for AI integration in teacher education (Figure 3).

Figure 3. PRISMA flow diagram: AI teacher education quality frameworks literature review.

3.3. Phase 3: Instrument Development—Constructing the Assessment Tool

The third phase focused on the initial instrument development. A comprehensive synthesis of 43 studies revealed well-established assessment approaches for TPACK measurement, consistently supported by validation evidence and reliability standards. From this review, 37 assessment criteria were extracted and adapted to the AI-TPACK framework, ensuring coverage of its three core dimensions.

The instrument was then refined through a rigorous content validation process. An expert panel of five specialists evaluated content validity using Lawshe’s method, yielding strong results (CVR = 0.86 and CVI = 0.91). All panel members held leadership roles in digital innovation units within teacher education settings, had at least five years’ practical experience in technology integration, demonstrated expertise in training educators in educational technologies, had an academic background in education or learning technologies, and were highly familiar with AI implementation in teacher education (Table 1). Each expert independently rated item relevance, from 1 (not relevant) to 4 (highly relevant), and item clarity, from 1 (unclear) to 4 (very clear), via an online survey. They then participated in a 90 min structured group discussion via the Zoom platform to resolve rating disagreements and finalize any necessary revisions. CVR was calculated using Lawshe’s formula. According to Lawshe’s (1975) original table, the minimum threshold for a 5-expert panel is 0.99 at p < 0.05 (one-tailed test); however, recalculations by Ayre and Scally (2014) suggest 1.00 due to exact binomial probabilities. Given the small panel size and practical considerations, we adopted a flexible approach, retaining items with CVR ≥ 0.60 when supported by high mean relevance ratings (M ≥ 3.6) and theoretical relevance, in line with common adaptations in validation studies. CVI was computed as the proportion of items that were rated 3 or 4 by at least 80% of the experts. Items with a CVI lower than 0.80 were revised.

Table 1. Expert panel demographics.

Expert	Degree/Field	Current Role	Years of Experience (Tech Integration/AI)
E1	Ph.D./Learning Tech	Head of Digital Innovation Unit	12
E2	Ed.D./Curriculum Design	Lecturer, Teacher Ed.	8
E3	Ph.D./Cognitive Science	AI in Education Consultant	10
E4	M.A./Special Education	Consultant of Educational Technology	5
E5	M.A./STEM Education	Instructional Designer, AI/Ed Tech	9

3.4. Phase 4: Implementation—Pilot Testing and Refinement

The fourth stage, implementation, involved applying the novel instrument to 60 authentic artifacts collected between 2023 and 2025 during three cycles of a 180 h course for teacher educators entitled “Digital learning design and innovation in academia” at the [name removed for blind review] R&D institution for teachers educators. The 60 participating teacher educators were composed of 48 academic faculty and 12 Teachers’ R&D staff, representing 15 different institutions from different regions across the country and various sectors.

The artifacts included lesson plans, instructional units, and reflective accounts drawn from diverse contexts, including special education (40%), arts and education (25%), values education (20%), and techno-pedagogy (15%). Beyond serving as descriptive products, these artifacts embodied authentic, process-oriented evidence of teaching design and reflective inquiry. They captured the integration of digital tools and platforms—such as Canva, Base44, and Nearpod—used for documenting, analyzing, and sharing pedagogical practices. Each artifact represented a dynamic synthesis of AI-TPACK dimensions, demonstrating how technological innovation was intentionally aligned with pedagogical reasoning and adaptive content design, following frameworks such as Universal Design for Learning (UDL) and reflective practice. This ensured representation across high levels of AI-TPACK knowledge and skills and enabled the instrument to be validated against a wide range of integration practices. To maintain methodological continuity, the same panel of five experts applied the instrument to the authentic artifacts.

As mentioned, the literature review yielded 37 theoretical criteria for assessing TPACK competency; yet the empirical analysis of the 60 artifacts revealed gaps that required the addition of four dimensions: (1) specific technological proficiency levels, (2) performance-based student engagement measures, (3) structured critical reflection, and (4) artifact-based innovation measures. This finding underscores the importance of authentic artifact-based validation for developing practical measurement tools.

3.5. Phase 5: Evaluation—Validation and Finalization

The fifth and final phase of the study involved evaluation and validation. Inter-rater reliability was assessed by randomly selecting 20 (33%) of the artifacts for independent analysis by the five experts. Inter-rater reliability was calculated using a two-way mixed-effects model for absolute agreement of average measures (ICC[3,k]), as recommended for assessment instruments (Koo & Li, 2016), supplemented by Cohen’s Kappa for ordinal-level agreement between raters. Statistical analysis was conducted using the SPSS software V.30 (IBM Corp., Armonk, NY, USA). Reliability and validity were assessed across several dimensions: content validity (CVR and CVI calculations following Lawshe’s method); inter-rater reliability; and intraclass correlation coefficient (ICC) with 95% confidence intervals. For CVR, the original threshold for a 5-expert panel is 0.99 (or 1.00 per Ayre & Scally, 2014 recalculations); items slightly below this (e.g., CVR = 0.60) were retained based on high mean ratings (M = 3.6), theoretical importance, and expert discussion. CVI was computed as the proportion of items rated 3 or 4 by ≥80% of experts, with revisions for those below 0.80. Internal consistency was not assessed, as the tool focuses on observable behaviors rather than latent constructs. These procedures confirmed the instrument’s robustness for capturing AI-TPACK in authentic contexts.

The validation process resulted in a comprehensive AI-TPACK assessment tool comprising 14 criteria and 65 observable indicators. The instrument supports both quantitative scoring (on a 1–4 scale) and qualitative analysis, enabling the identification of AI integration patterns within authentic pedagogical artifacts.

Ethical approval was obtained from the Ethics Committee at the authors’ affiliated institution [MOFET R&D Institution]. Informed consent was received from all participants. Data were anonymized using unique identifier codes, with linking information stored separately in password-protected files. All sensitive data were saved in encrypted, secured servers, compliant with institutional data protection protocols.

4. Findings

4.1. Instrument Purpose and Theoretical Foundation

The instrument presented in this paper was designed to measure the AI-TPACK of teacher educators who integrate AI into their teaching through written deliverables (planning), documented learning units (implementation), and written reflections. To address the situational and contextual nature of the AI-TPACK, the tool specifies three main components: (1) AI knowledge (AIK); (2) AI pedagogical knowledge (AIPK); and (3) AI content knowledge (AICK). Within this structure, AIPK, AICK, and AI-TPACK were chosen as the primary levels of assessment, while the AIK component and integration served as foundational elements. The traditional TPACK knowledge components (TK, PK, and CK) were excluded from the instrument, since the AI-related elements require separate assessment measurements.

To substantiate these components, established theorical models were incorporated. The SAMR model (Puentedura, 2006, 2013) was used to inform the AIPK by defining four levels of technology integration (substitution, augmentation, modification, and redefinition). The dual coding theory (Paivio, 1991) was used to support the AICK dimension, highlighting how technology influences knowledge processing. Finally, Schön’s model of reflection (1983)—which distinguishes between technical, practical, and critical reflection—was used to evaluate the depth of reflective practice.

4.2. Systematic Literature Review

To establish a robust foundation for the instrument, two systematic literature reviews were conducted. The first examined existing TPACK measurement tools to identify their strengths and limitations, while the second focused on AI integration in teacher education to determine key pedagogical criteria and competencies. Our analysis of existing tools revealed a significant gap: none address the unique characteristics of AI technologies. Although they do address digital technologies, they lack criteria for assessing the pedagogical, content, and ethical implications of advanced AI tools. This limitation underscores the need for a new, AI-specific instrument. The second review, which synthesized findings from 34 high-quality studies, identified key criteria for effective AI integration in teacher training. These criteria were categorized based on the conceptual framework presented above, providing the empirical basis for the instrument’s components and emphasizing the importance of moving beyond general technology use to target AI-specific competencies, such as language processing tools and generative AI. The complete set of criteria and their sources, derived from both reviews, are presented in Table 2.

Table 2. Summary of assessment criteria from the literature review (n = 37).

Category	Criterion ID Range	Core Assessment Indicator Examples
AI-Technology Knowledge (AIK)	ATK-1 to ATK-7	Understanding AI Technologies, Proficiency with Educational AI Tools, Technological Problem-Solving
AI-Pedagogical Knowledge (AIPK)	APK-1 to APK-8	Personalized Learning Implementation, Automated and Immediate Feedback, Differentiated Instruction
AI-Content Knowledge (AICK)	ACK-1 to ACK-7	Subject-Specific AI Applications, Concept Demonstration through AI, Quality Content Creation
AI-TPACK Integration	AIT-1 to AIT-8	Holistic Lesson Planning, Clear Learning Objectives, Effective Implementation, Reflection and Improvement
AI Ethics and Responsibility (AER)	AER-1 to AER-7	Privacy and Data Security, Algorithmic Transparency, Bias Prevention, Responsible Use, AI Literacy

This summary table provides a concise overview of the conceptual framework. The full, detailed list of criteria, including the “Description” column and all 65 observable indicators, is presented in Appendix A for complete methodological transparency.

4.3. Initial Instrument Development and Content Validity Assessments

The initial version of the instrument was developed by synthesizing findings from the two literature reviews. This process yielded 37 assessment criteria, ensuring the tool’s relevance to the specific AI context. These criteria were operationalized into observable indicators to create a four-level scoring system. To ensure consistency and objectivity, a coding manual was drafted with explicit examples for each indicator; moreover, a weighting system was applied based on the frequency of each indicator in the literature—an approach that focused on a variety of AI tools in teacher education and provided a clear contextual focus for the AI-TPACK components from the onset.

The instrument was further refined through rigorous content validation by five experts in AI and teacher education. As mentioned, the process yielded strong results, with a CVR of 0.86 (above the required threshold of 0.62) and a CVI of 0.91 (also above the required threshold of 0.78). Combined with overall expert agreement of 94%, these outcomes confirm the instrument’s strong content validity. In addition, the experts emphasized the need to align the tool with evolving digital realities in classrooms. They affirmed its strong relevance to the authentic artifacts and recommended enhancing its focus on accessibility, equity in AI use, and the cultivation of critical thinking towards AI technologies. Based on this feedback, revisions were made to the instrument, including refining definitions in phases 2–3 (particularly within the AIPK component), adding concrete examples from expert practice, simplifying some terminology to minimize technological jargon, and incorporating new AI tools that emerged during the research period.

4.4. Field-Testing and Final Instrument Development

The validation process progressed from expert review to application with 60 authentic artifacts that were produced by 60 teacher educators. Analyzing real pedagogical materials revealed gaps that had been overlooked in the literature, leading to the addition of new practice-driven criteria. As a result, the initial 37 criteria were revised into a refined, final version. The completed tool consists of 14 criteria organized under 4 primary components and 3 supporting measures, encompassing 65 observable indicators. These revisions were substantiated through expert ratings. Averages and CVR values for each item are presented in Figure 4.

Figure 4. Content Validity Ratio (CVR) per item, based on expert ratings.

The bar chart in Figure 4 illustrates the Content Validity Ratio (CVR) for each of the 13 core assessment items, as rated by the panel of five experts (N = 5). The red dashed line denotes a practical CVR threshold of 0.60 used in this study for retention, although Lawshe’s (1975) original table specifies 0.99 for a five-expert panel. Items below 0.99 but above 0.60 (e.g., ‘Advanced technical integration’ and ‘Synergy between components’) were retained based on their high average rating (M = 3.6) and essential theoretical contribution to the AI-TPACK construct, as recommended in recalibrations by Ayre and Scally (2014). The overall Content Validity analysis yielded strong results for the entire instrument, with an average CVR = 0.86 and a Content Validity Index (CVI) = 0.91. These values significantly exceed the minimum thresholds required (CVR_{min} = 0.62; CVI_{min} = 0.78) and confirm the tool’s robust content validity. Overall agreement among the experts stood at 94% (items approved without changes).

Note on Scale: The ratings for item relevance ranged from 1 (not relevant) to 4 (highly relevant). While the majority of items surpassed the CVR threshold, the items with CVR = 0.60 (such as ‘Advanced technical integration’ and ‘Synergy between components’) were marginally below the critical threshold. These items were retained in the final instrument based on their high average rating (M = 3.6) and their essential theoretical contribution to the AI-TPACK construct.

The pie chart in Figure 5 illustrates the normalized weight distribution of the final AI-TPACK assessment instrument, based on the fundamental 70%/30% theoretical split (Core Components/Supporting Measures). The overall scoring model allocates 70% of the total score to the four core AI-TPACK knowledge components (AIK, AIPK, AICK, Integration), and 30% to the three crucial supporting measures (Reflection, Engagement, and Innovation). This distribution highlights two key structural findings: (1) The Supporting Measures collectively constitute the largest single segment, emphasizing the tool’s focus on practical, non-technical aspects like reflection and innovation. (2) Among the Core Components, AI-Pedagogical Knowledge (AIPK) carries the greatest proportional weight, underscoring the priority of pedagogical application over pure technical knowledge in the instrument’s design. The full breakdown of the 14 criteria, including the final normalized weight percentage for each item, is detailed in the assessment rubric located in Appendix B.

Figure 5. Weight distribution of AI-TPACK assessment tool components (normalized).

The empirical analysis also informed the refinement of the instrument’s scoring system. Indicators were re-evaluated and their weights adjusted according to their demonstrated impact on teaching quality via the authentic artifacts. Additional indicators were inductively derived from these materials. For instance, new criteria were added to distinguish between essential and non-essential AI tools and to assess the alignment of time allocation with learning objectives.

This iterative process—combining theory-based development, expert validation, and empirical field-testing—ensured that the final instrument was both theoretically sound and practically applicable. It provided a valid and reliable tool for measuring AI-TPACK among teacher educators—through both quantitative scoring (on a 1–4 scale) and qualitative analysis of AI integration patterns in authentic pedagogical artifacts.

Figure 5 presents the final structure of the AI-TPACK assessment tool, refined through a rigorous two-phase validation process. The tool comprises 14 criteria and 65 observable indicators, enabling dual scoring and analysis. The weighting system allocates 70% of the total score to the four core AI-TPACK components and 30% to the three supporting measures, reflecting their relative importance in evaluating practical AI integration in education.

4.5. Inter-Rater Reliability

The instrument’s reliability was tested on a random subset of 20 (33%) of the artifacts, independently rated by the same five experts from the validation phase. Results indicated substantial agreement among raters, with Cohen’s Kappa = 0.704 (Landis & Koch, 1977), and an overall ICC = 0.84, indicating very good reliability (Koo & Li, 2016). As shown in Figure 6 the highest agreement was for AIK (ICC = 0.88, excellent) and the lowest for Integration (ICC = 0.76, good). Moreover, all components exceeded the accepted reliability threshold (>0.75), confirming the robustness of the tool across domains and providing a solid foundation for identifying competency patterns in teacher educators’ AI-TPACK.

Figure 6. Inter-rater reliability (ICC) per component.

4.6. Statistical Findings and AI-TPACK Patterns

The analysis of 60 pedagogical artifacts, all provided by teacher educators, revealed a comprehensive picture of their skill levels and patterns of AI integration in their teaching. The overall mean AI-TPACK score was M = 2.55 (out of 4; SD = 0.73), indicating a moderate level of competence across the sample. Distribution by competency level showed that 5% (n = 3) were at the basic level (M = 1.25); 38% (n = 23) were at the intermediate level (M = 2.15); 45% (n = 27) were at the intermediate–high level (M = 2.95); and 12% (n = 7) were at the advanced level (M = 3.65). This indicates that most teacher educators demonstrate intermediate-to-high levels of AI competency, suggesting a solid foundation with room for further development. Figure 7 presents a detailed breakdown of the descriptive statistics for each AI-TPACK component, highlighting variations across AIK, AIPK, and AICK.

Figure 7. Mean competency score per AI-TPACK component.

Figure 7 illustrates the average competency scores (mean) across the seven AI-TPACK components. The red dashed line represents the overall sample mean (M = 2.55). The analysis reveals significant variations in competence:

Highest Competency: The highest mean scores were recorded for Engagement (M = 2.89) and AICK (M = 2.78), suggesting teacher educators are most proficient in using AI to enrich content and involve students.

Lowest Competency: The lowest mean score was for Innovation (M = 2.23), followed by Integration (M = 2.34), highlighting a need for professional development focused on creating novel and holistic AI-based solutions.

Overall Status: Most components fell within the intermediate–high range, supporting the overall finding of a moderate level of AI-TPACK competence in the sample (Moverall = 2.55).

4.7. Identified Integration Patterns

The qualitative analysis revealed four distinct patterns of AI-TPACK integration among the participating teacher educators. First, the technological innovators (15%) exhibited advanced AIK (M ≥ 3.5) and moderate AIPK (M = 2.5–3.0)—demonstrating advanced technical skills and the ability to develop original AI solutions. Next, the pedagogical integrators (35%) exhibited high AIPK (M ≥ 3.0) and moderate AIK (M = 2.0–3.0)—demonstrating their ability to adapt AI tools for enhancing existing teaching methods. The content creators (30%) exhibited high AICK (M ≥ 3.0)—they prioritized using AI to generate and update subject-specific learning materials. Finally, the novice learners (20%) were characterized by low-to-intermediate scores across all components (M = 1.5–2.5)—they tended to only use basic, widely available AI tools, typically applied at a substitutional level.

The analysis also revealed significant correlations between the tool’s components (Figure 8). The strongest correlation (r = 0.78) was seen between AIPK and Integration, indicating that effective AI integration is heavily dependent on a robust pedagogical understanding of the technology. These findings underscore the persistent gap between technical expertise and pedagogical implementation.

Figure 8. Correlation of components with integration (INT).

The correlation analysis was conducted using Pearson’s correlation coefficient (r) to examine the linear relationships between the AI-TPACK components. This parametric test was justified because the data analyzed were the composite mean scores for each component (e.g., AIK, AIPK, Integration). Since these scores are calculated as the average of multiple ordinal indicators, they are statistically robustly treated as continuous, approximating interval-level data, which is a standard psychometric practice for correlational analysis between composite factor scores.

Figure 8 visually confirms the strongest correlation found in the analysis, which is between AIPK (AI Pedagogical Knowledge) and Integration (r = 0.78). This strong relationship, highlighted by the fact that AIPK is the only component to clearly surpass the r = 0.70 threshold, supports the theoretical claim that pedagogical understanding is the central driver for effective AI integration. Conversely, the technical component (AIK) exhibits a significantly weaker correlation r = 0.48, underscoring the gap between technical expertise and meaningful implementation. These findings underscore the persistent gap between technical expertise and pedagogical implementation.

Overall, the comprehensive analysis of 60 authentic artifacts confirmed the validity and reliability of the innovative AI-TPACK assessment tool presented in this paper. The results indicate a moderate level of AI-TPACK competency among teacher educators and reveal distinct integration patterns. Strong inter-component correlations further highlight the central role of pedagogical knowledge and reflective practices for successful AI integration. The tool thus provides a solid foundation for the objective and systematic measurement of AI-TPACK competency among teacher educators—enabling the identification of personal strengths and weaknesses, to guide future professional development.

5. Discussion

This study aimed to develop and validate a comprehensive tool for measuring AI-TPACK proficiency through the analysis of authentic pedagogical outputs of teacher educators. The findings demonstrate that the proposed tool is both valid and reliable, while providing a nuanced picture of teacher-educator’s AI competencies and patterns that characterize their integration of AI into teaching. Interpreting these results in line with the three research questions highlights several key contributions to both theory and practice.

The first question focused on the components of AI-TPACK as manifested in authentic teacher-educator artifacts. The study provides a clear response to this question by defining an assessment tool that extends the classic TPACK (Mishra & Koehler, 2006), to capture the unique affordances and challenges of AI. Unlike existing tools, which often remain generic and overlook the discrete features of AI (Chai et al., 2013; Valtonen et al., 2015), the present instrument incorporates specific knowledge components—AIK, AIPK, and AICK—tailored to the pedagogical use of AI. Developed iteratively through literature analysis and empirical materials from the field, the final structure of 14 criteria and 65 indicators underscores the necessity of grounding theoretical constructs in authentic practice—thereby echoing similar calls in the literature (Al-Abdullatif, 2024; Graham, 2011; Ng et al., 2023; Willermark, 2018).

The second research question confirmed the tool’s methodological rigor. A panel of five specialists confirmed strong content validity, while inter-related measures provided evidence of component reliability. These results address concerns about the limitations of existing TPACK tools with regard to AI (Drummond & Sweeney, 2017; Schmidt et al., 2009). Moreover, by grounding measurements in lesson plans, reflections, and other teacher-educator outputs—rather than relying on self-reports—the tool aligns with recommendations for artifact-based assessment tools (Harris et al., 2010; Hofer et al., 2011; König et al., 2020; Mourlam et al., 2021).

The third research question explored patterns and proficiency levels among teacher educators. The findings reveal moderate competency levels, but substantial variation across individuals. Four integration patterns emerged—technological innovator, pedagogical integrator, content developer, and beginner—mirroring typologies from previous studies on technology adoption (e.g., Puentedura, 2013; Kim, 2024; Shrestha et al., 2025; Tondeur et al., 2017). The distribution of these patterns suggests that while many teacher educators are capable of integrating AI meaningfully, their strengths are unevenly distributed, reinforcing the importance of tailored professional development programs (Tan et al., 2025).

Notably, the strong correlation between AIPK and Integration highlights pedagogy as the central driver of successful AI use. This supports theoretical models such SAMR (Puentedura, 2006), and affirms that transformative integration (e.g., modification and redefinition) depends less on technical knowledge (AIK) and more on pedagogical vision and application (Mishra et al., 2023; Ning et al., 2024). This insight provides theoretical validation for the TPACK model (Mishra & Koehler, 2006) and suggests that successful integration requires synergy among the various components. Additionally, the overemphasis on isolated components, particularly technical knowledge, at the expense of pedagogy underscores a recurring gap noted in recent reviews (Celik, 2023; Chiu & Chai, 2020).

Taken together, these findings confirm the tool’s theoretical contribution by extending TPACK to AI contexts, demonstrate its reliability and validity through empirical testing, and provide insights into the heterogeneous ways teacher educators approach AI integration. They also emphasize that professional development must move beyond generic training to address specific competency profiles and foster synergy across knowledge domains, thereby preparing teacher educators to act as effective change leaders in the era of AI.

5.1. Practical Implications

This study carries important implications regarding professional development programs for teacher educators. The validated tool enables the identification of AI-TPACK profiles, supporting teacher-training planners—in line with literature on the role of teacher educators as change leaders (Admiraal et al., 2017; Tondeur et al., 2018). Rather than offering generic training, the tool allows for targeted guidance. For example, technological innovators may benefit from greater emphasis on AI pedagogy, while pedagogical integrators likely require training on more advanced AI tools (Avalos, 2011; Garet et al., 2001). It also functions as an assessment tool, providing ongoing evaluation and actionable feedback (Kimmons et al., 2020; Voogt et al., 2011). Furthermore, the findings highlight the need for curriculum design that integrates content knowledge and holistic AI integration into technology courses—thereby avoiding a narrow technical focus (Bobula, 2024; Ng et al., 2021). The instrument’s reliance on authentic artifacts also makes it highly adaptable; researchers can easily modify the specific criteria to align with different national curriculum standards or institutional contexts without compromising the integrity of the overall AI-TPACK framework.

5.2. Study Limitations and Future Research

This study has several limitations that should be acknowledged. First, the sample, drawn from teacher educators who completed an extensive training course, may not be representative of the broader population. Their advanced outputs may reflect a more ideal state of AI-TPACK proficiency. Moreover, our reliance on written outputs alone may not fully capture the complexity of real-world classroom implementation, a limitation noted in previous artifact-based studies (e.g., Rosenberg & Koehler, 2015; Willermark, 2018). Future research should include a larger, more diverse sample from a range of educational backgrounds. Additionally, incorporating methods like classroom observations and conducting longitudinal studies could provide a more comprehensive understanding of the construction over time.

6. Conclusions

This study systematically addressed its three primary research objectives. First, by defining the AI-TPACK components that emerged from analyzing authentic pedagogical artifacts (RQ1), it provided a refined conceptual extension of the TPACK framework. Second, through rigorous expert validation and empirical analysis, the research confirmed the tool’s strong validity and reliability as a robust measurement instrument (RQ2). Finally, the study identified four distinct teacher educator competency patterns (RQ3) that offer actionable insights for guiding targeted professional development.

The current research makes key contributions to the TPACK field by developing and validating an initial tool for measuring AI-TPACK proficiency among teacher educators. The instrument, found to be both valid (CVR = 0.86, CVI = 0.91) and reliable (ICC = 0.84), effectively differentiates between knowledge components and proficiency levels. It offers a practical resource for future research, as well as for teacher-training planners and policymakers, supporting the diagnosis and development of key AI competencies in the emerging AI era. The analysis also advances the integration of AI-specific dimensions—AIK, AIPK, and AICK—into the existing framework and provides empirical evidence for the value of artifact-based assessment over subjective self-reports.

These findings highlight that successful AI integration is heavily driven by pedagogical understanding (AIPK) rather than pure technical knowledge (AIK), emphasizing the need for professional development to foster synergy between domains.

The demonstrated contributions provide a strong foundation for future research and for designing professional development that promotes both effective and responsible AI integration in education. However, the findings should be interpreted cautiously due to limitations related to the advanced sample composition (teacher educators in training) and the reliance on written artifacts alone, underscoring the necessity for future longitudinal and diverse cross-cultural validations to capture real-world classroom practices.