Evidence of Validity for the Artificial Intelligence Competence and Literacy Test (CAIA) in Spanish University Students

Abstract

This study presents the development and psychometric validation of the Artificial Intelligence Competence and Literacy Test (CAIA), designed to assess artificial intelligence (AI) literacy and competence in higher education contexts. As AI becomes increasingly integrated into academic and professional environments, reliable instruments are needed to evaluate individuals’ conceptual understanding, ethical awareness, and applied competencies related to AI. The instrument was administered to a sample of 510 university students from several faculties at a Spanish university. Exploratory and confirmatory factor analyses were conducted using a cross-validation design. Results supported a multidimensional structure, in the final 18-item version of the instrument, composed of two correlated factors (critical—conceptual AI literacy and creative—applied AI competence) and a second-order hierarchical model representing a global CAIA score. Model fit indices were acceptable-to-good, and reliability estimates, including ordinal coefficients and measurement error indicators, showed adequate precision for both individual and group-level interpretation. Evidence of construct validity was further supported through convergent and discriminant analyses, as well as hypothesis testing across academic subgroups. The findings suggest that the CAIA provides a theoretically grounded and psychometrically robust instrument for assessing AI-related competencies in higher education. The instrument may support research, curriculum design, and evaluation of educational initiatives aimed at promoting informed, critical, and responsible engagement with artificial intelligence in digitally mediated learning environments.

1. Introduction

Since the emergence of generative artificial intelligence tools such as ChatGPT, Generative Artificial Intelligences (GAIs) have become exponentially popular, rapidly being implemented across professional, educational, and social domains. This swift adoption has created a growing need for users not only to be familiar with these technologies but also to be adequately literate in their use or to possess advanced competencies that allow them to fully leverage their potential.

However, the speed of technological advancement has outpaced the theoretical and educational development surrounding AI. Consequently, the concepts of Artificial Intelligence Literacy (AIL) and Artificial Intelligence Competence (AIC) must be rigorously defined, characterized, differentiated, and measured. This distinction has not only epistemological implications but also practical ones, as it is essential for the design of measurement instruments, training programs, impact evaluation, and the identification of gaps in the responsible and ethical use of AI.

Despite recent efforts, significant conceptual confusion persists in the scientific literature. Various authors have proposed definitions of AIL [1,2], while others have focused on AIC, such as [3] or [4]. However, many of these works use the terms literacy and competence interchangeably, failing to acknowledge that they are theoretically related yet distinct constructs. This ambiguity has led to measurement instruments that combine indicators of both concepts, thereby compromising their structural validity and, consequently, their applicability.

The present study has two main objectives. First, to propose a clear, comprehensive, and differentiated conceptualization of AIL and AIC. Second, based on this theoretical clarity, to examine the psychometric properties of an instrument specifically designed to measure both constructs (competence and literacy in AI) among university students: the Artificial Intelligence Competence and Literacy Test (CAIA).

The theoretical framework presented below is organized into three main sections. First, a conceptual framework on literacy and competence is developed, addressing their definitions, similarities, and differences. Second, the relationship between AIL and AIC is examined, and, finally, an analysis is offered of the psychometric quality of several recently proposed instruments for measuring artificial intelligence literacy and/or competence.

Theoretical Framework

Conceptual foundations of literacy and competence. Literacy refers to individuals’ ability to access, interpret, evaluate, and communicate information in an ethical and contextually grounded manner [5,6]. Beyond technical skills, literacy also involves understanding conventions and meanings, applying basic strategies for information processing, and developing critical and responsible attitudes toward information practices [7,8,9,10]. Contemporary perspectives increasingly conceptualize literacy as a context-dependent and sociocultural construct rather than as a standardized set of isolated skills [7,8,9,10].

Literacy is commonly described as comprising three interrelated dimensions: cognitive (knowledge and understanding), procedural (basic strategies for searching and processing information), and dispositional (attitudes, values, and critical positioning) [5,6]. Consequently, literacy is typically inferred through declarative and interpretative evidence, such as comprehension tasks, critical analyses, rubrics, and self-report measures triangulated with actual performance [6].

Competence, in contrast, refers to the integrated ability to mobilize knowledge, skills, attitudes, and values to address tasks effectively and responsibly in specific contexts [11,12]. Unlike literacy, competence emphasizes situated action and observable performance, requiring individuals to apply knowledge under realistic conditions and make context-sensitive decisions [13,14,15]. Competence is therefore usually evidenced through simulations, portfolios, case resolution, and performance assessments [11,16].

Although literacy and competence differ in their degree of contextualization and the type of evidence required, they share important characteristics. Both involve the organization of knowledge into meaningful structures, self-regulatory processes, and progressive development through practice and experience [14,15,16]. Conceptually, literacy may be considered a foundation for more complex competencies, facilitating the transition from understanding and critical reflection toward increasingly autonomous and situated performance.

AI Literacy and AI competence. Artificial Intelligence Literacy (AIL) refers to the knowledge, understandings, and critical dispositions that enable individuals to describe what AI is, how it works at a basic level, where it is used, and what limitations, risks, and ethical implications it entails [1,17,18]. Operationally, AIL includes conceptual knowledge about data, models, machine learning, training and inference; basic procedural understanding of AI workflows; socio-technical and ethical awareness regarding bias, fairness, transparency, privacy, copyright, and responsible use; and dispositions of informed scepticism and public responsibility toward AI-generated content [1,17,18].

Recent frameworks organize AIL around understanding, evaluating, and using AI responsibly, without necessarily requiring evidence of situated performance or solution design [19,20,21]. Accordingly, AIL is usually assessed through declarative and interpretative evidence, such as items on AI concepts and risks, vignettes requiring critical judgment, analyses of AI-related information, and self-reports focused on beliefs, knowledge, and responsible use [19,20,22].

By contrast, Artificial Intelligence Competence (AIC) involves the integrated mobilization of knowledge, skills, judgment, and values to use, design, evaluate, and manage AI systems in real or simulated contexts according to criteria of effectiveness, safety, ethics, and transferability [23]. AIC therefore includes applied technical foundations, situated decision-making, socio-technical integration, and management or continuous improvement processes, such as selecting tools, preparing data, evaluating performance, balancing accuracy, cost, fairness, and traceability, documenting decisions, and monitoring impact [23].

Educational competence frameworks further emphasize human-centeredness, ethics, foundations and applications, pedagogy, and professional development as dimensions that guide learning outcomes and quality criteria for responsible AI use [5]. Unlike AIL, evidence of AIC requires authentic or simulated performance, such as case resolution, projects, simulations, functional indicators, or peer/user review assessed with rubrics that consider solution adequacy, risk management, documentation, and transparency [5].

In summary, AIL focuses primarily on knowing, understanding, and critically reflecting on AI, whereas AIC emphasizes knowing how to act with AI in situated contexts. This distinction requires differentiated measurement strategies: AIL is better captured through declarative and interpretative items assessing conceptual knowledge and critical judgment [18,19,20], whereas AIC requires situated items or tasks involving justified decisions, applied evaluation, creation, and responsible action under realistic constraints [5,23]. In terms of cognitive alignment, AIL is closer to remembering, understanding, and reflective evaluation, while AIC is more closely related to applying, analysing, evaluating in context, and creating, although the action verb alone does not determine the construct being assessed [16,24].

Existing AI Literacy and Competence Instruments. During the last decade, several instruments have been developed to assess AIL, whereas comparatively fewer tools have attempted to evaluate AIC, particularly through performance-oriented approaches. A recent systematic review summarized in

Table 1. Recent AI Literacy/AI Competence Instruments: Psychometric Synthesis and Limitations.

Instrument	Population	Focus	Main Strengths	Main Limitations
AILQ (AI Literacy Questionnaire) [26]	Students	AIL 4 domains	Good factorial structure and internal consistency; multidimensional design	Limited criterion validity and lack of measurement invariance evidence
SNAIL (Scale for Non-Experts’ AI Literacy) [27]	General public and non-experts	AIL	Initial factorial validity; non-expert focus	No invariance testing; limited cross-population replication
Meta AI Literacy Scale (additional validation) [28]	Various	AIL	Additional validation evidence; broader psychometric support	No test–retest evidence; limited responsiveness data
AILS (traducción árabe) [29]	Arabic University Students	AIL	Good fit, reliability, and gender invariance	Limited broader validity evidence
ALTL/AL Frameworks in HE [19]	HE instructors (PST, POST, PAS)	Conceptual framework	Clear conceptual framework for AI literacy	No psychometric validation
AI Literacy Framework (Digital Promise) [20,21]	K–HE	AIL	Clear conceptual framework for AI literacy	No psychometric validation
UNESCO AI Competency [17]	Teachers Students	AIC and AIL	Clear conceptual framework for AIL and AIC	No psychometric validation
AILIT (short AI literacy test) [30]	Students	AIL	Brief format; initial validity evidence	No replication, invariance, or test–retest evidence

Source: own elaboration based on [25].

identified 16 AI-related scales and highlighted substantial variability in conceptual definitions, domains assessed, and psychometric evidence across instruments [25].

Despite recent progress, several limitations remain across existing instruments. First, some studies combine self-report and performance indicators, complicating construct interpretation and score use [25]. Second, AIL and AIC are operationalized heterogeneously, leading to inconsistencies in domains and taxonomies that hinder comparability across studies [19,21]. Third, psychometric evidence often relies on basic indicators of validity and reliability, whereas more advanced evidence regarding content validity, measurement invariance, external validity, measurement error, and responsiveness remains scarce [25,26,27,28,29]. In particular, Cronbach’s α continues to predominate, while ordinal reliability coefficients, test–retest evidence, and sensitivity-to-change analyses are rarely reported [25]. These limitations highlight the need for instruments that clearly distinguish AIL from AIC while providing stronger psychometric evidence and more comprehensive validation procedures. The CAIA was developed to address these gaps.

Accordingly, the present study aimed: (a) to gather evidence of structural validity for the CAIA test through exploratory and confirmatory factor analyses, (b) to examine the reliability of the instrument using comprehensive indicators of internal consistency and ordinal reliability, and (c) to determine whether score differences exist across population subgroups (sex, faculty, and year of study) through hypothesis testing.

In concordance with the previously defined objectives, the present research has the following hypothesis:

H1. 

The CAIA will show a multidimensional structure composed of two correlated latent factors representing AI literacy (AIL) and AI competence (AIC).

H2. 

A second-order hierarchical model representing a general CAIA factor underlying the AIL and AIC dimensions will provide an adequate representation of the construct and support the interpretation of a global score.

H3. 

The CAIA scores will show adequate reliability estimates, including internal consistency coefficients and ordinal reliability indices, supporting their use for research and educational assessment purposes.

H4. 

Differences in CAIA scores are expected across academic faculties, reflecting variability in exposure to AI-related knowledge and practices, whereas no substantial differences are expected according to sex or year of study.

Item generation was guided by the conceptual distinction between AI literacy (AIL) and AI competence (AIC) identified in recent theoretical and educational frameworks. Items were developed to represent two theoretically differentiated domains: critical—conceptual understanding of AI (AIL) and creative—applied use of AI (AIC). This distinction formed the basis for Hypothesis 1, which anticipated a multidimensional structure rather than a unidimensional solution.

2. Materials and Methods

Design. The present study follows an instrumental research design, aimed at the development and administration of a measurement instrument and the evaluation of its psychometric properties. Instrumental studies are analytical designs focused on the construction and assessment of measurement tools and typically rely on multivariate statistical techniques.

Participants. The sample consisted of 510 university students from the Complutense University of Madrid, enrolled in the following faculties: Physical Sciences (n = 75), Mathematical Sciences (n = 82), Chemical Sciences (n = 77), Information Sciences (n = 91), Computer Science (n = 63), Biological Sciences (n = 77) and Medicine (n = 45). Regarding sex, the sample included 272 men (53.3%) and 238 women (46.7%). With respect to the academic year: first year (n = 97), second year (n = 110), third year (n = 140), and fourth year (n = 163).

Instrument. The present study used the CAIA test, which consists of 78 Likert-type items with five response options: Always, Almost Always, Sometimes, Almost Never, and Never. The test includes 9 items for each dimension of Bloom’s taxonomy [30], revised by [24]: remember, understand, apply, analyse, evaluate, and create. It also includes 12 ethics items and 12 awareness items. The instrument was designed to be administered to a population of young people aged 15 to 25 years.

The levels of Bloom’s taxonomy are ordered by complexity; therefore, the lower levels assess literacy, whereas the higher levels relate to the measurement of competencies. Below, each level is defined and accompanied by an example item corresponding to that dimension:

-

Remember. At this level, students recall previously learned information without requiring deep understanding. It involves retrieving facts, terms, basic concepts, and answers. Example: I remember chatbots that appear in instant messaging applications.

-

Understand. This involves comprehending information and being able to explain, interpret, or summarize ideas and concepts. Example: I understand how recommendation systems use AI techniques to suggest personalized content.

-

Apply. At this level, students can apply acquired knowledge to new situations or contexts. Example: I apply AI-based facial recognition techniques to tag friends in photos on social media.

-

Analyse. This involves breaking down information into its components or parts to better understand its structure and relationships. Example: I analyse common errors made by AI-based machine translation systems and propose possible improvements to address them.

-

Evaluate. At this level, students can make judgments based on specific criteria and standards. Example: I evaluate the ability of a virtual assistant to understand and respond appropriately to complex and contextual requests.

-

Create. The highest level of the taxonomy, which involves creating new knowledge or synthesizing ideas to generate original solutions. Example: I develop an AI-based strategy game that can adapt to and learn from player behaviour.

Additionally, ethical and awareness competencies were included. The former comprises items related to the fair, responsible, transparent, and beneficial use of AI, whereas the latter assesses awareness of AI’s presence in daily life, its influence on personal actions and decisions, and AI’s consequences and limitations. An example item for the ethics dimension is: “I avoid using AI for fraudulent or deceptive purposes”; while an example item for the awareness is: “I reflect on the effects of AI on the creation and dissemination of fake news”.

The full process of item construction and writing, the theoretical foundations, and the evidence for content validity can be found in [31].

The test was administered in various university settings (cafeterias, classrooms, etc.) with the support of the researchers and collaborating faculty members. Data were collected anonymously and with the participants’ prior informed consent. The sample was then randomly divided into two halves of equivalent size: one for the Exploratory Factor Analysis (EFA; n = 255) and the other for the Confirmatory Factor Analysis (CFA; n = 255), following a cross-validation scheme recommended in the psychometric literature [32,33].

2.1. Data Analysis

The analysis was conducted in four complementary phases:

Phase 1. Exploratory Factor Analysis (EFA). Given the polytomous nature of the items, a polychoric correlation matrix was used as the basis for the EFA. This procedure is more appropriate for ordinal variables [34,35]. The suitability of the data matrix for factor analysis was assessed using Bartlett’s test of sphericity and the Kaiser–Meyer–Olkin (KMO) index, both global and item-level. A KMO value above 0.80 is considered adequate and supports the appropriateness of the analysis [36,37]. Factor extraction was carried out using the minimum residual method (MINRES; [38]), recommended for its computational efficiency and stability in the estimation of communalities, particularly when multivariate normality cannot be assumed, as is the case [39].

The number of retained factors was determined based on Horn’s parallel analysis and the theoretical interpretability of the model. The factor rotation employed was simplimax, an oblique rotation that allows correlations among factors, as the literacy and competence components are assumed to be interrelated [40,41].

This exploratory phase aimed to identify and retain items according to predefined psychometric and theoretical criteria, and establish a solid empirical basis for subsequent confirmatory analyses. The combination of statistical and theoretical criteria ensured that the retained dimensions were not only statistically coherent but also interpretatively meaningful in relation to the conceptualization of AIL and AIC.

Phase 2. Confirmatory Factor Analysis (CFA. Several CFA models were estimated to empirically test the two-factor structure identified in the exploratory phase and to evaluate possible theoretical alternatives: a unidimensional model, a two-correlated-factors model, a second-order model, and a bifactor model. This comparative approach directly supports the validity of the scores and informs their appropriate use, as it allows controlling for bias and delimiting interpretable scores [33,40]; and gathering convergent and discriminant validity evidence through the analysis of standardized loadings and their R² values, as well as the Average Variance Extracted (AVE): an AVE above 0.50 indicates that the factor explains more true variance than error. Discriminant validity was also assessed using the Fornell–Larcker criterion, verifying that √AVE for each factor exceeded the interfactor correlations, and through latent-variable indices such as HTMT, which should remain below 0.85 [40,41].

The Akaike Information Criterion (AIC), Bayesian Information Criterion (BIC), and the sample-size adjusted BIC (SABIC) were calculated to evaluate the relative parsimony of the four models, with lower values indicating preferable models [42]. This indicator allows examining the parsimony of the models by comparing information criteria, favouring general structures, and avoiding capitalization on chance, thereby strengthening the interpretation of the scores [41].

Model estimation was performed using the Robust Maximum Likelihood (MLR) method, which adjusts significance tests and standard errors through the Huber–White corrections and the Yuan–Bentler scaled test (T₂*). This estimator provides more stable and accurate results in the presence of skewness, kurtosis, or heteroscedasticity, as is the case [35,43]. MLR requires moderate sample sizes and permits the simultaneous modeling of covariances among factors, making it particularly suitable when items have five or more categories and distributions are not extremely skewed [44]. Moreover, even with the ordinal nature of Likert items, robust estimators are recommended when categories ≥ 5.

The quality of model fit was evaluated using a set of complementary global and incremental indices. Among the absolute fit indices, the Satorra–Bentler scaled χ² statistic was considered. For approximate fit, the Root Mean Square Error of Approximation (RMSEA) was used, along with its 90% confidence interval and the probability of close fit (p_close); values below 0.06 indicate satisfactory fit [45]. The Standardized Root Mean Square Residual (SRMR) was also examined, with values below 0.08 recommended for acceptable fit [33].

Among the incremental fit indices, the Comparative Fit Index (CFI) and the Tucker–Lewis Index (TLI) were examined. Both indices compare the specified model with a null model that assumes no correlations among variables. Values of CFI and TLI above 0.95 are interpreted as evidence of excellent fit [40,45].

Phase 3. Reliability and Measurement Error. To assess reliability, the full sample of participants was used. Reliability was evaluated comprehensively through multiple coefficients capturing different aspects of internal consistency for each subscale and the total test score. Cronbach’s α, McDonald’s total ω, and the hierarchical ω_H coefficient were estimated, along with their ordinal versions (α_ord, ω_ord, ω_H,ord), which are based on polychoric correlation matrices and are more appropriate for Likert-type items [46]. Ninety-five percent confidence intervals were computed using bootstrap resampling with 1500 replicates, providing a robust estimate of the precision of each coefficient [47].

Additionally, indicators related to score precision and measurement error were estimated to quantify the magnitude of measurement error and the instrument’s sensitivity to detecting real differences. First, the Standard Error of Measurement (SEM) was calculated using the expression SEM = SD × √(1 − ρ), where SD is the standard deviation of the scale and ρ is the reliability coefficient considered.

Second, the Minimum Detectable Change at 95% confidence (MDC₉₅) was computed. At the individual level, this was obtained as MDC_95,ind = SEM × 1.96 × √2. This indicator represents the amount of change required to consider a difference between two measurements or groups as real rather than attributable to measurement error [48]. For the group level, the error was adjusted by the effective sample size of each subgroup using the formula MDC_95,group = MDC₉₅/√n, allowing for the comparison of the relative stability of means between two independent groups [49].

This comprehensive approach combines classical and ordinal reliability estimation with measures of error and sensitivity, allowing for a more rigorous interpretation of the extent to which the CAIA can distinguish between AIL and AIC among individuals and groups in a precise and replicable manner.

Phase 4. Hypotheses Testing. These analyses were made using the full sample and aimed to examine potential differences in CAIA scores according to sex, faculty (seven groups), and academic year (four groups). Group comparisons were conducted using robust statistical methods to account for potential deviations from normality and heterogeneous variances. First, Yuen’s robust t-test, based on 20% trimmed means, was applied to compare mean scores between men and women for each subscale and for the total CAIA score. This approach provides a more stable alternative to the classical t-test, particularly in asymmetric distributions or in the presence of outliers [50]. Second, comparisons across faculties and academic years were conducted using a Welch-type robust ANOVA and Games–Howell post hoc tests. This procedure is appropriate when variances are unequal or when group sizes differ [51].

2.2. Software

All analyses were conducted in the R statistical environment, version 4.4.1 [52]. Data management and processing followed the recommendations of [53] for reproducible psychometric analyses in R, prioritizing transparency and traceability for each procedure. For the EFA, the packages psych (version 2.4.6) [54] and MBESS (version 4.9.3) [55] were used. The CFA was carried out using the lavaan package (version 0.6-18) [56]. For the reliability and measurement error analyses, the ltm [57], ufs, and psych packages were used to compute the α, ω, and ω_H coefficients and their ordinal versions, along with bootstrap confidence intervals. Custom routines were also developed to estimate SEM and tMDC₉₅ at both the individual and group levels, following the formulas proposed by Stratford and Goldsmith [58].

3. Results

This section presents the empirical findings following the same analytical sequence described in the Materials and Methods section.

3.1. Exploratory Factor Analysis

The EFA was carried out in two successive stages: a first stage using the 78 original items, and a second stage using 34 items selected based on empirical and theoretical criteria. The aim was to compare the parsimony, interpretability, and overall fit of both solutions, thereby justifying the retention of the abbreviated version as the basis for the subsequent confirmatory factor analysis. The item reduction strategy followed both statistical and theoretical criteria. In the first stage, corresponding to the EFA of the 78 items, Bartlett’s test of sphericity was significant (χ² = 8975, df = 3003, p < 0.001), and the KMO index reached a high value (KMO = 0.865), indicating excellent sampling adequacy and supporting the suitability of factorization. The extraction yielded a five-factor solution with a relatively low cumulative variance (33.9%). The interfactor correlations were practically zero, suggesting independence between dimensions.

This pattern is theoretically implausible, as the components of AIL and AIC would be expected to organize along a cognitive continuum, rather than as entirely independent domains. Furthermore, the factor loadings showed notable dispersion, with several modest loadings and high uniqueness values for a considerable number of items, indicating a heterogeneous structure and the presence of measurement noise derived from the initial length and diversity of the instrument.

The global fit indices for this 78-item solution were moderate (see

Table 2. Comparison of global fit indices of EFA models.

Index	78 Items	34 Items
${\chi }^2$ (gl), $p$	3604 (2623), $<0.001$	779 (494), $<0.001$
RMSEA [IC 90%]	0.038 [0.035, 0.041]	0.047 [0.041, 0.054]
TLI	0.808	0.863
BIC	$-\mathrm{10,931}$	$-1958$
Nº of factors (% variance)	5 (33.9%)	2 (31.8%)

). Although the RMSEA value appears low, the TLI, falling below the 0.90 threshold, suggests that the five-factor model does not adequately reproduce the covariance structure among the items. Taken together, this first solution is characterized by weak factorial coherence and limited parsimony, which supports the need for item reduction and selection procedures.

In the second stage, corresponding to an EFA with the 34 items that showed the highest factor loadings, the same sample and estimation criteria were retained. Bartlett’s test remained significant (χ² = 2938, df = 561, p < 0.001), while the global KMO reached a high value (0.874), again confirming the suitability of the data for factor analysis.

The emerging factorial structure was two-dimensional and interpretable, with substantive and differentiated loadings on both factors. Factor 1 comprised items associated with the Create and Apply dimensions, along with an Ethical–operational component linked to practical AI competence; Factor 2 grouped items related to Awareness, Understand, Remember, and Evaluate, representing the domain of literacy and critical understanding.

The interfactor correlation was low (r = 0.184), indicating that the two factors are related yet clearly distinguishable, consistent with the theoretical distinction between competence and literacy in AI. The cumulative variance reached 31.8%, a value expected in psychological instruments, but with a cleaner and more parsimonious structure than in the initial version.

Regarding the global fit indices (Table 2), the 34-item solution showed a substantial improvement: although the TLI remains slightly below the optimal criterion (0.90), the improvement compared to the previous analysis is clear, and the overall set of indicators supports the adequacy of the two-correlated-factors structure. Moreover, the conceptual coherence between the item groupings and the underlying theoretical framework of the CAIA reinforces the instrument’s internal validity.

In summary, from both a psychometric and substantive standpoint, the 34-item version outperforms the 78-item version. First, it presents a cleaner simple structure, with higher and more consistent factor loadings within each factor and reduced cross-loading. Second, the interfactor correlation, although low, is theoretically coherent, with the model distinguishing between AI literacy and AI competence, as well as with the notion of a cognitive continuum grounded in Bloom’s taxonomy. Third, it offers a markedly improved global fit (TLI 0.863 versus 0.808) and maintains excellent sampling adequacy (high KMO and significant sphericity). Finally, it provides a more parsimonious and theoretically solid basis for subsequent confirmatory validation.

3.2. Confirmatory Factor Analysis

Four confirmatory specifications were estimated: (a) a unidimensional model, (b) a two-correlated-factors model, (c) a second-order hierarchical model in which CAIA explains both first-order factors, and (d) a bifactor model (a general factor plus two orthogonal specific factors). All estimations were conducted using robust maximum likelihood, Huber–White robust standard errors, and Yuan–Bentler scaled corrections, given evidence of multivariate non-normality (Mardia: skewness χ²(1140) = 1619, kurtosis z = 8.37, p < 0.001).

Table 3. Comparison of CFA Models and Global Fit Indices.

Model	χ2	df	p	CFI	TLI	RMSEA [90% CI]	SRMR	AIC	BIC	SABIC
1	425	135	<0.001	0.743	0.709	0.092 [0.083–0.101]	0.114	14,276.27	14,467.50	14,296.30
2	140	134	0.339	0.995	0.994	0.013 [0.000–0.033]	0.049	13,953.86	14,148.63	13,974.26
3	139	133	0.340	0.995	0.994	0.013 [0.000–0.033]	0.049	13,955.86	14,154.17	13,976.64
4	112	117	0.601	1.000	1.005	0.000 [0.000–0.028]	0.037	13,946.67	14,201.64	13,973.39

Note. Model: 1. Unidimensional. 2. Two correlated factors. 3. Second-order hierarchical. 4. Bifactor. T Robust/scaled fit indices based on Huber–White standard errors and Yuan–Bentler correction were reported. Lower AIC, BIC, and SABIC values indicate better relative model fit.

summarizes fit indices and information criteria for all tested CFA models. The unidimensional solution showed clearly inadequate fit. In contrast, the correlated two-factor, second-order, and bifactor models demonstrated excellent global fit. Although the bifactor model showed the best statistical fit, several substantive and interpretative problems limited its usefulness. Considering both fit and theoretical interpretability, the correlated and second-order solutions provided the most adequate representation of the CAIA structure.

Factor loadings and validity metrics for the selected model (second-order model, n = 255) are presented in

Table 4. Standardized loadings ($\beta$), R² by item and factor and validity metrics.

Indicator/Factor	$\mathsf{\beta }$	IC 95%	${\mathbf{R}}^2$
Factor 1: AIC (Creative—Applied Competence)
I develop an automatic music generator that uses neural networks to compose original pieces in different genres and styles.	0.737	[0.662, 0.811]	0.542
I create an AI system capable of generating visual art using generative neural networks.	0.720	[0.653, 0.787]	0.518
I develop an AI-based strategy game that can adapt to and learn from the player’s behavior.	0.704	[0.623, 0.784]	0.495
I create a simulation model of social interactions on social media to help users practice communication and empathy skills.	0.691	[0.616, 0.765]	0.477
I use AI-powered chatbots to automate responses on social media.	0.695	[0.618, 0.773]	0.484
I create a natural language generation model that enables a virtual assistant to communicate more naturally and fluently with users.	0.701	[0.626, 0.776]	0.491
I create a machine translation system based on deep learning that dynamically adapts to different linguistic and cultural contexts.	0.654	[0.570, 0.739]	0.428
I develop an AI model that can predict the user’s intentions and anticipate their needs to provide proactive responses.	0.597	[0.502, 0.693]	0.357
I use AI to develop innovative solutions that address global challenges such as poverty and inequality.	0.562	[0.462, 0.662]	0.316
I create an automatic text generation model to help users write more persuasive and engaging social media posts.	0.574	[0.465, 0.682]	0.329
Factor 2: AIL (Critical—Conceptual Literacy)
I recognize the security risks associated with the use of AI systems.	0.606	[0.504, 0.708]	0.367
I understand the role of AI in detecting and removing inappropriate content on social media platforms.	0.646	[0.552, 0.740]	0.417
I understand how AI-based recommendation systems suggest relevant content in my social media feeds.	0.565	[0.451, 0.679]	0.319
I remember that there are recommendation algorithms on mobile gaming platforms that suggest new games based on the player’s preferences.	0.593	[0.493, 0.693]	0.352
I evaluate the accuracy and coherence of summaries automatically generated by AI tools.	0.494	[0.390, 0.598]	0.244
I consider how AI can influence the quality of the education I receive.	0.502	[0.383, 0.621]	0.252
I remember that speech recognition algorithms are applied in personal assistant applications.	0.500	[0.387, 0.614]	0.250
I remember that there are AI algorithms for detecting and filtering fake news or misleading content on social media.	0.513	[0.411, 0.615]	0.263
Second-order structure
$\mathrm{CAIA} \to$ AIC	0.700	[0.597, 0.803]	0.491
$\mathrm{CAIA} \to$ AIL	0.334	[0.144, 0.525]	0.112
Convergent validity
AVE AIC	0.444
AVE AIL	0.306

. The two-correlated-factors model shows good item-level convergent validity, with moderate–high loadings and acceptable R² values, although AVE does not reach the 0.50 threshold. In the second-order model, the general factor explains the applied performance factor more strongly than the literacy factor, indicating moderate convergent validity for the global score. Both models provide solid evidence of discriminant validity. In the two-factor model, the interfactor correlation is low, HTMT is well below the cut-off, and the shared variance is lower than each factor’s AVE. The second-order model shows a similar pattern, with literacy retaining specific variance not captured by the general factor. Overall, the two domains are related but clearly distinct.

3.3. Item Reduction Procedure

Item reduction followed both statistical and theoretical criteria. In the first stage (78 to 34 items), items showing low factor loadings, substantial cross-loadings, high uniqueness values, and conceptual redundancy were removed. Subsequently, the reduction from 34 to the final 18 items was guided by CFA modification indices, item redundancy analyses, and theoretical considerations aimed at preserving balanced representation of AIL and AIC domains while improving model parsimony and interpretability (

Table 5. Item selection process.

Phase	Initial Items	Retained Items	Statistical Criteria	Theoretical Criteria
Initial pool	78	78	Expert review and content validation	Coverage of AIL and AIC domains
EFA refinement	78	34	Low loadings, cross-loadings, high uniqueness, redundancy	Conceptual clarity and representativeness
CFA refinement	34	18	Modification indices, redundancy, model fit improvement	Balanced representation of dimensions and interpretability

Note. Items were retained based on both statistical and theoretical considerations to preserve content coverage and improve model parsimony. AIL = Artificial Intelligence Literacy; AIC = Artificial Intelligence Competence.

).

Item response distributions were additionally examined to identify potential ceiling effects. Ceiling response frequencies were generally low, with most items remaining below 20%. Only a small number of items approached values around 25%, suggesting limited concentration in the highest response category.

3.4. Final Composition of CAIA

Factor 1. Creative—Applied Competence in Artificial Intelligence (AIC, 10 items). Refers to an integrated ability to mobilize knowledge, skills, and attitudes in real or simulated contexts to design, implement, adapt, and evaluate AI-based solutions responsibly. It is a procedural, strategic, and situated competence, involving informed decision-making, uncertainty management, transfer to new domains, and the incorporation of ethical and social-impact criteria [6,14,16,59]. From an educational perspective, AIC builds on AI literacy and extends it through higher-order cognitive performance like apply, analyse, and create [24]. It includes designing workflows with models, preparing and refining data, evaluating and improving performance, integrating AI services, and ensuring safety and fairness. AIC is shown when individuals design or adapt models and prompts, integrate AI systems into applications, automate processes, assess and mitigate risks, communicate technical and ethical decisions, and transfer solutions across domains with effectiveness and responsibility [1].

Factor 2. Critical—Conceptual Literacy in Artificial Intelligence (AIL, 8 items). Refers to declarative knowledge, conceptual understanding, and attitudinal dispositions that enable individuals to understand what AI is, how it works, and where it is applied, as well as to recognize its limits, risks, and socio-ethical implications. It emphasizes informed understanding and critical judgment, supporting the ability to interpret claims, identify biases, and establish responsible-use criteria [1,22,26].

Aligned with the cognitive levels of remembering, understanding, and critical evaluation [24], AIL includes knowledge of key concepts (e.g., recommendation algorithms, voice recognition, classification systems) and awareness of ethical considerations such as privacy, fairness, and social impact. It is demonstrated when individuals recognize and describe AI systems in everyday contexts and critically analyse their implications and risks, without requiring technical or situated action.

General factor. The global construct Competence and Literacy in Artificial Intelligence (CAIA) integrates two related yet distinct domains (AIC and AIL). Although often used interchangeably, these domains correspond to different forms of knowledge and performance and to different levels of cognitive and situational demand [1,17,22]. International policy frameworks similarly describe literacies and competences as complementary, progressive goals rather than equivalent ones [6,60]. CAIA reflects a comprehensive readiness to understand, think critically, apply, create, and act responsibly with AI, combining declarative knowledge and ethical dispositions with procedural and situated capabilities.

3.5. Reliability

Internal reliability was evaluated for the two subscales and for the composite CAIA score in the total sample (N = 510). For AIC, internal consistency was high (see

Table 6. Comparative summary of reliability indices.

Scale	$\mathit{\alpha }$	IC 95%	$\mathit{\omega }$	IC 95%	${\mathit{\omega }}_{\mathit{H}}$	GLB	H	${\mathit{\alpha }}_{\mathbf{ord}}$	${\mathit{\omega }}_{\mathbf{ord}}$
AIC	0.880	[0.864, 0.895]	0.880	[0.865, 0.896]	0.790	0.904	0.884	0.905 [0.893, 0.917]	0.905 [0.893, 0.918]
AIL	0.764	[0.733, 0.795]	0.765	[0.734, 0.796]	0.678	0.784	0.768	0.791 [0.764, 0.819]	0.792 [0.764, 0.819]
CAIA	0.833	[0.811, 0.854]	0.816	[0.792, 0.840]	0.637	0.853	0.885	0.852 [0.833, 0.871]	0.833 [0.812, 0.854]

). Inspection of the “alpha if item deleted” index did not indicate any substantive gains from item removal, suggesting that all 10 indicators contribute relatively homogeneously to the subscale. For AIL, internal consistency was adequate (Table 6). Although acceptable, the magnitude was lower than that observed for AIC, consistent with somewhat smaller loadings and greater item-specific variance. Deleting any indicator would minimally affect Cronbach’s α, providing no justification for item removal. For the global CAIA test score, reliability was good. Although overall consistency was sufficient, the proportion of variance attributable to a common general factor was lower than for the AIC factor. In summary, all scores show high and stable reliability across all estimators (including ordinal versions), suitable for group-level decisions and applied research. The total score reaches adequate levels of internal consistency; however, its ωH indicates that the common general variance is moderate, supporting the combined use of a global score and subscales, in line with the second-order hierarchical model [61,62].

3.6. Standard Errors of Measurement (SEM) and Minimum Detectable Change (MDC)

For AIC, SEM values ranged from 2.73 to 3.08, implying that individual changes below 7.6–8.5 points fall within measurement error. At the group level by sex, the MDC was very small, indicating high sensitivity for detecting mean differences. For AIL, SEM values ranged from 2.77 to 2.95, with individual MDC95 estimates of about 7.7–8.2 points. At the sex-group level, the MDC was around 0.48–0.51 points. This pattern reflects moderate/adequate reliability and lower score dispersion than in AIC (

Table 7. SEM and MDC95% by scale and reliability coefficient (total sample and sex-based groups).

Scale	Coefficient	SD	SEM	MDC95,ind	MDC95,group
AIC	$\alpha$	8.886	3.080	8.537	0.535
AIC	$\omega$	8.886	3.072	8.516	0.533
AIC	${\omega }_H$	8.886	3.073	8.518	0.533
AIC	${\alpha }_{\mathrm{ord}}$	8.886	2.738	7.588	0.475
AIC	${\omega }_{\mathrm{ord}}$	8.886	2.731	7.570	0.474
AIC	${\omega }_{H,\mathrm{ord}}$	8.886	2.732	7.573	0.474
AIL	$\alpha$	6.064	2.946	8.166	0.511
AIL	$\omega$	6.064	2.941	8.152	0.510
AIL	${\omega }_H$	6.064	2.941	8.151	0.510
AIL	${\alpha }_{\mathrm{ord}}$	6.064	2.770	7.678	0.481
AIL	${\omega }_{\mathrm{ord}}$	6.064	2.767	7.670	0.480
AIL	${\omega }_{H,\mathrm{ord}}$	6.064	2.767	7.670	0.480
CAIA	$\alpha$	11.582	4.735	13.124	0.822
CAIA	$\omega$	11.582	4.206	11.657	0.730
CAIA	${\omega }_H$	11.582	6.978	19.343	1.211
CAIA	${\alpha }_{\mathrm{ord}}$	11.582	4.457	12.355	0.774
CAIA	${\omega }_{\mathrm{ord}}$	11.582	3.835	10.630	0.666
CAIA	${\omega }_{H,\mathrm{ord}}$	11.582	6.800	18.848	1.180

). For the total score, SEM values ranged from 4.2 to 4.7 depending on the coefficient used, yielding individual MDC95 thresholds between 10.6 and 13.1 points. Ordinal estimates slightly improved precision, while hierarchical omega increased uncertainty due to the larger proportion of specific variance. At the sex-group level, MDC values were low (0.66–0.82), indicating good sensitivity to detect small mean differences between groups. In practical terms, individual changes smaller than 7.5–8.5 points in the subscales or 10.5–13 points in the total score may reflect measurement error, whereas sex-group differences above roughly 0.5–0.8 points can be interpreted as real.

3.7. Hypotheses Testing

Robust analyses showed no sex differences across AIL, AIC, or CAIA, with trivial effect sizes (

Table 8. Group comparisons across sex, faculty, and academic year.

Variable	Outcome	Statistical Test	Statistic	p	Effect Size [95% CI]	Interpretation
Sex	AIC	Yuen robust t-test	t(133) = 0.059	0.953	ξ = 0.018 [0.000, 0.205]	Trivial
Sex	AIL	Yuen robust t-test	t(152) = 0.041	0.967	ξ = 0.022 [0.000, 0.195]	Trivial
Sex	CAIA	Yuen robust t-test	t(135) = 0.278	0.782	ξ = 0.033 [0.000, 0.181]	Trivial
Faculty	AIC	Robust ANOVA	Q = 41.23	0.004	—	Significant
Faculty	AIL	Robust ANOVA	Q = 17.72	0.237	—	Non-significant
Faculty	CAIA	Robust ANOVA	Q = 28.36	0.030	—	Significant
Year	AIC	Robust ANOVA	F(3161) = 1.33	0.268	0.137 [0.049, 0.232]	Small
Year	AIL	Robust ANOVA	F(3159) = 1.99	0.117	0.164 [0.068, 0.290]	Small
Year	CAIA	Robust ANOVA	F(3157) = 0.452	0.716	0.114 [0.037, 0.210]	Small

Note. ξ = explanatory effect size for robust Yuen tests. Confidence intervals correspond to 95%. Robust analyses were based on trimmed means procedures.

). Faculty differences emerged for AIC and CAIA, although only one post hoc comparison reached statistical significance (

Table 9. Significant post hoc comparisons across faculties.

Outcome	Comparison	Mean Difference	p	Interpretation
AIC	Mathematics vs. Computer Science	−6.61	0.013	Significant difference favoring Computer Science

Note. Post hoc comparisons were conducted using family-wise confidence interval adjustment procedures. Only statistically significant pairwise contrasts are shown.

). No differences across academic year were observed, suggesting relative stability across training stages.

4. Discussion

The psychometric findings for the CAIA provide encouraging evidence of a clear and parsimonious latent structure: two correlated factors and a second-order hierarchical model with acceptable-to-good fit and theoretically interpretable multidimensional solutions. Overall, these findings suggest that the CAIA may extend aspects of the psychometric evidence commonly reported in recent AI literacy and competence instruments, which frequently emphasize internal consistency and confirmatory analyses but provide more limited information regarding ordinal-data estimation procedures, measurement invariance, and responsiveness analyses [25]. Literacy-focused instruments such as AILQ, SNAIL, the Meta AI Literacy Scale, and the Arabic version of AILS have reported good model fit and reliability; however, multi-group invariance testing, broader reliability indicators, and measurement-error or responsiveness indices remain less frequently examined. Moreover, many of these instruments focus primarily on declarative literacy rather than competence [25,27,59,63]. By contrast, the CAIA was designed to distinguish between critical—conceptual AI literacy (AIL) and creative—applied AI competence (AIC) while also modelling a global trait aligned with recent higher education frameworks requiring differentiated operationalisations [63,64].

From a construct validity perspective, confirmatory analyses supported the superiority of multidimensional models relative to the unidimensional alternative, with relatively homogeneous factor loadings, substantial R² values, and a moderate interfactor correlation suggesting differentiation between AIL and AIC. This pattern differs from previous studies in which the combination of declarative and performance indicators may complicate construct separation and structural interpretation [25]. Furthermore, the hierarchical modelling approach adopted for the CAIA may represent a useful alternative for integrating literacy and competence dimensions within a broader latent structure, although further replication is warranted [25,59].

Regarding convergent and discriminant validity, factor loadings and R² values provided initial support for convergent validity at the indicator level. However, AVE values did not reach conventional thresholds for AIL and remained moderate for AIC, suggesting that additional refinement of literacy indicators may improve shared variance. Nevertheless, discriminant validity was supported by low HTMT values, indicating adequate differentiation between AIL and AIC. Although these indices remain infrequently reported in previous validation studies using latent-variable approaches [25,27,59], taken together, these findings provide preliminary support for the proposed structural solution while also indicating areas for future refinement.

Reliability evidence extended beyond Cronbach’s α to include total ω, hierarchical ωH, and ordinal reliability estimates, accompanied by bootstrap confidence intervals and precision indicators intended to facilitate applied interpretation at both individual and group levels. This approach may broaden the type of psychometric information available relative to many previous instruments, which have often focused primarily on internal consistency estimates [25]. In addition, hypothesis-testing results showed no statistically significant differences according to gender or year of study, whereas differences across faculties emerged. These findings provide preliminary validity evidence that may be consistent with theoretical expectations regarding the contextual sensitivity of competence-related dimensions [27,63].

The present study contributes an instrumental design incorporating EFA and CFA with ordinal estimation procedures, explicit comparison of structural specifications, multiple reliability indices, and measurement precision indicators. In addition, the study attempts to clarify the conceptual and operational distinction between AI literacy and AI competence, an issue repeatedly identified as problematic in the recent literature [25,63,64]. The resulting instrument is brief and structurally interpretable and provides coherent global and subscale scores with quantitative interpretation guidelines that may support research and group-level applications in higher education contexts.

An additional contribution of the present study involves the evaluation of score precision beyond conventional internal consistency estimates frequently reported in AI literacy instruments. In addition to Cronbach’s α, the CAIA validation incorporated ordinal reliability coefficients and hierarchical omega (ωH), which may provide complementary information regarding the proportion of variance attributable to the general factor and may assist interpretation of global and subscale scores. Furthermore, measurement precision was examined through estimation of the Standard Error of Measurement (SEM) and the Minimum Detectable Change at the 95% confidence level (MDC95), providing interpretable thresholds that may help distinguish score differences from measurement error at individual and group levels. These indicators remain relatively uncommon in current AI literacy assessment studies, which frequently rely primarily on internal consistency estimates without incorporating broader indicators of measurement precision.

In addition, the comparison among correlated-factor, second-order, and bifactor specifications provided a detailed examination of the latent structure of the instrument. Although the bifactor model showed excellent global fit, the absence of a clearly defined general factor favored the interpretability and practical utility of the second-order hierarchical solution. Together with the low HTMT ratio observed between literacy and competence dimensions, these findings suggest structural differentiation between both domains while supporting the interpretation of an integrated higher-order construct. This modelling strategy may contribute to improving construct representation and score interpretability in the assessment of AI-related competencies in higher education contexts.

Despite these strengths, several limitations remain. AVE-based convergence for the literacy dimension should be improved through item revision and increased semantic homogeneity. Although the hierarchical CAIA model was stable, the general factor showed weaker loadings on AIL, suggesting the need to expand and refine the literacy item pool.

Exploratory measurement invariance analyses across sex and academic year were conducted; however, configural models showed inadequate fit and inadmissible latent covariance solutions in several subgroups, including interfactor correlations approaching or exceeding 1.0. Consequently, measurement invariance could not be established and subgroup comparisons should be interpreted cautiously. Multicentre replications and longitudinal or intervention studies are therefore needed to examine responsiveness and temporal stability (test–retest). As a Likert-type self-report instrument, the CAIA should be complemented with performance-based tasks and product-based assessments aligned with frameworks that emphasize behavioural evidence of competence [27,63,64]. In addition, some self-report items may be susceptible to socially desirable responding. Future studies should examine this possibility by incorporating specific response-bias measures.

In summary, the CAIA provides encouraging initial evidence regarding structural validity and reliability and may address several limitations identified in previous instruments through the explicit distinction between literacy and competence dimensions, the use of ordinal-data estimation procedures, and the incorporation of measurement precision indicators. Although additional evidence is needed regarding invariance, predictive validity, responsiveness, and broader generalizability, the instrument may provide a useful basis for research and educational assessment in higher education contexts. Taken together, these findings suggest that CAIA represents a promising instrument for future research examining AI literacy and AI competence while incorporating hierarchical score interpretation supported by ordinal reliability and measurement precision indicators.