An Analysis of Individual Differences in Within-Test Practice Effects in Progressive Matrices

Abstract

The present study aimed to investigate individual differences in practice effects during progressive matrices based on Carpenter et al.’s taxonomy of abstract rules. To this end, data from a non-verbal reasoning test, the Abstract Reasoning Test (ART), were used. Because the ART was developed from Carpenter et al.’s theory, the impact of extraneous factors unrelated to the theoretical model is minimized, thereby allowing for a more precise identification of practice effects. The sample consisted of 765 military recruits who responded to 34 items on the ART. Analyses were conducted using a random weights operation-specific learning model (RWOSLM), in which practice parameters were treated as random effects allowed to vary across individuals. The model measures within-test practice effects specific to each examinee, allowing the hypothesis of rule learning during the ART to be assessed at the individual level. Correlations between practice effects and external measures associated with intelligence were examined to investigate the nature of the practice effects. The results suggest individual differences in rule learning within the ART. Decreases in difficulty were observed for both pairwise progression and figure addition or subtraction, although between-person variability was evident only for the latter. Additionally, the results revealed between-person variability in decreases in difficulty associated with one of the items’ figural properties, which suggests the existence of individual differences in the process of increasing familiarity with this feature throughout the test. Individual differences in practice effects during the ART significantly correlated with external measures of abilities and intellect, suggesting that practice effects during progressive matrices are conceptually tied to intelligence.

1. Introduction

Identifying the cognitive processes underlying performance on intelligence tests is essential for developing a comprehensive understanding of the nature of intelligence. However, the inherent complexity of intelligence tests often makes this identification challenging, thereby perpetuating ongoing debates about the definition and scope of intelligence. To elucidate these processes, rigorous experimental and psychometric analyses of the most representative measures of intelligence are required.

One of the most influential and studied intelligence tests is the Raven’s Advanced Progressive Matrices (RAPM; J. C. Raven 1965). Originally developed by John C. Raven, the progressive matrices were designed to assess the ability to educe relations and correlates, aligning with Spearman’s (1927) definition of the general intelligence factor (g). The RAPM is a derivative of the original progressive matrices designed to spread the scores of the top 20% of the population (Raven and Raven 2003). The centrality of the RAPM among cognitive ability tests has led many experts to consider it the best measure of general intelligence (e.g., Jensen 1998; Snow et al. 1984). Each RAPM item consists of a 3 × 3 matrix of figural elements with the bottom right entry missing. The elements in the matrix follow a pattern based on abstract rules, so respondents must induce the rules in order to identify the element that fills the missing entry.

Based on the analysis of verbal protocols, eye fixation patterns, and errors, Carpenter et al. (1990) developed a processing theory of performance on the RAPM. The authors identified five analytic rules that govern the elements in the matrix (Carpenter et al. 1990): constant in a row (CR: the same value occurs throughout a row, but changes down a column); quantitative pairwise progression (PP: a quantitative increment or decrement occurs between adjacent entries in an attribute such as size, position, or number); figure addition or subtraction (A/S: a figure from one column is added to or subtracted from another figure to produce the third); the distribution of three values (D3: three values from a categorical attribute are distributed through a row); and the distribution of two values (D2: two values from a categorical attribute are distributed through a row; the third value is null). The theory postulates two processes involved in solving RAPM items: correspondence finding and goal management. Correspondence finding entails identifying the elements in the matrix that are governed by a rule, whereas goal management involves decomposing the problem into sub-goals in order to facilitate exploring tentative solution paths while preserving previous progress. These processes rely on different abilities and are affected by different item features. Specifically, correspondence finding relies on the ability to induce abstract relations and is influenced by the type of rule, whereas goal management relies on working memory and is influenced by the number of rules involved in the item. As a result, the difficulty of RAPM items depends on the type and number of rules involved in their resolution. According to Carpenter et al. (1990), the easiest rule to induce is CR, which requires storing only one value for an attribute. The induction of PP is also relatively easy, requiring only basic perceptual comparison of two elements for the pairwise relation to be extrapolated to the third. A/S, D3, and D2, in contrast, are more difficult to induce than PP since they involve conceptual processes and require simultaneous consideration of all three elements. Finally, D2 is the most difficult rule to induce as it requires the additional ability to form abstract correspondences involving null arguments.

In addition, subsequent research has shown that the drawing features involved in the matrices may constitute another source of difficulty in the RAPM (Embretson 2002). More specifically, Embretson (2002) proposed a classification of three figural properties that may impact performance on the RAPM: overlay, distortion, and fusion. Overlay occurs when multiple objects in a matrix entry are superimposed, distortion occurs when the shapes of corresponding objects are deformed, and fusion occurs when multiple objects in an entry can be perceived as one larger object. Embretson (2002) found that distortion was positively related to item difficulty in the RAPM, whereas overlay and fusion correlated negatively with item difficulty, although the effect of overlay was only marginally significant.

Despite the consensus on the relevance of Carpenter et al.’s (1990) theory, later studies have suggested that learning processes may also occur during the RAPM (e.g., Birney et al. 2017; Bui and Birney 2014; Ren et al. 2014; Verguts and De Boeck 2002). These processes unfold across items as individuals practice the rules required to solve them. The hypothesis of learning during the RAPM aligns with Raven’s own conception of the progressive matrices, as he acknowledged that certain learning processes took place while responding to the items (J. Raven 2008). Confirming the existence of rule-learning processes in a benchmark intelligence test such as the RAPM would have important theoretical implications, as it would require incorporating learning into the model as a fundamental component of intelligence.

Although the above-mentioned findings challenge the current theoretical framework of RAPM performance, the existing evidence supporting the learning hypothesis remains inconclusive. Most of the evidence was obtained either under conditions more favorable to learning (e.g., increased exposure to items, explicit feedback, etc.) than those involved in the standard administration of the RAPM or by using modified versions of the items (e.g., Bui and Birney 2014; Verguts and De Boeck 2002), which raises concerns about the validity of the measure as an indicator of general intelligence and the generalizability of the results to the standard use of the RAPM. Additionally, other studies have identified an item-position effect during the RAPM (e.g., Lozano 2015; Ren et al. 2014; Schweizer et al. 2009). However, attributing this effect to rule learning requires further empirical validation.

Lozano and Revuelta (2020) investigated practice effects associated with the abstract rules involved in the RAPM under standard administration conditions. To that end, the authors used operation-specific learning models (Fischer and Formann 1982; Lozano and Revuelta 2021; Spada 1977; Spada and McGaw 1985) in combination with Carpenter et al.’s (1990) taxonomy of rules. This approach is particularly suitable for gathering information on the rule-learning hypothesis, as it allows for modeling practice effects specifically associated with each of the abstract rules involved in the test. However, their results did not support the existence of rule learning in the RAPM. On the contrary, all rules showed a significant increase in difficulty throughout the test as a function of practice. Interestingly, the results suggested the presence of learning effects associated with the items’ figural properties, a possibility that had not received attention in the previous literature.

Nevertheless, Lozano and Revuelta’s (2020) study presents certain limitations that warrant consideration. The models used to analyze the data assume that item difficulty is solely determined by a well-defined set of components (in this case, the RAPM abstract rules and figural properties described above), along with the practice effects associated with these components. However, this assumption may be overly restrictive in the case of the RAPM, which was not designed based on a predefined set of rules. In this regard, unaccounted factors within the RAPM may influence item difficulty, contributing to the observed increase in rule difficulty during the test, and potentially obscuring the presence of rule-learning effects. Consequently, the application of operation-specific learning models to data from progressive matrices strictly generated based on Carpenter et al.’s (1990) taxonomy may help to circumvent this issue and provide complementary information regarding the learning hypothesis in the RAPM.

2. The Present Study

The present study aims to investigate within-test practice effects in theoretically generated progressive matrices based on Carpenter et al.’s (1990) model. This approach minimizes the impact of factors not included in the theoretical model on the study of practice effects. Consequently, changes in difficulty during the test associated with specific components can be more accurately attributed to practice effects such as learning or fatigue. Specifically, the present study analyzed data from the Abstract Reasoning Test (ART; Embretson 1998). The ART was developed using a cognitive design system approach, a method that integrates cognitive theory into test construction to ensure validity. The ART was specifically developed from Carpenter et al.’s (1990) cognitive theory, and its item structures were originally conceived to match the structures studied by Carpenter et al. (1990). The ART’s construct validity has been strongly supported through multiple studies. Specifically, the ART version used in the present study showed a correlation of .78 with the RAPM (Embretson 1998, 2002). In the present study, the analysis of practice effects within the ART was conducted using a random weights operation-specific learning model (RWOSLM; Lozano and Revuelta 2023). The RWOSLM measures within-test practice effects specific to each examinee, which may provide relevant information regarding rule learning during the ART progressive matrices at the individual level. Additionally, correlations between practice effects and external measures associated with intelligence were examined to investigate the nature of the practice effects.

3. Materials and Methods

3.1. Research Model and Statistical Analyses

The RWOSLM is a random weights linear logistic test model (Rijmen and De Boeck 2002) aimed to measure individual differences in operation-specific practice effects. According to the RWOSLM, the logit of the success probability of person i ($i=1,\dots ,n$) on item j ($j=1,\dots ,J$) is given by

$$\mathrm{logit}\left[P\left(X_{ij}=1\right)\right]={\mathsf{\theta }}_i-{\displaystyle \sum _{m=1}^M{w}_{jm}}{\mathsf{\alpha }}_m+{\displaystyle \sum _{m=1}^M{v}_{jm}}{\mathsf{\delta }}_{im},$$

(1)

where:

θ_i is the ability of person i;

α_m is the difficulty of rule m ($m=1,\dots ,M$);

δ_im is the effect of previous practice on the difficulty of rule m for person i;

w_jm is the weight of rule m on item j;

v_jm is given by

$$v_{jm}=w_{jm}{\displaystyle \sum _{k=1}^{j-1}{w}_{km}},$$

(2)

where w_km is the weight of rule m on the previous item k ($k=1,\dots ,j-1$).

The model also includes W, a J × M matrix that contains the weights w_jm, and V, a J × M matrix that contains the weights v_jm. The weights w_jm and v_jm are not estimated. Instead, they are part of the hypothesized test structure, which must be specified before the model is applied.

In this context, the ability parameter θ_i represents abstract reasoning; that is, the ability of person i to identify and apply the rules involved in the item regardless of practice effects. The difficulty parameter α_m represents the initial difficulty (without practice) of rule m. The practice parameter δ_im represents the effect of practicing rule m for person i. A positive sign for the parameter δ_im indicates a decrease in the difficulty of operation m for person i as a function of practice, which may be interpreted as learning. On the other hand, a negative sign for the parameter δ_im indicates an increase in the difficulty of operation m for person i as a function of practice, which may be interpreted as fatigue or loss of interest and/or attention. In this regard, the practice effect may be conceived as a change in the difficulty of a rule for a particular examinee as a function of practice:

$${\mathsf{\alpha }}_{ijm}={\mathsf{\alpha }}_m-{\displaystyle \sum _{k=1}^{j-1}{w}_{km}}{\mathsf{\delta }}_{im},$$

(3)

where α_ijm is the difficulty of rule m for person i at item j.

For the model to be identified, the matrices W and V⁺ = (V, 1) (i.e., V supplemented with a column vector of ones) and the concatenation of both matrices must have a full column rank. Additionally, the number of effects should not exceed the number of items (i.e., 2M + 1 ≤ J)1 and the covariance matrix of the practice effects must be symmetric positive definite.

Model estimation and evaluation were based on a Bayesian framework. The RWOSLM (Lozano and Revuelta 2023) was compared with other models: the operation-specific learning model (OSLM; Fischer and Formann 1982; Lozano and Revuelta 2021; Scheiblechner 1972; Spada 1977; Spada and McGaw 1985), the linear logistic test model (LLTM; Fischer 1973; Scheiblechner 1972), and the random weights linear logistic test model (RWLLTM; Rijmen and De Boeck 2002). While in the RWOSLM the difficulty parameters (α_m) are constant across individuals and the practice parameters (δ_im) are random across individuals2, in the OSLM both difficulty (α_m) and practice (δ_m) parameters are constant across individuals. In this regard, the OSLM represents the absence of individual differences in practice effects. Additionally, the LLTM and the RWLLTM explain item difficulty exclusively as a function of the difficulty of the abstract rules involved in the item. Therefore, both models represent the absence of within-test practice effects. The difference between these two models resides in that the LLTM treats the difficulty parameters as fixed effects constant across individuals (from now on denoted as η_m), whereas the RWLLTM treats them as random effects varying across individuals (from now on denoted as η_im).

Based on previous research (Lozano and Revuelta 2023), the prior distributions for the RWOSLM parameters were

$${\mathsf{\theta }}_i\sim N\left(0,1\right),$$

(4)

$${\mathsf{\alpha }}_m\sim N\left(0,100\right),\mathrm{and}$$

(5)

$${\mathsf{\delta }}_i\sim N\left({\mathsf{\mu }}_{\mathsf{\delta }},{\mathsf{\Sigma }}_{\mathsf{\delta }}\right),$$

(6)

where ${\mathsf{\delta }}_i=\left({\mathsf{\delta }}_{i1},\dots ,{\mathsf{\delta }}_{iM}\right)$ is an M vector of practice parameters of person i; ${\mathsf{\mu }}_{\mathsf{\delta }}=\left({\mathsf{\mu }}_{{\mathsf{\delta }}_1},\dots ,{\mathsf{\mu }}_{{\mathsf{\delta }}_M}\right)$ is an M prior mean vector; ${\mathsf{\Sigma }}_{\mathsf{\delta }}=\mathrm{diag}\left({\mathsf{\sigma }}_{\mathsf{\delta }}\right){\mathsf{\Omega }}_{\mathsf{\delta }}\mathrm{diag}\left({\mathsf{\sigma }}_{\mathsf{\delta }}\right)$ is an M × M prior covariance matrix; ${\mathsf{\sigma }}_{\mathsf{\delta }}=\left({\mathsf{\sigma }}_{{\mathsf{\delta }}_1},\dots ,{\mathsf{\sigma }}_{{\mathsf{\delta }}_M}\right)$ is an M vector of prior standard deviations; and ${\mathsf{\Omega }}_{\mathsf{\delta }}$ is a prior correlation matrix.

The hyper-prior distributions for the hyper-parameters were

$${\mathsf{\mu }}_{{\mathsf{\delta }}_m}\sim N\left(0,1\right),$$

(7)

$${\mathsf{\sigma }}_{{\mathsf{\delta }}_m}\sim \text{half-Cauchy}\left(0,5\right),\mathrm{and}$$

(8)

$${\mathsf{\Omega }}_{\mathsf{\delta }}\sim \mathrm{LKJ}\left(2\right),$$

(9)

where LKJ is the distribution developed by Lewandowski et al. (2009), which is less informative than the Wishart prior traditionally used for covariance matrices. Note that the prior standard deviations of the practice parameters (${\mathsf{\sigma }}_{{\mathsf{\delta }}_m}$) are informative about individual differences in practice effects.

Additionally, based on previous research (Lozano and Revuelta 2021, 2023), an N(0, 100) prior distribution was specified for the fixed-effects parameters of the OSLM (α_m and δ_m). In line with the previous specifications, the random-effects operation difficulty parameter of the RWLLTM (η_im) was assumed to follow the same multivariate normal prior distribution specified for δ_im in the RWOSLM, whereas the fixed-effects operation difficulty parameter of the LLTM (η_m) was assumed to follow an N(0, 100) prior distribution. A standard normal prior distribution was specified for the person ability parameter (θ_i) in all models.

Two information criterion measures were used for model comparison and selection: the widely applicable information criteria (WAIC; Watanabe 2010, 2013) and the leave-one-out information criterion (LOOIC; Gelman et al. 2014; Vehtari et al. 2017). These measures quantify the discrepancy between the model and the data while penalizing for model complexity. This penalty compensates for the over-fitting exhibited by more complex models as a result of their greater flexibility. Lower values indicate a better balance between fit and parsimony. Both WAIC and LOOIC have proven useful for detecting within-test practice effects (Lozano and Revuelta 2021, 2023).

Complementarily, posterior predictive model checking (PPMC; Gelman et al. 1996) was used to assess the fit of the models to the data. The posterior predictive p-value (p_post; Meng 1994) was computed using the odds ratio (OR; Chen and Thissen 1997; Sinharay 2005) as a test statistic. The OR statistic is a measure of association between pairs of items that have been effective in identifying within-test practice effects (Lozano and Revuelta 2021, 2023).

3.2. Software

The analyses were conducted with R version 4.4.0 (R Core Team 2024) and the RStan R package version 2.35 (Stan Development Team 2024). Markov chain Monte Carlo simulation was performed using the No-U-Turn Sampler (NUTS; Hoffman and Gelman 2014). Four Markov chains of 5000 samples each were run. The first half of the samples were discarded as burn-in. The potential scale reduction statistic ($\widehat{R}$; Gelman and Rubin 1992) was used to evaluate the convergence of the chains. WAIC and LOOIC were obtained using the loo R package version 2.8.0 (Vehtari et al. 2024).

3.3. Measures and Data

The present study is based on data originally collected by Embretson (1998). The dataset comprised responses of 818 military recruits (80% males) to 34 items of the ART. Participants were randomly assigned to three groups which were administered equivalent computerized forms of the ART. The items administered to the three groups were generated from the same theoretical structure (same number and type of rules) and figural properties, although the specific objects varied between groups. Each form included thirty structurally matched items and four anchor items distributed across the test. Items were ordered by predicted difficulty and their position was equated across forms. Appendix A shows the theoretical structure of the three test forms. Figure 1 presents an example of an ART item involving CR, D3, and overlay; the correct answer is 8. The matrices W and V⁺, and the concatenation of both matrices, satisfy the full column rank condition, and the number of effects does not exceed the number of items.

A total of 6.5% of participants were excluded due to response latencies less than one second, which were interpreted as involuntary responses resulting from the computerized administration of the items. The final group sample size was n = 765. A linear regression analysis showed that abstract rules and figural properties accounted for 70% of the variance in item difficulty, operationalized as Rasch item difficulty parameters. Item position did not significantly contribute to the explanation of item difficulty beyond the abstract rules and figural properties.

In addition to the ART, participants completed several other measures as part of the original study. For the present study, we utilized a General Test Battery (GTB) and the Revised NEO Personality Inventory (NEO PI-R; Costa and McCrae 1992) to examine their relationships with individual differences in practice effects in the ART. GTB aggregates several measures of cognitive abilities and exhibits a high load on the general factor of intelligence. The correlation between GTB and the RAPM is .48 (Embretson 1998). Regarding the NEO PI-R, previous research suggests that intelligence is positively and moderately related to Openness, while marginal and mostly inconsistent relationships have been observed with the other personality dimensions (DeYoung 2020).

4. Results

Table 1. Goodness-of-fit estimates for the fitted models.

	WAIC				LOO				OR
Model	elpdwaic	pwaic	waic		elpdloo	ploo	looic		Observed	Simulated (Sd)	ppost
RWOSLM	−14,320.8	1060.8	28,641.7		−14,330.5	1070.5	28,661.0		1344.92	1112.31 (30.40)	.000
RWOSLMconst.	−14,326.0	848.2	28,652.0		−14,331.5	853.6	28,662.9		1344.92	1118.90 (29.45)	.000
OSLM	−14,339.8	653.9	28,679.5		−14,341.4	655.5	28,682.8		1344.92	1117.81 (30.13)	.000
RWLLTM	−14,805.8	1052.0	29,611.7		−14,812.6	1058.7	29,625.2		1344.92	1064.48 (25.69)	.000
LLTM	−14,814.3	646.8	29,628.6		−14,816.1	648.6	29,632.3		1344.92	1059.84 (25.65)	.000

shows the goodness-of-fit estimates for the fitted models. The RWOSLM yielded the lowest WAIC and LOOIC values, indicating superior out-of-sample predictive performance. However, an inspection of the prior variances associated with the RWOSLM practice parameters revealed that only A/S and distortion exhibited substantial variability in practice effects. Subsequently, a constrained version of the RWOSLM (hereafter referred to as RWOSLM_const.) was fitted to the data in which all practice parameters, except those associated with A/S and distortion, were treated as fixed effects. The RWOSLM_const. retained the lowest WAIC and LOOIC values among all models. Although posterior predictive checks indicated that none of the models showed a good fit to the data (all p_post < .05), the RWOSLM_const. generated replicated OR values that more closely approximated the realized values, suggesting that this model was the most appropriate for analyzing within-test practice effects in the ART.

Table 2. Expected a posteriori estimates (EAP), posterior standard deviations (SD), and posterior probability intervals (2.5–97.5%) for the RWOSLM_const. parameters.

	EAP	SD	2.5%	97.5%		EAP	SD	2.5%	97.5%		EAP	SD	2.5%	97.5%
αCR	−1.585	0.103	−1.784	−1.391	δCR	−0.090	0.011	−0.112	−0.068
αPP	1.017	0.069	0.892	1.156	δPP	0.095	0.010	0.075	0.115
αA/S	−0.538	0.072	−0.674	−0.399	${\mathsf{\mu }}_{{\mathsf{\delta }}_{\mathrm{A}/\mathrm{S}}}$	−0.022	0.009	−0.040	−0.004	${\mathsf{\sigma }}_{{\mathsf{\delta }}_{\mathrm{A}/\mathrm{S}}}^2$	0.003	0.001	0.001	0.004
αD3	0.045	0.064	−0.080	0.171	δD3	−0.004	0.005	−0.014	0.006
αD2	0.213	0.045	0.122	0.305	δD2	−0.060	0.015	−0.089	−0.030
αOverlay	−0.503	0.106	−0.719	−0.290	δOverlay	−0.066	0.008	−0.080	−0.051
αDistortion	−0.004	0.092	−0.184	0.176	${\mathsf{\mu }}_{{\mathsf{\delta }}_{\mathrm{Distortion}}}$	−0.017	0.075	−0.168	0.136	${\mathsf{\sigma }}_{{\mathsf{\delta }}_{\mathrm{Distortion}}}^2$	0.042	0.011	0.001	0.144
αFusion	−0.906	0.122	−1.138	−0.673	δFusion	−0.125	0.034	−0.191	−0.059

shows the expected a posteriori (EAP) estimates, posterior standard deviations, and posterior probability intervals for the RWOSLM_const. parameters. The positive sign of the EAP estimate of the practice parameter associated with PP (δ_PP), together with the absence of zero within the posterior probability interval, suggested a decrease in the difficulty of this rule throughout the test. In the case of D3 and distortion, zero was included in the posterior probability interval of the practice parameter (δ_D3) and in the posterior probability interval of the prior mean of the practice parameter (${\mathsf{\mu }}_{{\mathsf{\delta }}_{\mathrm{Distortion}}}$), respectively, indicating a low posterior probability of a decrease in difficulty throughout the test: P(δ_D3 > 0 | x) = .196 and P(${\mathsf{\mu }}_{{\mathsf{\delta }}_{\mathrm{Distortion}}}$> 0 | x) = .399. Conversely, the negative signs of the EAP estimates of the practice parameters associated with CR, D2, overlay, and fusion (δ_CR, δ_D2, δ_Overlay, and δ_Fusion) and the negative sign of the EAP estimate of the prior mean of the practice parameter associated with A/S (${\mathsf{\mu }}_{{\mathsf{\delta }}_{\mathrm{A}/\mathrm{S}}}$), together with the absence of zero within the corresponding posterior probability intervals, suggested an increase in difficulty for these components throughout the test.

The previous results represent overall trends. However, the estimated prior variances of the practice parameters of A/S and distortion (${\mathsf{\sigma }}_{{\mathsf{\delta }}_{\mathrm{A}/\mathrm{S}}}^2$ and ${\mathsf{\sigma }}_{{\mathsf{\delta }}_{\mathrm{Distortion}}}^2$) indicated the presence of inter-individual variability in the practice effects associated with these components. Figure 2 shows the difficulty of the ART rules and figural properties as a function of practice, revealing the presence of individual differences in the practice effects associated with A/S and distortion. An analysis at the individual level revealed that 19% and 37% of the participants exhibited a positive EAP estimate of the practice parameter associated with A/S (δ_iA/S) and distortion (δ_iDistortion), respectively, suggesting an operation-specific reduction in difficulty throughout the test for the respective proportions of participants.

Table 3. Pearson correlations between the EAP estimates of the random-effects practice parameters of the RWOSLM_const. and the reference measures.

	θi	δiA/S	δiDistortion
GTB	.513 **	.254 **	.221 **
Openness	.175 **	.075 *	.097 **
Conscientiousness	−.098 **	−.103 **	−.079 *
Extraversion	.014	−.065	−.085 *
Agreeableness	−.076 *	−.094 **	−.091 *
Neuroticism	−.071 *	−.040	−.005

** significant at the .01 level (two-tailed); * significant at the .05 level (two-tailed).

shows the Pearson correlations between the EAP estimates of the random-effects parameters of the RWOSLM_const. and the reference measures GTB and NEO PI-R. As expected, the ability parameter (θ_i) correlated positively and significantly with GTB and Openness. It also correlated negatively and significantly with Conscientiousness, and showed marginal negative correlations with Agreeableness and Neuroticism. With regard to the practice parameters, the practice parameter associated with A/S correlated positively and significantly with GTB, negatively and significantly with Conscientiousness and Agreeableness, and exhibited a marginal positive correlation with Openness; whereas the practice parameter associated with distortion correlated positively and significantly with GTB and Openness, and showed marginal negative correlations with Conscientiousness, Extraversion, and Agreeableness.

5. Discussion

The conceptualization of intelligence has historically lagged behind its measurement. Understanding the cognitive processes underlying performance on intelligence tests has played a crucial role in shaping our definition of intelligence. Consequently, identifying individual differences in learning processes when using reference measurement instruments could significantly influence our conceptualization of intelligence. In this vein, this study aimed to examine inter-individual variability in the practice effects associated with the abstract rules involved in the ART. The fact that the ART was developed based on Carpenter et al.’s (1990) theory minimizes the impact of extraneous factors unrelated to the theoretical model, thereby allowing for a more accurate identification of practice effects.

Evidence of decreases in difficulty associated with abstract rules during the ART was scarce, primarily observed in PP and, to a lesser extent, in A/S. The evidence concerning PP was consistent across participants, suggesting that performance improvements in this rule occur naturally throughout the test. In contrast, evidence related to A/S was far more limited, indicating that improvements with this rule are considerably more challenging to achieve. Specifically, only 19% of participants showed a decrease in difficulty associated with A/S. Individual differences in practice effects associated with A/S correlated positively with the reference ability measure and Openness. These findings suggest that practice effects during the ART are fundamentally linked to intelligence, as they predict performance on complex ability tasks and relate to measures of intellect.

Regarding the items’ figural properties, a decrease in difficulty throughout the test was observed only for distortion. Specifically, 37% of participants showed a reduction in difficulty, suggesting the presence of learning effects related to this property. The positive and significant correlations with GTB and Openness further support the association between individual differences in practice effects within the ART and intelligence.

The increases in difficulty observed in CR, D2, overlay, and fusion, as well as in A/S and distortion for most participants, may be interpreted as progressive fatigue or a decline in interest and/or attention throughout the test. The results therefore suggest that participants improve performance on some components and exhibit impairments on others, both as a result of practice. Alternatively, an increasing presence of difficulty factors not considered by the model as the test progresses could also explain the rising difficulty observed for some rules. However, the fact that the ART was generated directly from theory minimizes the risk of unmodeled factors. Moreover, the results indicated that item position did not significantly contribute to explaining item difficulty beyond what the model’s components accounted for, which provides no evidence for extraneous factors accounting for the observed increases in difficulty.

Nevertheless, it is important to add two cautionary notes to the discussion. First, the additive nature of the fitted models in regard to item difficulty ignored the possibility of interaction effects between components. In this sense, an observed increase in rule difficulty throughout the test might be partially due to interaction effects between rules. Although a linear modeling approach to performance on the ART may be efficient (see also Embretson 1998), the possibility of interaction effects cannot be dismissed. The RWOSLM allows specifying interactions between operations by including new components in the structure matrix that represent the combination of two or more operations within the same item. However, in the case of the ART, the high number of components in the concatenation of the matrices W and V makes it difficult to explore these potential interactions without the estimates becoming highly unstable. Future research on operation-specific learning modeling may help address this issue. Second, the linearity of the fitted models assumed that the relationship between rule difficulty and practice is constant throughout the entire test. However, the literature on learning and fatigue has traditionally suggested that these processes are better represented by curves. In this regard, future studies may be aimed at investigating nonlinear practice effects within progressive matrices.

In summary, this study provides evidence of inter-individual variability in within-test practice effects during the ART. Although evidence supporting learning effects during the test was limited to only a few components of the structure matrix, the results provide relevant information regarding the cognitive processes involved in solving progressive matrices. Given the centrality of progressive matrices in the measurement of intelligence, it would be valuable to investigate whether similar effects occur in other inductive reasoning tests based on pattern recognition in figural matrices. Future research using advanced dynamic item response models could further explore this possibility.