# A Solution to the Measurement Problem in the Idiographic Approach Using Computer Adaptive Practicing

^{1}

^{2}

^{3}

^{*}

^{†}

## Abstract

**:**

## 1. Introduction

#### 1.1. Computer Adaptive Practice

#### 1.2. Math Garden

## 2. Scientific Analysis

#### 2.1. Intraindividual Analysis

## 3. Study 1: Learning Analytics

- Response probabilities of the last two responses.
- Transition probability matrix of correct and incorrect responses.
- Coefficients of a logistic regression model.

#### Results

## 4. Study 2: The Problematic Assumption of Unidimensionality

#### Results

## 5. Discussion

## Acknowledgments

## Author Contributions

## Conflicts of Interest

## Abbreviations

IRT | Item response theory |

CAT | Computer adaptive testing |

## References

- Van der Maas, H.L.J.; Kan, K.J.; Hofman, A.D.; Raijmakers, M.E.J. Dynamics of Development: A complex systems approach. In Inductive Developmental Systems; Guilford Press: New York, NY, USA, 2013; pp. 270–286. [Google Scholar]
- Molenaar, P.C.M. A manifesto on psychology as idiographic science: Bringing the person back into scientific psychology, this time forever. Measurement
**2004**, 2, 201–218. [Google Scholar] [CrossRef] - Adolph, K.E.; Robinson, S.R.; Young, J.W.; Gill-Alvarez, F. What is the shape of developmental change? Psychol. Rev.
**2008**, 115, 527. [Google Scholar] [CrossRef] [PubMed] - Siegler, R.S.; Crowley, K. The microgenetic method: A direct means for studying cognitive development. Am. Psychol.
**1991**, 46, 606–620. [Google Scholar] [CrossRef] [PubMed] - Dolan, C.V.; van der Maas, H.L.J. Fitting multivariage normal finite mixtures subject to structural equation modeling. Psychometrika
**1998**, 63, 227–253. [Google Scholar] [CrossRef] - Klinkenberg, S.; Straatemeier, M.; van der Maas, H.L.J. Computer adaptive practice of maths ability using a new item response model for on the fly ability and difficulty estimation. Comput. Educ.
**2011**, 57, 1813–1824. [Google Scholar] [CrossRef] - Straatemeier, M. Math Garden: A New Educational and Scientific Instrument. PhD Thesis, University of Amsterdam, Amsterdam, The Netherlands, 2014. [Google Scholar]
- OECD. Education at a Glance: OECD Indicators; OECD: Paris, France, 2006. [Google Scholar]
- Wainer, H. Computerized Adaptive Testing: A Primer; Lawrence Erlbaum: Hillsdale, MI, USA, 2000. [Google Scholar]
- Rasch, G. Probabilistic Models for Some Intelligence and Achievement Tests; Danish Institute for Educational Research: Copenhagen, Denmark, 1960. [Google Scholar]
- Eggen, T.J.H.M.; Verschoor, A.J. Optimal testing with easy or difficult items in computerized adaptive testing. Appl. Psychol. Meas.
**2006**, 30, 379–393. [Google Scholar] [CrossRef] - Elo, A.E. The Rating of Chessplayers, Past and Present; Arco Pub: New York, USA, 1978. [Google Scholar]
- Maris, G.; van der Maas, H.L.J. Speed-accuracy response models: Scoring rules based on response time and accuracy. Psychometrika
**2012**, 77, 615–633. [Google Scholar] [CrossRef] - Van Der Maas, H.L.J.; Wagenmakers, E.J. A psychometric analysis of chess expertise. Am. J. Psychol.
**2005**, 118, 29–60. [Google Scholar] [PubMed] - Birnbaum, Z.W.; Saunders, S.C. A new family of life distributions. J. Appl. Probab.
**1969**, 6, 319–327. [Google Scholar] [CrossRef] - Van den Bergh, M.; Hofman, A.D.; Schmittmann, V.D.; van der Maas, H.L.J. Tracing the development of typewriting skills in an adaptive e-learning environment. Percept. Mot. Skills
**2015**, 121, 727–745. [Google Scholar] [CrossRef] [PubMed] - Ericsson, K.A. The influence of experience and deliberate practice on the development of superior expert performance. In The Cambridge Handbook of Expertise and Expert Performance; Cambridge University Press: Cambridge, UK, 2006; Volume 38, pp. 685–705. [Google Scholar]
- Gierasimczuk, N.; Van der Maas, H.L.J.; Raijmakers, M.E.J. An analytic tableaux model for deductive mastermind empirically tested with a massively used online learning system. J. Log. Lang. Inf.
**2013**, 22, 297–314. [Google Scholar] [CrossRef] - Jansen, B.R.J.; De Lange, E.; Van der Molen, M.J. Math practice and its influence on math skills and executive functions in adolescents with mild to borderline intellectual disability. Res. Dev. Disabil.
**2013**, 34, 1815–1824. [Google Scholar] [CrossRef] [PubMed] - Jansen, B.R.J.; Hofman, A.D.; Straatemeier, M.; Bers, B.M.C.W.; Raijmakers, M.E.J.; Maas, H.L.J. The role of pattern recognition in children’s exact enumeration of small numbers. Br. J. Dev. Psychol.
**2014**, 32, 178–194. [Google Scholar] [CrossRef] [PubMed] - Jansen, B.R.J.; Louwerse, J.; Straatemeier, M.; Van der Ven, S.H.G.; Klinkenberg, S.; Van der Maas, H.L.J. The influence of experiencing success in math on math anxiety, perceived math competence, and math performance. Learn. Individ. Differ.
**2013**, 24, 190–197. [Google Scholar] [CrossRef] - Nyamsuren, E.; van der Maas, H.L.J.; Taatgen, N.A. How does prevalence shape errors in complex tasks? In Proceedings of the International Conference on Cognitive Modeling, Groningen, The Netherlands, 9–11 April 2015; Volume 13, pp. 160–165. [Google Scholar]
- Hofman, A.D.; Visser, I.; Jansen, B.R.J.; van der Maas, H.L.J. The balance-scale task revisited: A comparison of statistical models for rule-based and information-integration theories of proportional reasoning. PLoS ONE
**2015**, 10, e0136449. [Google Scholar] [CrossRef] [PubMed] - Van Der Ven, S.H.G.; Van Der Maas, H.L.J.; Straatemeier, M.; Jansen, B.R.J. Visuospatial working memory and mathematical ability at different ages throughout primary school. Learn. Individ. Differ.
**2013**, 27, 182–192. [Google Scholar] [CrossRef] - Jansen, B.R.J.; Hofman, A.D.; Savi, A.O.; Visser, I.; van der Maas, H.L.J. Self-adapting the success rate when practicing math. Learn. Individ. Differ.
**2016**, 51, 1–10. [Google Scholar] [CrossRef] - Kadengye, D.T.; Ceulemans, E.; Van den Noortgate, W. A generalized longitudinal mixture IRT model for measuring differential growth in learning environments. Behav. Res. Methods
**2014**, 46, 823–840. [Google Scholar] [CrossRef] [PubMed] - Buwalda, T.A.; Borst, J.P.; van der Maas, H.J.L.; Taatgen, N.A. Explaining mistakes in single digit multiplication: A cognitive model. In Proceedings of the 14th International Conference on Cognitive Modeling, University Park, PA, USA, 11–18 September 2016; pp. 131–136. [Google Scholar]
- Hofman, A.D.; Visser, I.; Jansen, B.R.J.; Marsman, M.; van der Maas, H.L.J. Fast and Slow Strategies in Multiplication; PsyArXiv: Grand Rapids, MI, USA, 2017. [Google Scholar]
- Brinkhuis, M.J.S.; Savi, A.O.; Hofman, A.D.; Coomans, F.; van der Maas, H.L.J.; Maris, G. Learning as It Happens: A Decade of Analyzing and Shaping a Large-Scale Online Learning System. PsyArXiv 2018. Available online: https://osf.io/hfnxg/ (accessed on 22 December 2017).
- Savi, A.; Ruijs, N.; Maris, G.; van der Maas, H. Online Learning Solves Sturdy Problems in Education; PsyArXiv: Grand Rapids, MI, USA, 2017. [Google Scholar]
- Coomans, F.; Hofman, A.D.; Brinkhuis, M.J.S.; van der Maas, H.L.J.; Maris, G. Distinguishing fast and slow processes in accuracy-response time data. PLoS ONE
**2016**, 11, e0155149. [Google Scholar] [CrossRef] [PubMed] [Green Version] - De Bree, E.; van der Ven, S.H.G.; van der Maas, H.L.J. The voice of Holland: Allograph production in written Dutch past tense inflection. Lang. Learn. Dev.
**2017**, 13, 215–240. [Google Scholar] [CrossRef] - Braithwaite, D.W.; Goldstone, R.L.; van der Maas, H.L.J.; Landy, D.H. Non-formal mechanisms in mathematical cognitive development: The case of arithmetic. Cognition
**2016**, 149, 40–55. [Google Scholar] [CrossRef] [PubMed] - Van der Ven, S.H.G.; Klaiber, J.D.; van der Maas, H.L.J. Four and twenty blackbirds: How transcoding ability mediates the relationship between visuospatial working memory and math in a language with inversion. Educ. Psychol.
**2017**, 37, 487–505. [Google Scholar] [CrossRef] - Hoyles, C.; Wolf, A.; Molyneux-Hodgson, S.; Kent, P. Mathematical Skills in the Workplace: Final Report to the Science Technology and Mathematics Council; Institute of Education, University of London; Science, Technology and Mathematics Council: London, UK, 2002.
- Finnie, R.; Meng, R. The Importance of Functional Literacy: Reading and Math Skills and Labour Market Outcomes of High School Drop-Outs; Technical Report; Business and Labour Market Analysis Division: Ottawa, ON, Canada, 2006.
- Reyna, V.F.; Brainerd, C.J. The importance of mathematics in health and human judgment: Numeracy, risk communication, and medical decision making. Learn. Individ. Differ.
**2007**, 17, 147–159. [Google Scholar] [CrossRef] - Van der Ven, S.H.G.; Kroesbergen, E.H.; Boom, J.; Leseman, P.P.M. The development of executive functions and early mathematics: A dynamic relationship. Br. J. Educ. Psychol.
**2012**, 82, 100–119. [Google Scholar] [CrossRef] [PubMed] - Gelman, A.; Jakulin, A.; Pittau, M.G.; Su, Y.S. A weakly informative default prior distribution for logistic and other regression models. Ann. Appl. Stat.
**2008**, 2, 1360–1383. [Google Scholar] [CrossRef] - Gelman, A.; Su, Y.S.; Yajima, M.; Hill, J.; Pittau, M.G.; Kerman, J.; Zheng, T.; Dorie, V. Arm: Data Analysis Using Regression and Multilevel/Hierarchical Models; R Package, Version 9.01; Cambridge University Press: Cambridge, UK, 2006. [Google Scholar]
- R Core Team. R: A Language and Environment for Statistical Computing; R Core Team: Vienna, Austria, 2013. [Google Scholar]
- Van der Ven, S.H.G.; Straatemeier, M.; Jansen, B.R.J.; Klinkenberg, S.; van der Maas, H.L.J. Learning multiplication: An integrated analysis of the multiplication ability of primary school children and the difficulty of single digit and multidigit multiplication problems. Learn. Individ. Differ.
**2015**, 43, 48–62. [Google Scholar] [CrossRef] - Siegler, R.S. Microgenetic analyses of learning. In Handbook of Child Psychology; John Wiley & Sons, Inc.: Hoboken, NJ, USA, 2006. [Google Scholar]
- Nesselroade, J.R.; Gerstorf, D.; Hardy, S.A.; Ram, N. Focus article: Idiographic filters for psychological constructs. Measurement
**2007**, 5, 217–235. [Google Scholar] [CrossRef] - Pelánek, R.; Papoušek, J.; Řihák, J.; Stanislav, V.; Nižnan, J. Elo-based learner modeling for the adaptive practice of facts. User Model. User-Adapt. Interact.
**2016**, 27, 89–118. [Google Scholar] [CrossRef] - Kuhn, D.; Garcia-Mila, M.; Zohar, A.; Andersen, C.; White, S.H.; Klahr, D.; Carver, S.M. Strategies of knowledge acquisition. In Monographs of the Society for Research in Child Development; John Wiley & Sons, Inc.: Hoboken, NJ, USA, 1995. [Google Scholar]
- Siegler, R.S. Emerging Minds: The Process of Change in Children’s Thinking; Oxford University Press: Oxford, UK, 1996. [Google Scholar]
- Lemaire, P. Executive functions and strategic aspects of arithmetic performance: The case of adults’ and children’s arithmetic. Psychol. Belg.
**2010**, 50, 335–352. [Google Scholar] [CrossRef] - Ambrose, R.; Baek, J.; Carpenter, T.P. Children’s invention of multidigit multiplication and division algorithms. In The Development of Arithmetic Concepts and Skills: Constructive Adaptive Expertise; Routledge: Mahwah, NJ, USA, 2003; pp. 305–336. [Google Scholar]
- Dowker, A. Individual Differences in Arithmetic: Implications for Psychology, Neuroscience and Education; Psychology Press: East Sussex, UK, 2005. [Google Scholar]
- Ashcraft, M.H.; Guillaume, M.M. Mathematical cognition and the problem size effect. Psychol. Learn. Motiv.
**2009**, 51, 121–151. [Google Scholar] - Lemaire, P.; Siegler, R.S. Four aspects of strategic change: Contributions to children’s learning of multiplication. J. Exp. Psychol. Gen.
**1995**, 124, 83. [Google Scholar] [CrossRef] [PubMed]

Sample Availability: Interested researchers can contact the first author or last author to request parts of the data for research purposes. | |

1 | Given the current large number of Math Garden players item ratings converge within about a week. |

2 | For many results in this section, formal significance tests could be provided. However, we refrain from doing so because: (1) no clear a priori hypothesis could be formulated and (2) due to the large data sets very small and uninteresting results become significant in a null-hypothesis testing framework. For example, the 35% did significantly differ from the expected 36% according to a simple proportion test (${X}^{2}$(1) = 58.60, p = 1.936 ×${10}^{-14}$) |

3 | For the permutation test we used 5000 replications where in each replication all items were randomly distributed between two sets. For each set we calculated each player’s average slopes (i.e., mean learning speed) and the correlation between both of these averages values. |

**Figure 2.**One player’s development in learning to solve multiplication problems correctly. The colors refer to correct (green), incorrect (red), question mark (blue) or responses that were too late (yellow). The minimum number of responses for each time-series was 5. The items are sorted by item difficulty (low = easy and high = difficult). Plots for other players, providing different patterns, are available on www.abehofman.com/papers. The arrows along the Y axis indicate the items that are further described in the text (black arrows illustrate different response patterns; grey arrows indicate different trajectories of mastery).

**Figure 3.**The distribution of the transition probabilities of switching from incorrect to correct responses (

**top**) and remaining at a correct response (

**bottom**) for addition (

**left**) and multiplication (

**right**) for all collected time-series. For, example, the transition (learning) probability $(0\to 1)$ of 0.7 indicates that 70% of incorrect responses are immediately followed by a correct response. The bar at 0.7 in the upper-left panel indicates that this is the case in about 10% of all the collected series in the addition game.

**Figure 4.**An example of three different developmental patterns of responses to different items by the same player. The left panel shows a time-series with a clear increase in the probability of a correct response. The middle and right panel respectively show a series of a previously learned item and a series that indicates no learning.

**Figure 5.**The distribution of the estimated slope parameters, i.e., indicator of learning speed, for both data sets (

**left**-panel), and the relation between the length of the series and learning speed (

**middle**-panel) and the probability of switching from an incorrect to a correct response (

**right**-panel).

**Figure 6.**The distributions of the estimated correlations between the local skills in the addition, multiplication and simulated data.

**Figure 7.**A heatmap based on the correlation matrix of the local skills estimated on a simulated data set based on the unidimensional model. Gray squares indicate missing values in the correlation matrix, resulting from adaptive item selection. The figure can also be found on www.abehofman.com/papers allowing for more detailed inspection.

**Figure 8.**A heatmap based on the correlations matrix of the local skills of 200 multiplication items. Gray squares indicate missing values in the correlation matrix, resulting from the adaptive item selection. The figure can also be found on www.abehofman.com/papers allowing for more detailed inspection.

**Figure 9.**A heatmap based on the correlations matrix of the local user abilities of 200 addition items. Gray squares indicate missing values in the correlation matrix. See www.abehofman.com/papers for a downloadable version.

© 2018 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Hofman, A.D.; Jansen, B.R.J.; De Mooij, S.M.M.; Stevenson, C.E.; Van der Maas, H.L.J.
A Solution to the Measurement Problem in the Idiographic Approach Using Computer Adaptive Practicing. *J. Intell.* **2018**, *6*, 14.
https://doi.org/10.3390/jintelligence6010014

**AMA Style**

Hofman AD, Jansen BRJ, De Mooij SMM, Stevenson CE, Van der Maas HLJ.
A Solution to the Measurement Problem in the Idiographic Approach Using Computer Adaptive Practicing. *Journal of Intelligence*. 2018; 6(1):14.
https://doi.org/10.3390/jintelligence6010014

**Chicago/Turabian Style**

Hofman, Abe D., Brenda R. J. Jansen, Susanne M. M. De Mooij, Claire E. Stevenson, and Han L. J. Van der Maas.
2018. "A Solution to the Measurement Problem in the Idiographic Approach Using Computer Adaptive Practicing" *Journal of Intelligence* 6, no. 1: 14.
https://doi.org/10.3390/jintelligence6010014