# Exploring the Influence of Item Characteristics in a Spatial Reasoning Task

^{*}

## Abstract

**:**

## 1. Introduction

#### 1.1. Construct Representation, Response Processes, and Explanatory Item Response Theory

#### 1.2. Object Assembly and Its Item Characteristics

#### 1.3. The Current Study

- What are the overall psychometric characteristics of an object assembly task used to assess spatial ability?
- How do the characteristics of the object assembly items contribute to item difficulty?

#### 1.4. Linear Logistic Test Model

## 2. Materials and Methods

#### 2.1. Participants

#### 2.2. Instrument

#### 2.3. Data Analysis

## 3. Results

#### 3.1. Dichotomous Rasch Model

#### DIF Analysis of the Two Subgroups

#### 3.2. LLTM

## 4. Discussion

#### 4.1. What Are the Overall Psychometric Characteristics of Object Assembly?

#### 4.2. How Do the Characteristics of the Object Assembly Items Contribute to Item Difficulty?

#### 4.3. Implications

#### 4.4. Limitations

#### 4.5. Directions for Future Research

## Author Contributions

## Funding

## Institutional Review Board Statement

## Informed Consent Statement

## Data Availability Statement

## Acknowledgments

## Conflicts of Interest

## References

- Alexandrowicz, Rainer W. 2011. Statistical and Practical Significance of the Likelihood Ratio Test of the Linear Logistic Test Model Versus the Rasch Model. Educational Research and Evaluation: An International Journal on Theory and Practice 17: 335–50. [Google Scholar] [CrossRef]
- Baghaei, Purya, and Christine Hohensinn. 2017. A Method of Q-Matrix Validation for the Linear Logistic Test Model. Frontiers in Psychology 8: 897. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Baghaei, Purya, and Hamdollah Ravand. 2015. A Cognitive Processing Model of Reading Comprehension in English as a Foreign Language Using the Linear Logistic Test Model. Learning and Individual Differences 43: 100–5. [Google Scholar] [CrossRef]
- Baghaei, Purya, and Klaus D. Kubinger. 2015. Linear Logistic Test Modeling with R. Practical Assessment, Research, and Evaluation 20: 1. [Google Scholar] [CrossRef]
- Bond, Trevor G., Zi Yan, and Moritz Heene. 2020. Applying the Rasch Model: Fundamental Measurement in the Human Sciences, 4th ed. London: Routledge. [Google Scholar]
- Cao, Chunhua, Yan Wang, Yi-Hsin Chen, and Isaac Y. Li. 2014. Parameter Estimation of Cognitive At-tributes Using the Crossed Random-Effects Linear Logistic Test Model with PROC GLIMMIX. Paper presented at SAS Global Forum (SASGF), Washington, DC, USA, March 23–26; Available online: https://support.sas.com/resources/papers/proceedings14/1766-2014.pdf (accessed on 26 March 2023).
- Cheng, Yi-Ling, and Kelly S. Mix. 2014. Spatial Training Improves Children’s Mathematics Ability. Journal of Cognition and Development 15: 2–11. [Google Scholar] [CrossRef]
- Chou, Yeh-Tai, and Wen-Chung Wang. 2010. Checking Dimensionality in Item Response Models with Principal Component Analysis on Standardized Residuals. Educational and Psychological Measurement 70: 717–31. [Google Scholar] [CrossRef]
- Cohen, Jacob. 1992. A Power Primer. Psychological Bulletin 112: 155–59. [Google Scholar] [CrossRef]
- Cooper, Lynn A., and Roger N. Shepard. 1973. Chronometric Studies of the Rotation of Mental Images. In Visual Information Processing. Edited by William G. Chase. New York: Academic Press, pp. 75–176. [Google Scholar] [CrossRef]
- Daniel, Robert C., and Susan E. Embretson. 2010. Designing Cognitive Complexity in Mathematical Prob-lem-Solving Items. Applied Psychological Measurement 34: 348–64. [Google Scholar] [CrossRef]
- de Boeck, Paul, and Mark Wilson, eds. 2004. Explanatory Item Response Models: A Generalized Linear and Nonlinear Approach. Berlin: Springer. [Google Scholar]
- Defense Manpower Data Center. 2006. CAT-ASVAB Forms 1 & 2 (Technical Bulletin No. 1). Seaside: Defense Manpower Data Center. [Google Scholar]
- Eliot, John, and Ian Macfarlane Smith. 1983. An International Directory of Spatial Tests. Berkshire: NFER-Nelson. [Google Scholar]
- Embretson, Susan E. 2002. Generating Abstract Reasoning Items with Cognitive Theory. In Item Generation for Test Development. Edited by Sidney H. Irvine and Patrick C. Kyllonen. Mahwah: Lawrence Erlbaum Associates, Inc., pp. 251–82. [Google Scholar]
- Embretson, Susan E. 2007. Construct Validity: A Universal Validity System or Just Another Test Evalua-tion Procedure? Educational Researcher 36: 449–55. [Google Scholar] [CrossRef]
- Embretson, Susan E. 2016. Understanding Examinees’ Responses to Items: Implications for Measurement. Educational Measurement Issues and Practice 35: 6–22. [Google Scholar] [CrossRef]
- Embretson, Susan, and Joanna Gorin. 2001. Improving Construct Validity with Cognitive Psychology Principles. Journal of Educational Measurement 38: 343–68. [Google Scholar] [CrossRef]
- Engelhard, George, Jr., and Jue Wang. 2021. Rasch Models for Solving Measurement Problems. Thousand Oaks: SAGE Publications, Inc. Available online: https://us.sagepub.com/en-us/nam/rasch-models-for-solving-measurement-problems/book267292 (accessed on 30 January 2022).
- Fischer, Gerhard H. 1973. The Linear Logistic Test Model as an Instrument in Educational Research. Acta Psychologica 37: 359–74. [Google Scholar] [CrossRef]
- Fischer, Gerhard H. 1997. Unidimensional Linear Logistic Rasch Models. In Handbook of Modern Item Response Theory. Edited by Wim J. Linden and Ronald K. Hambleton. New York: Springer, pp. 225–43. [Google Scholar] [CrossRef]
- Fischer, Gerhard H. 2005. Linear Logistic Test Models. In Encyclopedia of Social Measurement. Edited by Kimberly Kempf-Leonard. Amsterdam: Elsevier, vol. 2, pp. 505–14. [Google Scholar]
- Fischer, Gerhard H., and Anton K. Formann. 1982. Some Applications of Logistic Latent Trait Models with Linear Constraints on the Parameters. Applied Psychological Measurement 6: 397–416. [Google Scholar] [CrossRef] [Green Version]
- Gagnier, Kristin M., and Kelly R. Fisher. 2020. Unpacking the Black Box of Translation: A Framework for Infusing Spatial Thinking into Curricula. Cognitive Research: Principles and Implications 5: 29. [Google Scholar] [CrossRef]
- Gierl, Mark J., Hollis Lai, and Vasily Tanygin. 2021. Cognitive Model Development. In Advanced Methods in Automatic Item Generation. London: Routledge. [Google Scholar]
- Gilligan-Lee, Katie A., Zachary C. K. Hawes, and Kelly S. Mix. 2022. Spatial Thinking as the Missing Piece in Mathematics Curricula. Npj Science of Learning 7: 10. [Google Scholar] [CrossRef]
- Gorin, Joanna S., and Susan E. Embretson. 2006. Item Difficulty Modeling of Paragraph Comprehension Items. Applied Psychological Measurement 30: 394–411. [Google Scholar] [CrossRef]
- Green, Kathy E., and Richard M. Smith. 1987. A Comparison of Two Methods of Decomposing Item Dif-ficulties. Journal of Educational Statistics 12: 369–81. [Google Scholar] [CrossRef]
- Ivie, Jennifer L., and Susan E. Embretson. 2010. Cognitive Process Modeling of Spatial Ability: The Assembling Objects Task. Intelligence 38: 324–35. [Google Scholar] [CrossRef]
- Linacre, John M. 1989. Many-Facet Rasch Measurement. Chicago: Mesa Press. [Google Scholar]
- Linacre, John M. 1994. Sample Size and Item Calibration Stability. Rasch Measurement Transactions 7: 328. [Google Scholar]
- Loe, Bao Sheng, Luning Sun, Filip Simonfy, and Philipp Doebler. 2018. Evaluating an Automated Number Series Item Generator Using Linear Logistic Test Models. Journal of Intelligence 6: 20. [Google Scholar] [CrossRef] [Green Version]
- Lohman, David F. 1979. Spatial Ability: A Review and Reanalysis of the Correlational Literature. Technical Report No. 8. Stanford: School of Education, Stanford University. [Google Scholar]
- Mair, Patrick, Reinhold Hatzinger, and Marco J. Maier. 2021. eRm: Extended Rasch Modeling, Version 1.0-2; Available online: https://cran.r-project.org/package=eRm (accessed on 26 March 2023).
- Mumaw, Randall J., and James W. Pellegrino. 1984. Individual Differences in Complex Spatial Processing. Journal of Educational Psychology 76: 920–39. [Google Scholar] [CrossRef]
- Newcombe, Nora S. 2010. Picture This: Increasing Math and Science Learning by Improving Spatial Thinking. American Educator 34: 29–35. Available online: https://files.eric.ed.gov/fulltext/EJ889152.pdf (accessed on 26 March 2023).
- Next Generation Science Standards. 2013. Next Generation Science Standards: For States, by States. Washington, DC: National Academies Press. [Google Scholar]
- Pellegrino, James W., Richard Mumaw, and Valerie J. Shute. 1985. Analyses of Spatial Aptitude and Ex-pertise. In Test Design: Developments in Psychology and Psychometrics. Edited by Susan E. Em-bretson. Orlando: Academic Press, pp. 45–76. [Google Scholar] [CrossRef]
- Quasha, William H., and Rensis Likert. 1937. The Revised Minnesota Paper Form Board Test. Journal of Educational Psychology 28: 197–204. [Google Scholar] [CrossRef]
- R Core Team. 2022. R: A Language and Environment for Statistical Computing. Vienna: R Foundation for Statistical Computing. Available online: https://www.R-project.org/ (accessed on 26 March 2023).
- Rasch, Georg. 1960. Probabilistic Models for Some Intelligence and Achievement Tests, exp. ed. Chicago: University of Chicago Press. [Google Scholar]
- Revelle, William. 2023. psych: Procedures for Psychological, Psychometric, and Personality Research. Version 2.3.3. Evanston: Northwestern University. Available online: https://CRAN.R-project.org/package=psych (accessed on 26 March 2023).
- Rijmen, Frank, and Paul de Boeck. 2002. The Random Weights Linear Logistic Test Model. Applied Psychological Measurement 26: 271–85. [Google Scholar] [CrossRef]
- Rupp, André A., Jonathan Templin, and Robert A. Henson. 2010. Diagnostic Measurement: Theory, Methods, and Applications. New York: Guilford Press. [Google Scholar]
- Schneider, W. Joel, and Kevin S. McGrew. 2018. The Cattell–Horn–Carroll Theory of Cognitive Abilities. In Contemporary Intellectual Assessment: Theories, Tests, and Issues. Edited by Dawn P. Flanagan and Erin M. McDonough. New York: The Guilford Press, pp. 73–163. [Google Scholar]
- Smith, Richard M. 2004. Fit Analysis in Latent Trait Models. In Introduction to Rasch Measurement. Edited by Everett. V. Smith and Richard. M. Smith. Maple Grove: JAM Press, pp. 73–92. [Google Scholar]
- Wai, Jonathan, and Joni M. Lakin. 2020. Finding the Missing Einsteins: Expanding the Breadth of Cogni-tive and Noncognitive Measures Used in Academic Services. Contemporary Educational Psychology 63: 101920. [Google Scholar] [CrossRef]
- Wai, Jonathan, David Lubinski, and Camilla P. Benbow. 2009. Spatial Ability for STEM Domains: Aligning over 50 Years of Cumulative Psychological Knowledge Solidifies Its Importance. Journal of Educational Psychology 101: 817–35. [Google Scholar] [CrossRef]
- Wright, Benjamin D., and Geofferey N. Masters. 1982. Rating Scale Analysis: Rasch Measurement. Chicago: MESA Press. [Google Scholar]
- Wu, Margaret, and Richard J. Adams. 2013. Properties of Rasch Residual Fit Statistics. Journal of Applied Measurement 14: 339–55. [Google Scholar]
- Young, Christopher J., Susan C. Levine, and Kelly S. Mix. 2018. The Connection Between Spatial and Mathematical Ability Across Development. Frontiers in Psychology 9: 755. [Google Scholar] [CrossRef]
- Zumbo, Bruno D., and Anita M. Hubley, eds. 2017. Understanding and Investigating Response Processes in Validation Research. Cham: Springer. [Google Scholar] [CrossRef]

**Figure 1.**Example of an object assembly item. The pieces to the left of the line are the item stem, and the figures (

**A**–

**D**) are the options.

**Figure 2.**Plot of standardized differences (z) in subgroup-specific item difficulty estimates between subgroups.

Item Characteristics | Description |
---|---|

Number of pieces (Npieces) | The number of pieces in the stem |

Total edges (Tedges) | The total number of edges across pieces in the stem |

Maximum edges (Medges) | The maximum number of edges on any one piece in the stem |

Curved pieces (Cpieces) | Pieces in the stem containing at least one curved edge |

Pieces with labels (Lpieces) | All pieces in the stem with clear labels (square, triangle, [pie] slice) |

Regular-shape solution (RSS) | The key has a standard shape (circle, equilateral triangle, right triangle, or square) |

Displaced pieces (Dpieces) | The number of pieces in the stem that were moved to a different location in the key |

Rotated pieces (Rpieces) | The number of pieces in the key that had to be rotated from the key to the stem to reach the correct answer |

Easily excluded distractors (EED) | The number of distractors with a different number of pieces or obviously different shapes from the stem |

Item | Npieces | Lpieces | Tedges | Medges | Cpieces | EED | RSS | Dpieces | Rpieces |
---|---|---|---|---|---|---|---|---|---|

1 | 0 | 0 | 0 | 1 | 0 | 1 | 1 | 0 | 0 |

2 | 0 | 0 | 0 | 1 | 0 | 1 | 1 | 0 | 0 |

3 | 0 | 0 | 0 | 0 | 1 | 1 | 1 | 0 | 0 |

4 | 0 | 0 | 1 | 0 | 0 | 1 | 1 | 0 | 0 |

5 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 0 |

6 | 1 | 1 | 1 | 1 | 0 | 1 | 1 | 0 | 0 |

7 | 1 | 0 | 1 | 1 | 1 | 1 | 1 | 0 | 0 |

8 | 0 | 0 | 1 | 1 | 0 | 1 | 1 | 1 | 1 |

9 | 1 | 0 | 1 | 0 | 1 | 1 | 1 | 0 | 0 |

10 | 0 | 0 | 1 | 1 | 0 | 1 | 0 | 0 | 0 |

11 | 1 | 1 | 1 | 0 | 0 | 0 | 1 | 0 | 1 |

12 | 1 | 1 | 1 | 0 | 0 | 0 | 1 | 0 | 0 |

13 | 1 | 0 | 1 | 0 | 1 | 1 | 1 | 1 | 1 |

14 | 1 | 1 | 1 | 0 | 1 | 0 | 1 | 0 | 0 |

15 | 1 | 1 | 1 | 0 | 1 | 0 | 1 | 0 | 1 |

Item | n | Proportion Correct (Mean) | SD | Corrected Item–Total Correlation |
---|---|---|---|---|

1 | 123 | 0.87 | 0.34 | 0.51 |

2 | 123 | 0.82 | 0.39 | 0.60 |

3 | 123 | 0.85 | 0.36 | 0.48 |

4 | 123 | 0.77 | 0.42 | 0.59 |

5 | 123 | 0.63 | 0.48 | 0.71 |

6 | 123 | 0.69 | 0.46 | 0.58 |

7 | 123 | 0.61 | 0.49 | 0.49 |

8 | 123 | 0.43 | 0.50 | 0.49 |

9 | 170 | 0.68 | 0.47 | 0.67 |

10 | 170 | 0.65 | 0.48 | 0.52 |

11 | 169 | 0.64 | 0.48 | 0.35 |

12 | 170 | 0.46 | 0.50 | 0.53 |

13 | 169 | 0.52 | 0.50 | 0.45 |

14 | 168 | 0.50 | 0.50 | 0.53 |

15 | 170 | 0.43 | 0.50 | 0.54 |

Item | $\mathit{\delta}$ | SE | Lower CI | Upper CI | Outfit MSE | Infit MSE | Outfit z | Infit z |
---|---|---|---|---|---|---|---|---|

1 | −1.92 | 0.29 | −1.35 | −2.49 | 0.36 | 0.73 | −1.28 | −1.70 |

2 | −1.43 | 0.27 | −0.92 | −1.95 | 0.35 | 0.65 | −1.77 | −2.80 * |

3 | −1.66 | 0.28 | −1.12 | −2.21 | 0.85 | 0.81 | −0.15 | −1.24 |

4 | −1.02 | 0.25 | −0.53 | −1.51 | 0.52 | 0.79 | −1.46 | −1.70 |

5 | −0.01 | 0.23 | 0.44 | −0.46 | 0.52 | 0.70 | −2.44 | −2.70 |

6 | −0.41 | 0.24 | 0.05 | −0.87 | 0.77 | 0.90 | −0.83 | −0.82 |

7 | 0.16 | 0.23 | 0.61 | −0.29 | 1.26 | 1.15 | 1.16 | 1.16 |

8 | 1.35 | 0.23 | 1.81 | 0.90 | 1.21 | 1.10 | 0.86 | 0.80 |

9 | −0.29 | 0.23 | 0.17 | −0.75 | 0.62 | 0.79 | −1.62 | −1.85 |

10 | 0.16 | 0.23 | 0.61 | −0.29 | 1.18 | 1.15 | 0.85 | 1.16 |

11 | −0.29 | 0.23 | 0.17 | −0.75 | 1.32 | 1.36 | 1.20 | 2.70 |

12 | 1.52 | 0.23 | 1.97 | 1.06 | 1.13 | 1.01 | 0.56 | 0.11 |

13 | 0.65 | 0.23 | 1.10 | 0.21 | 1.26 | 1.20 | 1.22 | 1.51 |

14 | 1.41 | 0.23 | 1.86 | 0.96 | 1.15 | 1.05 | 0.62 | 0.44 |

15 | 1.79 | 0.23 | 2.24 | 1.33 | 1.04 | 1.01 | 0.22 | 0.16 |

M | 0.00 | 0.24 | 0.47 | −0.47 | 0.90 | 0.96 | −0.19 | −0.32 |

SD | 1.19 | 0.02 | 1.15 | 1.22 | 0.35 | 0.21 | 1.26 | 1.65 |

Item | ${\mathit{d}}_{1}$ | ${\mathit{se}}_{1}$ | ${\mathit{d}}_{2}$ | ${\mathit{se}}_{2}$ | z | p |
---|---|---|---|---|---|---|

3 | −2.81 | 0.96 | −1.61 | 0.29 | −1.20 | 0.23 |

4 | −2.81 | 0.96 | −0.86 | 0.28 | −1.96 | 0.05 * |

5 | −0.43 | 0.40 | 0.01 | 0.30 | −0.86 | 0.39 |

6 | −1.03 | 0.48 | −0.36 | 0.29 | −1.19 | 0.23 |

7 | 0.49 | 0.33 | −0.36 | 0.29 | 1.92 | 0.05 * |

8 | 1.46 | 0.30 | 0.82 | 0.36 | 1.38 | 0.17 |

9 | −0.80 | 0.45 | −0.27 | 0.29 | −0.99 | 0.32 |

10 | −0.12 | 0.37 | 0.11 | 0.31 | −0.47 | 0.64 |

11 | 0.26 | 0.35 | −0.86 | 0.28 | 2.53 | 0.01 * |

12 | 1.69 | 0.30 | 0.82 | 0.36 | 1.87 | 0.06 |

13 | 0.88 | 0.31 | 0.11 | 0.31 | 1.75 | 0.08 |

14 | 1.46 | 0.30 | 0.97 | 0.37 | 1.04 | 0.30 |

15 | 1.76 | 0.30 | 1.50 | 0.44 | 0.51 | 0.61 |

Item Characteristic | $\mathit{\eta}$ | SE | Lower CI | Upper CI |
---|---|---|---|---|

Npieces | −0.85 | 0.29 | −1.41 | −0.29 |

Lpieces | 1.20 | 0.25 | 0.71 | 1.68 |

Tedges | 1.54 | 0.28 | 0.99 | 2.08 |

Medges | 0.64 | 0.19 | 0.27 | 1.01 |

Cpieces | 1.02 | 0.20 | 0.62 | 1.41 |

EED | −1.19 | 0.23 | −1.65 | −0.74 |

RSS | −0.39 | 0.32 | −1.02 | 0.25 |

Dpieces | 1.89 | 0.32 | 1.25 | 2.52 |

Rpieces | −0.44 | 0.22 | −0.88 | −0.01 |

Item | ${\mathit{\delta}}^{\prime}$ | SE | Lower CI | Upper CI |
---|---|---|---|---|

1 | −0.94 | 0.42 | −1.77 | −0.11 |

2 | −0.94 | 0.42 | −1.77 | −0.11 |

3 | −0.57 | 0.45 | −1.45 | 0.32 |

4 | −0.05 | 0.57 | −1.16 | 1.07 |

5 | 1.32 | 0.44 | 0.45 | 2.18 |

6 | 0.94 | 0.55 | −0.14 | 2.02 |

7 | 0.76 | 0.56 | −0.33 | 1.86 |

8 | 2.04 | 0.55 | 0.97 | 3.11 |

9 | 0.12 | 0.50 | −0.87 | 1.11 |

10 | 0.98 | 0.39 | 0.22 | 1.74 |

11 | 1.05 | 0.46 | 0.16 | 1.95 |

12 | 1.50 | 0.42 | 0.68 | 2.31 |

13 | 1.57 | 0.45 | 0.68 | 2.45 |

14 | 2.51 | 0.50 | 1.54 | 3.48 |

15 | 2.07 | 0.52 | 1.05 | 3.10 |

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |

© 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Shi, Q.; Wind, S.A.; Lakin, J.M.
Exploring the Influence of Item Characteristics in a Spatial Reasoning Task. *J. Intell.* **2023**, *11*, 152.
https://doi.org/10.3390/jintelligence11080152

**AMA Style**

Shi Q, Wind SA, Lakin JM.
Exploring the Influence of Item Characteristics in a Spatial Reasoning Task. *Journal of Intelligence*. 2023; 11(8):152.
https://doi.org/10.3390/jintelligence11080152

**Chicago/Turabian Style**

Shi, Qingzhou, Stefanie A. Wind, and Joni M. Lakin.
2023. "Exploring the Influence of Item Characteristics in a Spatial Reasoning Task" *Journal of Intelligence* 11, no. 8: 152.
https://doi.org/10.3390/jintelligence11080152