Next Article in Journal
Using the 16PF to Test the Differentiation of Personality by Intelligence Hypothesis
Next Article in Special Issue
Diagnosing a 12-Item Dataset of Raven Matrices: With Dexter
Previous Article in Journal
Tracking with (Un)Certainty
Previous Article in Special Issue
Analysing Standard Progressive Matrices (SPM-LS) with Bayesian Item Response Models
Article

How Much g Is in the Distractor? Re-Thinking Item-Analysis of Multiple-Choice Items

Institute of Psychology in Education, University of Münster, 48149 Münster, Germany
*
Author to whom correspondence should be addressed.
Received: 31 January 2020 / Revised: 27 February 2020 / Accepted: 3 March 2020 / Published: 9 March 2020
(This article belongs to the Special Issue Analysis of an Intelligence Dataset)
Distractors might display discriminatory power with respect to the construct of interest (e.g., intelligence), which was shown in recent applications of nested logit models to the short-form of Raven’s progressive matrices and other reasoning tests. In this vein, a simulation study was carried out to examine two effect size measures (i.e., a variant of Cohen’s ω and the canonical correlation RCC) for their potential to detect distractors with ability-related discriminatory power. The simulation design was adopted to item selection scenarios relying on rather small sample sizes (e.g., N = 100 or N = 200). Both suggested effect size measures (Cohen’s ω only when based on two ability groups) yielded acceptable to conservative type-I-error rates, whereas, the canonical correlation outperformed Cohen’s ω in terms of empirical power. The simulation results further suggest that an effect size threshold of 0.30 is more appropriate as compared to more lenient (0.10) or stricter thresholds (0.50). The suggested item-analysis procedure is illustrated with an analysis of twelve Raven’s progressive matrices items in a sample of N = 499 participants. Finally, strategies for item selection for cognitive ability tests with the goal of scaling by means of nested logit models are discussed. View Full-Text
Keywords: Raven’s progressive matrices; intelligence; distractors; item analysis Raven’s progressive matrices; intelligence; distractors; item analysis
Show Figures

Figure 1

MDPI and ACS Style

Forthmann, B.; Förster, N.; Schütze, B.; Hebbecker, K.; Flessner, J.; Peters, M.T.; Souvignier, E. How Much g Is in the Distractor? Re-Thinking Item-Analysis of Multiple-Choice Items. J. Intell. 2020, 8, 11. https://doi.org/10.3390/jintelligence8010011

AMA Style

Forthmann B, Förster N, Schütze B, Hebbecker K, Flessner J, Peters MT, Souvignier E. How Much g Is in the Distractor? Re-Thinking Item-Analysis of Multiple-Choice Items. Journal of Intelligence. 2020; 8(1):11. https://doi.org/10.3390/jintelligence8010011

Chicago/Turabian Style

Forthmann, Boris, Natalie Förster, Birgit Schütze, Karin Hebbecker, Janis Flessner, Martin T. Peters, and Elmar Souvignier. 2020. "How Much g Is in the Distractor? Re-Thinking Item-Analysis of Multiple-Choice Items" Journal of Intelligence 8, no. 1: 11. https://doi.org/10.3390/jintelligence8010011

Find Other Styles
Note that from the first issue of 2016, MDPI journals use article numbers instead of page numbers. See further details here.

Article Access Map by Country/Region

1
Back to TopTop