# Examining Position Effects on Students’ Ability and Test-Taking Speed in the TIMSS 2019 Problem-Solving and Inquiry Tasks: A Structural Equation Modeling Approach

^{1}

^{2}

^{3}

^{*}

## Abstract

**:**

## 1. Introduction

## 2. Theoretical Framework

## 3. Methods

#### 3.1. Data Source

#### 3.2. Measures

_{min}is the minimum response time (out of all students) on a given screen, and X

_{max}is the maximum response time on that screen. This transformation normalizes the data and bounds the data into the range of [0, 1]. Following this step, z scores were computed. In this study, screen response times associated with a z-score that was either larger than 3 or smaller than −3 were removed. This application of Cousineau and Chartier’s [37] method removed between 0.4% and 1.7% of the response time data for each screen.

#### 3.3. Data Analysis

Block | Task | Item | M | SD | Max Score | % Data Present |
---|---|---|---|---|---|---|

M1 | Penguins | MA01 | 0.29 | 0.45 | 1 | 99.3% |

MA02A | 0.63 | 0.48 | 1 | 99.2% | ||

MA02B | 0.44 | 0.50 | 1 | 99.0% | ||

MA02C | 0.55 | 0.50 | 1 | 99.0% | ||

MA03A | 0.41 | 0.49 | 1 | 98.9% | ||

MA03B | 0.55 | 0.50 | 1 | 98.7% | ||

MA04A | 0.57 | 0.80 | 2 | 98.6% | ||

MA04B | 0.40 | 0.49 | 1 | 98.2% | ||

MA05A | 0.33 | 0.47 | 1 | 98.0% | ||

MA05B | 0.55 | 0.50 | 1 | 97.5% | ||

MA06A | 0.49 | 0.50 | 1 | 97.0% | ||

MA06B | 0.20 | 0.40 | 1 | 96.5% | ||

Robots-4 | MR01A | 0.69 | 0.46 | 1 | 96.0% | |

MR01B | 0.60 | 0.49 | 1 | 95.1% | ||

MR02A | 0.28 | 0.45 | 1 | 94.8% | ||

MR02B | 0.44 | 0.50 | 1 | 88.2% | ||

MR03 | 0.32 | 0.46 | 1 | 87.6% | ||

MR04 | 0.56 | 0.84 | 2 | 83.8% | ||

M2 | School Party | MP01A | 0.38 | 0.49 | 1 | 97.1% |

MP01B | 0.43 | 0.71 | 2 | 96.1% | ||

MP02 | 0.46 | 0.50 | 1 | 95.6% | ||

MP03 | 1.26 | 0.90 | 2 | 95.2% | ||

MP04 | 0.39 | 0.75 | 2 | 94.5% | ||

MP05A | 0.62 | 0.48 | 1 | 92.0% | ||

MP05B | 0.13 | 0.34 | 1 | 91.2% | ||

MP06A | 0.13 | 0.34 | 1 | 89.0% | ||

MP06B | 0.21 | 0.41 | 1 | 87.5% | ||

MP07A | 0.14 | 0.35 | 1 | 84.0% | ||

MP07B | 0.10 | 0.30 | 1 | 80.6% | ||

S1 | Farm Investigation | SF01 | 0.70 | 0.84 | 2 | 97.6% |

SF02 | 0.44 | 0.50 | 1 | 95.6% | ||

SF03 | 0.53 | 0.50 | 1 | 94.5% | ||

SF04 | 0.51 | 0.50 | 1 | 92.7% | ||

SF05 | 0.51 | 0.50 | 1 | 90.0% | ||

SF06 | 0.61 | 0.49 | 1 | 88.3% | ||

SF07A | 0.58 | 0.49 | 1 | 86.2% | ||

SF07B | 0.11 | 0.32 | 1 | 86.9% | ||

SF08 | 0.64 | 0.48 | 1 | 85.4% | ||

SF09 | 0.72 | 0.79 | 2 | 84.1% | ||

S2 | Sugar Experiment | SS01 | 0.56 | 0.50 | 1 | 99.3% |

SS02 | 0.41 | 0.49 | 1 | 99.2% | ||

SS03 | 0.66 | 0.88 | 2 | 98.9% | ||

SS04 | 0.73 | 0.80 | 2 | 96.6% | ||

SS05 | 0.68 | 0.74 | 2 | 96.0% | ||

SS07 | 0.82 | 0.74 | 2 | 91.2% | ||

SS08 | 0.39 | 0.49 | 1 | 90.8% | ||

SS09 | 0.47 | 0.50 | 1 | 88.4% |

Block | Task | Item | M | SD | Min | Max | % Data Present | % Outliers Removed |
---|---|---|---|---|---|---|---|---|

M1 | Penguins | MA01_S | 56.20 | 33.38 | 0.26 | 207.86 | 97.6% | 1.4% |

MA02_S | 102.92 | 50.02 | 3.54 | 326.07 | 97.2% | 1.7% | ||

MA03_S | 67.86 | 36.39 | 0.72 | 235.88 | 97.3% | 1.4% | ||

MA04_S | 119.52 | 72.04 | 0.12 | 443.71 | 97.5% | 0.9% | ||

MA05_S | 115.66 | 69.22 | 0.09 | 434.35 | 96.9% | 1.1% | ||

MA06_S | 102.87 | 70.69 | 0.14 | 419.68 | 96.2% | 1.1% | ||

Robots-4 | MR01_S | 76.39 | 37.51 | 2.57 | 242.22 | 94.8% | 1.4% | |

MR01_S | 133.97 | 82.72 | 0.13 | 490.76 | 94.8% | 0.6% | ||

MR02_S | 103.25 | 62.71 | 0.10 | 384.53 | 93.0% | 0.7% | ||

MR02_S | 164.79 | 87.76 | 1.18 | 555.92 | 90.7% | 1.2% | ||

M2 | School Party | MP01_S | 134.21 | 81.35 | 0.16 | 499.81 | 95.8% | 1.0% |

MP02_S | 36.56 | 22.69 | 0.09 | 144.95 | 94.2% | 1.3% | ||

MP03_S | 52.44 | 31.75 | 0.09 | 202.00 | 93.6% | 1.4% | ||

MP04_S | 138.21 | 84.34 | 0.18 | 516.02 | 93.2% | 1.0% | ||

MP05_S | 106.83 | 59.96 | 0.67 | 372.77 | 90.9% | 1.3% | ||

MP06_S | 136.99 | 88.46 | 0.11 | 531.50 | 89.8% | 0.4% | ||

MP07_S | 121.48 | 80.91 | 0.13 | 482.05 | 86.8% | 0.5% | ||

S1 | Farm Investigation | SF01_S | 117.76 | 68.37 | 0.50 | 421.01 | 96.1% | 1.3% |

SF02_S | 109.72 | 64.82 | 0.22 | 396.96 | 95.3% | 1.0% | ||

SF03_S | 113.35 | 52.20 | 5.19 | 340.71 | 93.3% | 1.7% | ||

SF04_S | 106.81 | 49.84 | 4.19 | 324.96 | 91.8% | 1.4% | ||

SF05_S | 66.16 | 41.68 | 0.07 | 251.27 | 89.6% | 1.0% | ||

SF06_S | 29.61 | 14.46 | 0.94 | 93.53 | 87.8% | 1.1% | ||

SF07_S | 100.66 | 61.59 | 0.08 | 374.66 | 87.7% | 0.5% | ||

SF08_S | 35.50 | 18.81 | 0.38 | 119.59 | 85.5% | 0.8% | ||

SF09_S | 90.91 | 58.13 | 0.08 | 349.53 | 84.8% | 0.6% | ||

S2 | Sugar Experiment | SS01_S | 102.39 | 69.85 | 0.09 | 421.17 | 97.8% | 1.3% |

SS02_S | 92.10 | 57.67 | 0.13 | 348.79 | 97.8% | 1.0% | ||

SS03_S | 156.22 | 94.97 | 0.17 | 577.84 | 97.5% | 1.0% | ||

SS04_S | 143.47 | 86.48 | 0.10 | 527.48 | 96.9% | 0.7% | ||

SS05_S | 98.37 | 65.93 | 0.10 | 399.17 | 95.8% | 0.8% | ||

SS07_S | 113.12 | 64.87 | 0.22 | 400.78 | 93.8% | 0.6% | ||

SS08_S | 59.79 | 40.42 | 0.06 | 241.21 | 92.1% | 0.6% | ||

SS09_S | 38.93 | 28.87 | 0.07 | 174.02 | 90.7% | 1.0% |

## 4. Results

#### 4.1. Descriptive and Correlation Analyses

#### 4.2. Model Specification and Fit

^{2}) test, which could be due to the large sample size in this study.

^{2}. Thus, the measurement model for ability was retained as such, which also fits well with the theory. TIMSS uses item response theory for achievement scaling, which assumes local independence of item responses given ability. Since the items used in this study were those included in the eTIMSS achievement scaling, they can be assumed to be high-quality items, and there is thus no basis for correlating errors between any pair of items.

#### 4.3. Booklet Effect on Ability

#### 4.4. Booklet Effect on Speed

## 5. Discussion

#### Limitations and Future Research

## Author Contributions

## Funding

## Institutional Review Board Statement

## Informed Consent Statement

## Data Availability Statement

## Conflicts of Interest

## References

- Mullis, I.V.S.; Martin, M.O.; Foy, P.; Kelly, D.L.; Fishbein, B. TIMSS 2019 International Results in Mathematics and Science; TIMSS & PIRLS International Study Centre, Boston College: Chestnut Hill, MA, USA, 2020. [Google Scholar]
- OECD. PISA 2015 Technical Report; OECD: Paris, France, 2017. [Google Scholar]
- Mullis, I.V.S.; Martin, M.O.; Fishbein, B.; Foy, P.; Moncaleano, S. Findings from the TIMSS 2019 Problem Solving and Inquiry Tasks; TIMSS & PIRLS International Study Centre, Boston College: Chestnut Hill, MA, USA, 2021. [Google Scholar]
- Fishbein, B.; Foy, P.; Yin, L. TIMSS 2019 User Guide for the International Database, 2nd ed.; TIMSS & PIRLS International Study Centre, Boston College: Chestnut Hill, MA, USA, 2021. [Google Scholar]
- Albano, A.D. Multilevel Modeling of Item Position Effects: Modeling Item Position Effects. J. Educ. Meas.
**2013**, 50, 408–426. [Google Scholar] [CrossRef] - Bulut, O.; Quo, Q.; Gierl, M.J. A Structural Equation Modeling Approach for Examining Position Effects in Large-Scale Assessments. Large-Scale Assess. Educ.
**2017**, 5, 716. [Google Scholar] [CrossRef] - Hahne, J. Analyzing Position Effects within Reasoning Items Using the LLTM for Structurally Incomplete Data. Psychol. Sci. Q.
**2008**, 50, 379–390. [Google Scholar] - Hohensinn, C.; Kubinger, K.D.; Reif, M.; Schleicher, E.; Khorramdel, L. Analysing Item Position Effects Due to Test Booklet Design within Large-Scale Assessment. Educ. Res. Eval.
**2011**, 17, 497–509. [Google Scholar] [CrossRef] - Nagy, G.; Nagengast, B.; Becker, M.; Rose, N.; Frey, A. Item Position Effects in a Reading Comprehension Test: An IRT Study of Individual Differences and Individual Correlates. Psychol. Test Assess. Model.
**2018**, 60, 165–187. [Google Scholar] - Christiansen, A.; Janssen, R. Item Position Effects in Listening but Not in Reading in the European Survey of Language Competences. Educ. Assess. Eval. Acc.
**2020**, 33, 49–69. [Google Scholar] [CrossRef] - Demirkol, S.; Kelecioğlu, H. Investigating the Effect of Item Position on Person and Item Parameters: PISA 2015 Turkey Sample. J. Meas. Eval. Educ. Psychol.
**2022**, 13, 69–85. [Google Scholar] [CrossRef] - Ong, T.Q.; Pastor, D.A. Uncovering the Complexity of Item Position Effects in a Low-Stakes Testing Context. Appl. Psychol. Meas.
**2022**, 46, 571–588. [Google Scholar] [CrossRef] [PubMed] - Debeer, D.; Janssen, R. Modeling Item-Position Effects Within an IRT Framework: Modeling Item-Position Effects. J. Educ. Meas.
**2013**, 50, 164–185. [Google Scholar] [CrossRef] - Sideridis, G.; Hamed, H.; Jaffari, F. The Item Position Effects in International Examinations: The Roles of Gender. Front. Psychol.
**2023**, 14, 1220384. [Google Scholar] [CrossRef] - Lindner, M.A.; Lüdtke, O.; Nagy, G. The Onset of Rapid-Guessing Behavior Over the Course of Testing Time: A Matter of Motivation and Cognitive Resources. Front. Psychol.
**2019**, 10, 1533. [Google Scholar] [CrossRef] - Weirich, S.; Hecht, M.; Penk, C.; Roppelt, A.; Böhme, K. Item Position Effects Are Moderated by Changes in Test-Taking Effort. Appl. Psychol. Meas.
**2017**, 41, 115–129. [Google Scholar] [CrossRef] [PubMed] - Tan, B. Response Time as a Predictor of Test Performance: Assessing the Value of Examinees’ Response Time Profiles. Master’s Thesis, University of Alberta, Edmonton, AB, Canada, 2023. [Google Scholar]
- Fox, J.-P.; Marianti, S. Joint Modeling of Ability and Differential Speed Using Responses and Response Times. Multivar. Behav. Res.
**2016**, 51, 540–553. [Google Scholar] [CrossRef] [PubMed] - Goldhammer, F. Measuring Ability, Speed, or Both? Challenges, Psychometric Solutions, and What Can Be Gained From Experimental Control. Meas. Interdiscip. Res. Perspect.
**2015**, 13, 133–164. [Google Scholar] [CrossRef] [PubMed] - Wise, S.L. Response Time as an Indicator of Test Taker Speed: Assumptions Meet Reality. Meas. Interdiscip. Res. Perspect.
**2015**, 13, 186–188. [Google Scholar] [CrossRef] - Hecht, M.; Weirich, S.; Siegle, T.; Frey, A. Effects of Design Properties on Parameter Estimation in Large-Scale Assessments. Educ. Psychol. Meas.
**2015**, 75, 1021–1044. [Google Scholar] [CrossRef] - Fishbein, B.; Foy, P. Scaling the TIMSS 2019 Problem Solving and Inquiry Data. In Methods and Procedures: TIMSS 2019 Technical Report; Martin, M.O., von Davier, M., Mullis, I.V.S., Eds.; TIMSS & PIRLS International Study Centre, Boston College: Chestnut Hill, MA, USA, 2021; pp. 17.1–17.51. [Google Scholar]
- Kingston, N.M.; Dorans, N.J. Item Location Effects and Their Implications for IRT Equating and Adaptive Testing. Appl. Psychol. Meas.
**1984**, 8, 147–154. [Google Scholar] [CrossRef] - Wu, Q.; Debeer, D.; Buchholz, J.; Hartig, J.; Janssen, R. Predictors of Individual Performance Changes Related to Item Positions in PISA Assessments. Large-Scale Assess. Educ.
**2019**, 7, 5. [Google Scholar] [CrossRef] - Thurstone, L.L. Ability, Motivation, and Speed. Psychometrika
**1937**, 2, 249–254. [Google Scholar] [CrossRef] - van der Linden, W.J. A Hierarchical Framework for Modeling Speed and Accuracy on Test Items. Psychometrika
**2007**, 72, 287–308. [Google Scholar] [CrossRef] - Tijmstra, J.; Bolsinova, M. On the Importance of the Speed-Ability Trade-Off When Dealing with Not Reached Items. Front. Psychol.
**2018**, 9, 964. [Google Scholar] [CrossRef] [PubMed] - Goldhammer, F.; Naumann, J.; Stelter, A.; Tóth, K.; Rölke, H.; Klieme, E. The Time on Task Effect in Reading and Problem Solving Is Moderated by Task Difficulty and Skill: Insights from a Computer-Based Large-Scale Assessment. J. Educ. Psychol.
**2014**, 106, 608–626. [Google Scholar] [CrossRef] - van der Linden, W.J. A Lognormal Model for Response Times on Test Items. J. Educ. Behav. Stat.
**2006**, 31, 181–204. [Google Scholar] [CrossRef] - Klein Entink, R.H.; Fox, J.-P.; van der Linden, W.J. A Multivariate Multilevel Approach to the Modeling of Accuracy and Speed of Test Takers. Psychometrika
**2009**, 74, 21–48. [Google Scholar] [CrossRef] [PubMed] - Scherer, R.; Greiff, S.; Hautamäki, J. Exploring the Relation between Time on Task and Ability in Complex Problem Solving. Intelligence
**2015**, 48, 37–50. [Google Scholar] [CrossRef] - De Boeck, P.; Jeon, M. An Overview of Models for Response Times and Processes in Cognitive Tests. Front. Psychol.
**2019**, 10, 102. [Google Scholar] [CrossRef] [PubMed] - Li, F.; Cohen, A.; Shen, L. Investigating the Effect of Item Position in Computer-Based Tests. J. Educ. Meas.
**2012**, 49, 362–379. [Google Scholar] [CrossRef] - Yin, L.; Foy, P. TIMSS 2023 Assessment Design. In TIMSS 2023 Assessment Frameworks; Mullis, I.V.S., Martin, M.O., von Davier, M., Eds.; TIMSS & PIRLS International Study Centre, Boston College: Chestnut Hill, MA, USA, 2023; pp. 71–85. [Google Scholar]
- Vida, L.J.; Brinkhuis, M.J.S.; Bolsinova, M. Speeding up without loss of accuracy: Item position effects on performance in university exams. In Proceedings of the 14th International Conference on Educational Data Mining, Virtual, 29 June–2 July 2021. [Google Scholar]
- Martin, M.O.; Mullis, I.V.S.; Foy, P. TIMSS 2019 Assessment Design. In TIMSS 2019 Assessment Frameworks; Mullis, I.V.S., Martin, M.O., Eds.; TIMSS & PIRLS International Study Centre, Boston College: Chestnut Hill, MA, USA, 2017; pp. 81–91. [Google Scholar]
- Cousineau, D.; Chartier, S. Outliers Detection and Treatment: A Review. Int. J. Psychol. Res.
**2010**, 3, 58–67. [Google Scholar] [CrossRef] - Berger, A.; Kiefer, M. Comparison of Different Response Time Outlier Exclusion Methods: A Simulation Study. Front. Psychol.
**2021**, 12, 675558. [Google Scholar] [CrossRef] - Muthén, L.K.; Muthén, B.O. Mplus User’s Guide, 8th ed.; Muthén & Muthén: Los Angeles, CA, USA, 2017. [Google Scholar]
- Hu, L.; Bentler, P.M. Cutoff Criteria for Fit Indexes in Covariance Structure Analysis: Conventional Criteria versus New Alternatives. Struct. Equ. Model.
**1999**, 6, 1–55. [Google Scholar] [CrossRef] - Xia, Y.; Yang, Y. RMSEA, CFI, and TLI in Structural Equation Modeling with Ordered Categorical Data: The Story They Tell Depends on the Estimation Methods. Behav. Res. Methods
**2019**, 51, 409–428. [Google Scholar] [CrossRef] [PubMed] - Kline, R.B. Principles and Practice of Structural Equation Modeling, 5th ed.; The Guildford Press: New York, NY, USA, 2023. [Google Scholar]
- Bowerman, B.L.; O’Connell, R.T. Linear Statistical Models: An Applied Approach, 2nd ed.; Duxbury: Belmont, CA, USA, 1990. [Google Scholar]
- Comrey, A.L.; Lee, H.B. A First Course in Factor Analysis, 2nd ed.; Lawrence Erlbaum Associates, Inc.: Hillsdale, NJ, USA, 1992. [Google Scholar]
- Hambleton, R.K.; Traub, R.E. The Effects of Item Order on Test Performance and Stress. J. Exp. Educ.
**1974**, 43, 40–46. [Google Scholar] [CrossRef] - Ollennu, S.N.N.; Etsey, Y.K.A. The Impact of Item Position in Multiple-Choice Test on Student Performance at the Basic Education Certificate Examination (BECE) Level. Univers. J. Educ. Res.
**2015**, 3, 718–723. [Google Scholar] [CrossRef] - Sax, G.; Cromack, T.R. The Effects of Various Forms of Item Arrangements on Test Performance. J. Educ. Meas.
**1966**, 3, 309–311. [Google Scholar] [CrossRef] - Davis, J.; Ferdous, A. Using Item Difficulty and Item Position to Measure Test Fatigue; American Institutes for Research: Washington, DC, USA, 2005. [Google Scholar]
- Yoo, N. Item Position and Motivation Effects in Large-Scale Assessments. Ph.D. Thesis, Columbia University, New York, NY, USA, 2020. [Google Scholar]
- Rose, N.; Nagy, G.; Nagengast, B.; Frey, A.; Becker, M. Modeling Multiple Item Context Effects with Generalized Linear Mixed Models. Front. Psychol.
**2019**, 10, 248. [Google Scholar] [CrossRef] - Goldhammer, F.; Kroehne, U.; Hahnel, C.; De Boeck, P. Controlling Speed in Component Skills of Reading Improves the Explanation of Reading Comprehension. J. Educ. Psychol.
**2021**, 113, 861–878. [Google Scholar] [CrossRef] - OECD. PISA 2018 Technical Report; OECD: Paris, France, 2019. [Google Scholar]

**Figure 1.**Theoretical model for examining booklet effects on ability and test-taking speed. S

_{i}represents item-level scores, and RT

_{j}represents screen-level response times.

**Figure 2.**Final structural regression model for booklet effect on mathematics and science ability. Model fit indices: χ

^{2}(1078) = 33,827.461; p < .001; RMSEA = 0.033; CFI = 0.948; TLI = 0.945; SRMR = 0.040.

**Figure 3.**Final structural regression model for booklet effect on task speeds. Correlations between latent variables are not shown to minimize clutter. Model fit indices: χ

^{2}(545) = 22,557.918; p < .001; RMSEA = 0.038; CFI = 0.892; TLI = 0.882; SRMR = 0.039.

Booklet | Session 1 | Session 2 | ||
---|---|---|---|---|

Block Position 1 | Block Position 2 | Block Position 3 | Block Position 4 | |

Booklet 15 | M1 | M2 | S1 | S2 |

Booklet 16 | S2 | S1 | M2 | M1 |

Booklet | N of Students | N of Countries | Age | Gender |
---|---|---|---|---|

Booklet 15 | 13,829 | 36 | M: 10.15 y | F: 49.9% |

SD: 0.56 y | M: 50.1% | |||

Booklet 16 | 13,853 | 36 | M: 10.14 y | F: 48.8% |

SD: 0.57 y | M: 51.2% |

Model | χ^{2} | df | p | RMSEA [90% CI] | CFI | TLI | SRMR |
---|---|---|---|---|---|---|---|

Measurement Model | |||||||

1-factor CFA (ability) | 43,785.064 | 1034 | <.001 | 0.039 [0.038, 0.039] | 0.931 | 0.927 | 0.049 |

2-factor CFA (ability—math and science) | 31,660.815 | 1033 | <.001 | 0.033 [0.032, 0.033] | 0.950 | 0.948 | 0.042 |

1-factor CFA (speed) | 97,587.490 | 527 | <.001 | 0.082 [0.081, 0.082] | 0.462 | 0.427 | 0.113 |

2-factor CFA (speed—math and science) | 89,237.944 | 526 | <.001 | 0.078 [0.078, 0.079] | 0.508 | 0.475 | 0.115 |

5-factor CFA (speed—5 tasks) | 22,238.995 | 517 | <.001 | 0.039 [0.039, 0.039] | 0.880 | 0.869 | 0.041 |

5-factor CFA (speed—5 tasks, modified) | 20,438.493 | 516 | <.001 | 0.037 [0.037, 0.038] | 0.890 | 0.880 | 0.040 |

Structural Model | |||||||

Original theoretical model | 285,358.551 | 3237 | <.001 | 0.056 [0.056, 0.056] | 0.682 | 0.674 | 0.094 |

Booklet on 2-factor CFA (ability) | 33,827.461 | 1078 | <.001 | 0.033 [0.033, 0.033] | 0.948 | 0.945 | 0.043 |

Booklet on 5-factor CFA (speed) | 22,557.918 | 545 | <.001 | 0.038 [0.038, 0.039] | 0.892 | 0.882 | 0.039 |

**Table 6.**WLSMV estimates for the structural regression model of the booklet effect in the mathematics (M) and science (S) tasks.

Parameter | Unstandardized | Standardized | Parameter | Unstandardized | Standardized | |||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|

Estimate | SE | Estimate | SE | Estimate | SE | Estimate | SE | |||||

Factor loadings | Factor loadings | |||||||||||

MA01 | 1.000 | - | 0.454 | 0.007 | SF01 | 1.000 | - | 0.500 | 0.006 | |||

MA02A | 1.469 | 0.027 | 0.667 | 0.006 | SF02 | 1.253 | 0.019 | 0.627 | 0.006 | |||

MA02B | 1.337 | 0.025 | 0.608 | 0.006 | SF03 | 1.211 | 0.020 | 0.606 | 0.007 | |||

MA02C | 1.548 | 0.027 | 0.703 | 0.005 | SF04 | 1.221 | 0.020 | 0.611 | 0.007 | |||

MA03A | 0.988 | 0.022 | 0.449 | 0.007 | SF05 | 1.075 | 0.019 | 0.537 | 0.007 | |||

MA03B | 1.551 | 0.028 | 0.705 | 0.005 | SF06 | 0.807 | 0.019 | 0.403 | 0.008 | |||

MA04A | 1.396 | 0.025 | 0.634 | 0.005 | SF07A | 0.777 | 0.019 | 0.389 | 0.009 | |||

MA04B | 1.507 | 0.026 | 0.685 | 0.005 | SF07B | 0.920 | 0.022 | 0.460 | 0.009 | |||

MA05A | 0.883 | 0.021 | 0.401 | 0.008 | SF08 | 0.916 | 0.020 | 0.458 | 0.009 | |||

MA05B | 1.515 | 0.027 | 0.688 | 0.005 | SF09 | 1.225 | 0.019 | 0.613 | 0.006 | |||

MA06A | 1.723 | 0.029 | 0.783 | 0.005 | SS01 | 1.311 | 0.020 | 0.656 | 0.006 | |||

MA06B | 1.649 | 0.028 | 0.749 | 0.005 | SS02 | 1.151 | 0.019 | 0.575 | 0.006 | |||

MR01A | 1.090 | 0.024 | 0.496 | 0.007 | SS03 | 1.129 | 0.019 | 0.565 | 0.006 | |||

MR01B | 1.321 | 0.026 | 0.600 | 0.006 | SS04 | 1.190 | 0.019 | 0.595 | 0.006 | |||

MR02A | 1.569 | 0.027 | 0.713 | 0.005 | SS05 | 1.192 | 0.019 | 0.596 | 0.006 | |||

MR02B | 1.523 | 0.027 | 0.692 | 0.005 | SS07 | 0.992 | 0.018 | 0.496 | 0.006 | |||

MR03 | 1.641 | 0.028 | 0.745 | 0.005 | SS08 | 0.914 | 0.019 | 0.457 | 0.008 | |||

MR04 | 1.362 | 0.025 | 0.619 | 0.006 | SS09 | 1.100 | 0.020 | 0.550 | 0.007 | |||

MP01A | 1.500 | 0.026 | 0.682 | 0.005 | ||||||||

MP01B | 1.082 | 0.022 | 0.492 | 0.007 | Direct effects on ability | |||||||

MP02 | 1.220 | 0.024 | 0.555 | 0.006 | Booklet → M | −0.044 | 0.006 | −0.049 | 0.006 | |||

MP03 | 1.070 | 0.023 | 0.486 | 0.007 | Booklet → S | −0.040 | 0.007 | −0.040 | 0.007 | |||

MP04 | 1.592 | 0.027 | 0.723 | 0.005 | ||||||||

MP05A | 1.606 | 0.029 | 0.730 | 0.006 | ||||||||

MP05B | 1.275 | 0.027 | 0.579 | 0.008 | ||||||||

MP06A | 1.260 | 0.027 | 0.573 | 0.008 | ||||||||

MP06B | 1.477 | 0.027 | 0.671 | 0.006 | ||||||||

MP07A | 1.434 | 0.027 | 0.652 | 0.007 | ||||||||

MP07B | 1.468 | 0.029 | 0.667 | 0.008 |

^{2}(1078) = 33,827.461; p < .001; RMSEA = 0.033; CFI = 0.948; TLI = 0.945; SRMR = 0.040.

**Table 7.**WLSMV estimates for the structural regression model of the booklet effect on performance in individual mathematics (M) and science (S) tasks.

Item | Unstandardized | Standardized | Item | Unstandardized | Standardized | |||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|

Estimate | SE | Estimate | SE | Estimate | SE | Estimate | SE | |||||

MA01 | 0.046 | 0.015 | 0.023 | 0.008 | MP05B | 0.201 | 0.019 | 0.100 | 0.009 | |||

MA02A | −0.058 | 0.014 | −0.029 | 0.007 | MP06A | 0.224 | 0.019 | 0.111 | 0.010 | |||

MA02B | −0.151 | 0.014 | −0.075 | 0.007 | MP06B | 0.281 | 0.017 | 0.140 | 0.008 | |||

MA02C | −0.053 | 0.013 | −0.026 | 0.006 | MP07A | 0.196 | 0.019 | 0.098 | 0.010 | |||

MA03A | −0.060 | 0.015 | −0.030 | 0.007 | MP07B | 0.166 | 0.022 | 0.083 | 0.011 | |||

MA03B | −0.091 | 0.013 | −0.045 | 0.007 | SF01 | −0.068 | 0.013 | −0.034 | 0.007 | |||

MA04A | −0.096 | 0.013 | −0.048 | 0.006 | SF02 | 0.063 | 0.014 | 0.032 | 0.007 | |||

MA04B | −0.131 | 0.013 | −0.065 | 0.007 | SF03 | −0.059 | 0.015 | −0.029 | 0.007 | |||

MA05A | −0.003 | 0.015 | −0.001 | 0.008 | SF04 | −0.044 | 0.014 | −0.022 | 0.007 | |||

MA05B | −0.132 | 0.013 | −0.066 | 0.007 | SF05 | −0.153 | 0.015 | −0.076 | 0.007 | |||

MA06A | −0.178 | 0.013 | −0.088 | 0.006 | SF06 | −0.124 | 0.016 | −0.062 | 0.008 | |||

MA06B | −0.215 | 0.015 | −0.106 | 0.007 | SF07A | −0.168 | 0.016 | −0.084 | 0.008 | |||

MR01A | −0.091 | 0.015 | −0.046 | 0.008 | SF07B | −0.079 | 0.021 | −0.040 | 0.011 | |||

MR01B | −0.061 | 0.014 | −0.031 | 0.007 | SF08 | −0.281 | 0.016 | −0.139 | 0.008 | |||

MR02A | −0.128 | 0.014 | −0.064 | 0.007 | SF09 | −0.013 | 0.014 | −0.006 | 0.007 | |||

MR02B | −0.042 | 0.014 | −0.021 | 0.007 | SS01 | −0.049 | 0.014 | −0.024 | 0.007 | |||

MR03 | −0.074 | 0.014 | −0.037 | 0.007 | SS02 | 0.021 | 0.014 | 0.011 | 0.007 | |||

MR04 | −0.032 | 0.015 | −0.016 | 0.007 | SS03 | −0.049 | 0.014 | −0.024 | 0.007 | |||

MP01A | 0.140 | 0.013 | 0.070 | 0.007 | SS04 | 0.165 | 0.013 | 0.083 | 0.006 | |||

MP01B | 0.118 | 0.014 | 0.059 | 0.007 | SS05 | 0.183 | 0.013 | 0.091 | 0.006 | |||

MP02 | 0.132 | 0.014 | 0.066 | 0.007 | SS07 | 0.150 | 0.013 | 0.075 | 0.007 | |||

MP03 | 0.183 | 0.014 | 0.091 | 0.007 | SS08 | 0.132 | 0.015 | 0.066 | 0.008 | |||

MP04 | 0.247 | 0.015 | 0.123 | 0.007 | SS09 | 0.076 | 0.015 | 0.038 | 0.008 | |||

MP05A | 0.203 | 0.014 | 0.101 | 0.007 |

**Table 8.**Maximum likelihood estimates for the structural regression model of the booklet effect on speed at the mathematics (M) and science (S) task levels.

Parameter | Unstandardized | Standardized | Parameter | Unstandardized | Standardized | |||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|

Estimate | SE | Estimate | SE | Estimate | SE | Estimate | SE | |||||

Factor loadings | Factor loadings | |||||||||||

Speed MTask1 | Speed STask1 | |||||||||||

MA01_S | 1.000 | - | 0.439 | 0.006 | SF01_S | 1.000 | - | 0.367 | 0.006 | |||

MA02_S | 1.941 | 0.034 | 0.568 | 0.005 | SF02_S | 1.012 | 0.021 | 0.392 | 0.006 | |||

MA03_S | 1.347 | 0.024 | 0.542 | 0.005 | SF03_S | 0.785 | 0.020 | 0.378 | 0.006 | |||

MA04_S | 3.134 | 0.053 | 0.637 | 0.005 | SF04_S | 0.852 | 0.020 | 0.429 | 0.006 | |||

MA05_S | 3.105 | 0.052 | 0.656 | 0.004 | SF05_S | 0.828 | 0.018 | 0.500 | 0.006 | |||

MA06_S | 2.773 | 0.049 | 0.575 | 0.005 | SF06_S | 0.307 | 0.007 | 0.534 | 0.006 | |||

SF07_S | 1.594 | 0.033 | 0.653 | 0.005 | ||||||||

Speed MTask2 | SF08_S | 0.386 | 0.009 | 0.517 | 0.006 | |||||||

MR01_S | 1.000 | - | 0.508 | 0.006 | SF09_S | 1.293 | 0.027 | 0.562 | 0.005 | |||

MR02_S | 2.852 | 0.045 | 0.657 | 0.005 | ||||||||

MR03_S | 1.976 | 0.033 | 0.601 | 0.005 | Speed STask2 | |||||||

MR04_S | 1.947 | 0.040 | 0.423 | 0.006 | SS01_S | 1.000 | - | 0.473 | 0.005 | |||

SS02_S | 1.029 | 0.016 | 0.589 | 0.005 | ||||||||

Speed MTask3 | SS03_S | 1.914 | 0.028 | 0.665 | 0.004 | |||||||

MP01_S | 1.000 | - | 0.591 | 0.005 | SS04_S | 1.790 | 0.027 | 0.684 | 0.004 | |||

MP02_S | 0.150 | 0.003 | 0.319 | 0.006 | SS05_S | 1.163 | 0.019 | 0.583 | 0.005 | |||

MP03_S | 0.279 | 0.005 | 0.423 | 0.006 | SS07_S | 1.125 | 0.019 | 0.574 | 0.005 | |||

MP04_S | 1.049 | 0.015 | 0.598 | 0.005 | SS08_S | 0.585 | 0.011 | 0.479 | 0.006 | |||

MP05_S | 0.663 | 0.010 | 0.533 | 0.005 | SS09_S | 0.332 | 0.007 | 0.381 | 0.006 | |||

MP06_S | 1.199 | 0.016 | 0.655 | 0.005 | ||||||||

MP07_S | 0.920 | 0.014 | 0.549 | 0.005 | ||||||||

Direct effects | ||||||||||||

Booklet → | ||||||||||||

Speed MTask1 | −15.027 | 0.276 | −0.511 | 0.005 | ||||||||

Speed MTask2 | −23.427 | 0.371 | −0.614 | 0.006 | ||||||||

Speed MTask3 | 43.479 | 0.743 | 0.451 | 0.006 | ||||||||

Speed STask1 | −22.646 | 0.525 | −0.450 | 0.006 | ||||||||

Speed STask2 | 39.259 | 0.614 | 0.593 | 0.005 |

^{2}(545) = 22,557.918; p < .001; RMSEA = 0.038; CFI = 0.892; TLI = 0.882; SRMR = 0.039.

**Table 9.**Maximum likelihood estimates for the structural regression model of the booklet effect on speed at individual screen level in mathematics (M) and science (S) tasks.

Screen | Unstandardized | Standardized | Screen | Unstandardized | Standardized | |||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|

Estimate | SE | Estimate | SE | Estimate | SE | Estimate | SE | |||||

MA01_S | 0.267 | 0.469 | 0.004 | 0.007 | SF01_S | −5.308 | 0.898 | −0.039 | 0.007 | |||

MA02_S | 4.235 | 0.691 | 0.042 | 0.007 | SF02_S | 4.416 | 0.857 | 0.034 | 0.007 | |||

MA03_S | 6.338 | 0.510 | 0.087 | 0.007 | SF03_S | 14.372 | 0.751 | 0.137 | 0.007 | |||

MA04_S | 6.442 | 0.982 | 0.045 | 0.007 | SF04_S | 8.194 | 0.714 | 0.082 | 0.007 | |||

MA05_S | −7.187 | 0.912 | −0.052 | 0.007 | SF05_S | −6.601 | 0.571 | −0.079 | 0.007 | |||

MA06_S | −15.611 | 0.932 | −0.110 | 0.007 | SF06_S | 2.397 | 0.206 | 0.083 | 0.007 | |||

MR01_S | 4.458 | 0.729 | 0.059 | 0.010 | SF07_S | −5.775 | 0.829 | −0.047 | 0.007 | |||

MR02_S | 8.610 | 1.623 | 0.052 | 0.010 | SF08_S | 1.039 | 0.270 | 0.028 | 0.007 | |||

MR03_S | −4.115 | 1.078 | −0.033 | 0.009 | SF09_S | −14.056 | 0.797 | −0.121 | 0.007 | |||

MR04_S | −14.938 | 1.492 | −0.085 | 0.008 | SS01_S | −1.198 | 1.043 | −0.009 | 0.007 | |||

MP01_S | −4.090 | 1.085 | −0.025 | 0.007 | SS02_S | −7.340 | 0.840 | −0.063 | 0.007 | |||

MP02_S | −2.018 | 0.323 | −0.044 | 0.007 | SS03_S | 5.347 | 1.316 | 0.028 | 0.007 | |||

MP03_S | −4.575 | 0.446 | −0.072 | 0.007 | SS04_S | −9.010 | 1.228 | −0.052 | 0.007 | |||

MP04_S | −3.432 | 1.132 | −0.020 | 0.007 | SS05_S | −0.101 | 0.960 | −0.001 | 0.007 | |||

MP05_S | −0.433 | 0.825 | −0.004 | 0.007 | SS07_S | 9.754 | 0.950 | 0.075 | 0.007 | |||

MP06_S | 12.892 | 1.155 | 0.073 | 0.007 | SS08_S | 3.591 | 0.630 | 0.044 | 0.008 | |||

MP07_S | 6.903 | 1.126 | 0.043 | 0.007 | SS09_S | 0.065 | 0.472 | 0.001 | 0.008 |

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |

© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Liu, J.X.; Bulut, O.; Johnson, M.D.
Examining Position Effects on Students’ Ability and Test-Taking Speed in the TIMSS 2019 Problem-Solving and Inquiry Tasks: A Structural Equation Modeling Approach. *Psychol. Int.* **2024**, *6*, 492-508.
https://doi.org/10.3390/psycholint6020030

**AMA Style**

Liu JX, Bulut O, Johnson MD.
Examining Position Effects on Students’ Ability and Test-Taking Speed in the TIMSS 2019 Problem-Solving and Inquiry Tasks: A Structural Equation Modeling Approach. *Psychology International*. 2024; 6(2):492-508.
https://doi.org/10.3390/psycholint6020030

**Chicago/Turabian Style**

Liu, Joyce Xinle, Okan Bulut, and Matthew D. Johnson.
2024. "Examining Position Effects on Students’ Ability and Test-Taking Speed in the TIMSS 2019 Problem-Solving and Inquiry Tasks: A Structural Equation Modeling Approach" *Psychology International* 6, no. 2: 492-508.
https://doi.org/10.3390/psycholint6020030