# Expanding NAEP and TIMSS Analysis to Include Additional Variables or a New Scoring Model Using the R Package Dire

^{*}

## Abstract

**:**

## 1. Introduction

`Dire`package for direct estimation and generation of new plausible values (Dire 2.1.1 CRAN: https://cran.r-project.org/web/packages/Dire/index.html, GitHub: https://github.com/American-Institutes-for-Research/Dire, Vignette: https://cran.r-project.org/web/packages/Dire/vignettes/MML.pdf (accessed on 16 August 2023)). The existing

`TAM`software performs all of the necessary steps to go from item responses to estimate item parameters and produce new plausible values [5].

`Dire`, especially when paired with

`EdSurvey`[6], instead focuses on estimating new latent regression models and plausible values for LSA where the user wants to use the existing item parameters published by the statistical agency sharing the results. In addition,

`Dire`is intended to be able to estimate high-dimensional models used in NAEP where the mathematics scale is a composite of five correlated subscales, as well as TIMSS where four- (grade 4) or five- (grade 8) constructs are estimated in a multi-dimensional model to estimate subscales [7].

`Dire`to estimate a conditioning model. We then show the estimation strategy used by

`Dire`to estimate high-dimensional models without suffering from exponential-time costs, and how plausible values are generated. We provide a simple example of using

`Dire`to estimate a new conditioning model with

`EdSurvey`; for completeness we also show how to estimate a toy example (that is not an existing LSA) in

`Dire`without using

`EdSurvey`.

## 2. Background

`Dire`package):

`TAM`[5],

`mirt`[11], and

`Dire`. Additionally, the

`NEPSscaling`package may be used to draw plausible values focusing on data from the German National Educational Panel Study (NEPS) [12]. The models shown in this paper may be fit with

`TAM`,

`mirt`, or

`Dire`. In contrast to

`mirt`and

`TAM`,

`Dire`uses a different method of calculating the multi-dimensional integral. Where

`TAM`and

`mirt`calculate a multi-dimensional integral by calculating all of the dimensions at once,

`Dire`does so one- or two-dimensions at a time. When

`Dire`estimates the conditioning model, it uses the existing item parameters. The

`EdSurvey`package will download and format them for NAEP and TIMSS data for the user so that the process is seamless.

`Dire`, similar to other plausible value approaches, mirror those used in the statistical package AM [13] and result in unbiased estimates of parameters included in the conditioning model [1,4].

`Dire`package is integrated into the

`EdSurvey`package, which allows users to download, read in, manipulate, and analyze U.S. NCES, IEA, and OECD LSA data. For a detailed overview of the functionality of the

`EdSurvey`package, see [6]. This paper synthesizes portions of the methodology covered in the

`Dire`vignette with worked examples to provide an overview of the methods and application of

`Dire`in analyzing LSA data.

## 3. Methodology in Dire

#### 3.1. Marginal Maximum Likelihood

#### 3.1.1. Likelihood

`Dire`treats them as fixed]. The parameters to estimate are $\mathsf{\beta}$ and $\sigma $.

`Dire`fits a between-item multidimensional model]. Now ${\mathsf{\theta}}_{i}$ is a vector, with an element for each construct, and dependence between the constructs is allowed by modeling the residual as a multivariate normal with residual $\Sigma $. We assume here that each item is associated with only one construct. The target integral to evaluate is then:

#### 3.1.2. Integral Approximation

`Dire`, following the methodology of [4], transforms the problem of approximating a multi-dimensional integral into one of estimating a uni-dimensional integral per subscale. To do this, we make the observation that the multi-dimensional case is analogous to a seemingly unrelated regression (SUR) model with identical regressors and normally distributed errors that are correlated across equations. It is shown in [23] that for this special case of a SUR model, estimates are no more efficient when regressors are estimated simultaneously vs. separately. We begin by writing the joint density as a product of a marginal and a conditional, as in Equation (4).

`Dire`creates a fixed grid of integration points that the integral is then evaluated over. Because latent ability is assumed to follow a standard normal distribution when calibrating item difficulty, the default range for this grid is chosen to be $[-4,4]$. Within this range, 30 equidistant quadrature points are chosen by default. The user may specify an alternative range and number of quadrature points.

- Estimate ${\mathsf{\beta}}_{j}$ and ${\sigma}_{j}$ for each subscale j by optimizing a univariate density (5) and
- Hold the ${\mathsf{\beta}}_{j}$ and ${\sigma}_{j}^{2}$ (the diagonal terms in $\Sigma $) estimates fixed and estimate the correlations for each of the $\left(\right)$ pairs of subscales (the non-diagonal elements of $\Sigma $).

#### 3.2. Parameter Estimation

#### 3.2.1. Estimating $\beta $ and $\sigma $

`Dire`documentation for how to specify the parameters).

`Dire`calculates the item response likelihoods outside of the optimized function, since these do not depend on $\mathsf{\beta}$ and $\sigma $. The optimization then only takes place over the univariate normal portion of the likelihood.

`Dire`takes a robust approach to optimization that begins with using the memory-efficient L-BFGS-B algorithm to identify a maximum. Because the L-BFGS-B implementation programmed in R does not have a convergence criterion based on the gradient,

`Dire`further refines these estimates with a series of Newton steps, using the gradient as a convergence criterion—since a maximum has zero gradient, by definition. Because the integration nodes and item parameters are fixed during optimization, the term $\mathrm{Pr}\left({R}_{ih}\right|{t}_{q},{\mathit{P}}_{h})$ will not change over the surface of $\mathsf{\beta}$ or $\sigma $, so it is only necessary to evaluate it once per student over the grid.

`Dire`user could fit the model we fit for NAEP for TIMSS data and recover the subscales (throwing out the implied but never used composite score that would also have to be calculated)).

#### 3.2.2. Estimating covariance terms, ${\sigma}_{ij}$

`Dire`optimizes ${\sigma}_{ij}$ by optimizing the correlation, r, in the Fisher Z space [Since variance terms have already been estimated at this stage, the covariance can be obtained by optimizing the correlation and calculating ${\sigma}_{ij}=r{\sigma}_{ii}{\sigma}_{jj}$]. This is a transformation defined by

`Dire`introduces a novel solution for accurately estimating correlations that relies on spline interpolation of the likelihood surfaces. Consider the Gaussian densities in Figure 3—the narrow, elongated shape suggests that the algebra and geometry constructs are highly correlated for this student. In Figure 4, we compare a student’s likelihood using the standard grid and the grid produced with a cubic spline interpolation of the likelihood surface. The approximation of the likelihood surface using the standard grid is subject to discontinuities that are not real but the result of discretization.

#### 3.3. Variance Estimation

`Dire`estimates the variances of $\mathsf{\beta}$, as well as the residual variances (diagonal of the covariance matrix), using the cluster-robust (CR-0) method in Binder [26]. This method is also known as the Taylor Series Method, e.g., in Wolter’s survey sampling text [27]. These methods account for the weights and are cluster-robust in the sense of estimating arbitrary covariances between units within strata.

#### 3.4. Plausible Values

- Holding $\Sigma $ fixed, draw $\tilde{\mathsf{\beta}}$ from a normal approximation to the posterior of $\mathsf{\beta}$
- Using the same $\Sigma $ and $\tilde{\mathsf{\beta}}$, compute the posterior distribution of ${\theta}_{i}$ for each student using the same quadrature nodes as in the initial optimization routine. Compute the mean and variance of each student’s posterior distribution as:$$\begin{array}{c}\hfill {\overline{\theta}}_{i}\approx \frac{{\sum}_{q=1}^{Q}{t}_{q}{l}_{i}\left(\right)open="("\; close=")">{t}_{q}f\left(\right)open="("\; close=")">{t}_{q};{\mathit{X}}_{\mathbf{i}}\tilde{\mathsf{\beta}},\Sigma}{}{\sum}_{q=1}^{Q}{l}_{i}\left(\right)open="("\; close=")">{t}_{q}f\left(\right)open="("\; close=")">{t}_{q};{\mathit{X}}_{\mathbf{i}}\tilde{\mathsf{\beta}},\Sigma \end{array}$$$$\begin{array}{c}\hfill var\left(\right)open="("\; close=")">{\theta}_{i}|\phantom{\rule{4pt}{0ex}}\mathit{R},\mathit{X},\mathit{P},\tilde{\mathsf{\beta}},\Sigma \approx \frac{{\sum}_{q=1}^{Q}{\left(\right)}^{{t}_{q}}2}{{l}_{i}}f\left(\right)open="("\; close=")">{t}_{q};{\mathit{X}}_{\mathbf{i}}\tilde{\mathsf{\beta}},\Sigma \\ {\sum}_{q=1}^{Q}{l}_{i}\left(\right)open="("\; close=")">{t}_{q}f\left(\right)open="("\; close=")">{t}_{q};{\mathit{X}}_{\mathbf{i}}\tilde{\mathsf{\beta}},\Sigma \end{array}$$
- Using the mean and standard deviation from Step 2, randomly draw from a normal approximation to the posterior distribution.

`Dire`’s focus on LSA, this follows the methodology of NAEP and TIMSS [16,17].

## 4. Examples

`Dire`package, the minimum necessary components are:

- Student data, including covariates (and weights, if applicable)
- Student item responses
- Item parameters
- Scaling information

`Dire`out of the box when reading in data using the

`EdSurvey`package. For users wishing to simulate some or all of this information, the

`lsasim`R package provides a set of functions for simulating LSA data [30].

`Dire`to estimate regression parameters and draw new plausible values using simulated data, and (2) how to use

`Dire`with existing LSA data to address a particular research question.

#### 4.1. Simulated Data

`lsasim`, we simulated background variables for 2000 students divided into 40 strata and 2 sampling units, as well as their responses to 20 dichotomous items divided into two subscales and the parameters for those items (see Appendix A).

`formula`: the model to be fit, expressed as Y X1 + …+ XK`stuItems`: student item responses`stuDat`: student background variables, weights, and other sampling information`idVar`: student identifier variable in stuDat`dichotParamTab`: item parameters for dichotomous items`polyParamTab`: item parameters for polytomous items`testScale`: location, scale, and weights for each (sub)test`strataVar`: a variable in stuDat, indicating the stratum for each row`PSUVar`: a variable in stuDat, the primary sampling unit (PSU)

#### 4.2. TIMSS Data

`EdSurvey`R package [(EdSurvey 4.0.1 CRAN: https://cran.r-project.org/web/packages/EdSurvey/index.html, GitHub: https://github.com/American-Institutes-for-Research/EdSurvey (accessed on 16 August 2023)]. These data bear information about how much screen time test takers spend on each item in the assessment, and these variables are not included in the conditioning model.

`first_subject`is 1 when the student takes science first and zero otherwise.

## 5. Results

## 6. Discussion and Conclusions

`Dire`enables users to use direct estimation, or simply fit a large conditioning model that includes external or process data variables. Once the conditioning model is fit, the user can draw new plausible values in order to have unbiased estimators of conditional statistics.

`Dire`estimates the conditioning model efficiently by maximizing one subtest at a time (in the case of composite scores such as in NAEP or for TIMSS subtests). In addition, when the correlations are high it is often neceessary to use a higher density grid. To quickly calculate that, spline interpolation is applied, for accurate and fast estimation of correlations between subscales—this is novel in

`Dire`and was used because the approximation was observed to be fast and accurate.

## Author Contributions

## Funding

## Institutional Review Board Statement

## Informed Consent Statement

## Data Availability Statement

## Acknowledgments

## Conflicts of Interest

## Appendix A. Data Simulation Code

## References

- Mislevy, R.; Beaton, A.; Kaplan, B.; Sheehan, K. Estimating population characteristics from sparse matrix samples of item responses. J. Educ. Meas.
**1992**, 29, 133–162. [Google Scholar] [CrossRef] - OECD. PISA 2018 Technical Report: Appendix H; 2019; Available online: https://www.oecd.org/pisa/data/pisa2018technicalreport/ (accessed on 16 August 2023).
- Hicks, J.C. The Use of Process Data to Examine Reading Strategies. Ph.D. Thesis, The Graduate School at The University of North Carolina at Greensboro, Greensboro, NC, USA, 2019. [Google Scholar]
- Cohen, J.; Jiang, T. Comparison of partially measured latent traits across nominal subgroups. J. Am. Stat. Assoc.
**1999**, 94, 1035–1044. [Google Scholar] [CrossRef] - Robitzsch, A.; Kiefer, T.; Wu, M. TAM: Test Analysis Modules; R Package Version 4.1-4; 2022; Available online: https://cran.r-project.org/web/packages/TAM/index.html (accessed on 16 August 2023).
- Michael, L.; Zhang, T.; Bailey, P.; Buehler, E.; Fink, T.; Huo, H.; Lee, S.J.; Liao, S.J.; Sikali, E. Analyzing NCES Data Using EdSurvey: A User’s Guide; 2022; Available online: https://naep-research.airprojects.org/Portals/0/EdSurvey_A_Users_Guide/_book/index.html (accessed on 16 August 2023).
- Foy, P.; Fishbein, B.; von Davier, M.; Yin, L. Implementing the TIMSS 2019 Scaling Methodology. In Methods and Procedures: TIMSS 2019 Technical Report; Martin, M.O., von Davier, M., Mullis, I.V.S., Eds.; TIMSS and PIRLS International Study Center at Boston College: Chestnut Hill, MA, USA, 2020; Chapter 12; pp. 12.1–12.146. [Google Scholar]
- R Core Team. R: A Language and Environment for Statistical Computing; R Foundation for Statistical Computing: Vienna, Austria, 2023. [Google Scholar]
- Asparouhov, T.; Muthén, B. Plausible Values for Latent Variables Using Mplus. Unpublished. Available online: http://www.statmodel.com/download/Plausible.pdf (accessed on 16 August 2023).
- Mair, P.; Rosseel, Y.; Gruber, K. CRAN Task View: Psychometric Models and Methods. Version 2022-09-25. 2022. Available online: https://CRAN.R-project.org/view=Psychometrics (accessed on 16 August 2023).
- Chalmers, R.P. mirt: A Multidimensional Item Response Theory Package for the R Environment. J. Stat. Softw.
**2012**, 48, 1–29. [Google Scholar] [CrossRef] - Scharl, A.C.; Zink, E. NEPSscaling: Plausible value estimation for competence tests administered in the German National Educational Panel Study. Large-Scale Assessm. Educ.
**2022**, 10, 28. [Google Scholar] [CrossRef] - American Institutes for Research. AM [Software]. 2003. Available online: https://am.air.org/ (accessed on 16 August 2023).
- Mislevy, R.J.; Wilson, M. Marginal maximum likelihood estimation for a psychometric model of discontinuous development. Psychometrika
**1996**, 61, 41–71. [Google Scholar] [CrossRef] - Averett, C.; Ferraro, D.; Tang, J.; Erberber, E.; Stearns, P.; Provasnik, S. Trends in International Mathematics and Science Study (TIMSS): U.S. TIMSS 2015 and TIMSS Advanced 1995 & 2015 Technical Report and User’s Guide (NCES 2018-020); U.S. Department of Education, National Center for Education Statistics, Institute of Education Sciences: Washington, DC, USA, 2017. Available online: https://nces.ed.gov/pubs2018/2018020_1.pdf (accessed on 16 August 2023).
- National Center for Education Statistics. NAEP Assessment Sample Deisgn TDW. 2023. Available online: https://nces.ed.gov/nationsreportcard/tdw/sample_design/ (accessed on 16 August 2023).
- von Davier, M. TIMSS 2019 Scaling Methodology: Item Response Theory, Population Models, and Linking Across Modes. In Methods and Procedures: TIMSS 2019 Technical Report; Martin, M.O., von Davier, M., Mullis, I.V.S., Eds.; TIMSS and PIRLS International Study Center at Boston College: Chestnut Hill, MA, USA, 2020; Chapter 11; pp. 11.1–11.25. [Google Scholar]
- Burns, S.; Wang, X.; Henning, A. NCES Handbook of Survey Methods. NCES 2011-609; U.S. Department of Education, National Center for Education Statistics, U.S. Government Printing Office: Washington, DC, USA, 2011. Available online: https://nces.ed.gov/pubs2011/2011609.pdf (accessed on 16 August 2023).
- Doran, H. A Collection of Numerical Recipes Useful for Building Scalable Psychometric Applications. J. Educ. Behav. Stat.
**2023**, 48, 37–69. [Google Scholar] [CrossRef] - Andersson, B.; Xin, T. Estimation of latent regression item response theory models using a second-order Laplace approximation. J. Educ. Behav. Stat.
**2021**, 46, 244–265. [Google Scholar] [CrossRef] - Schilling, S.; Bock, R.D. High-dimensional maximum marginal likelihood item factor analysis by adaptive quadrature. Psychometrika
**2005**, 70, 533–555. [Google Scholar] [CrossRef] - Cai, L. Metropolis-Hastings Robbins-Monro algorithm for confirmatory item factor analysis. J. Educ. Behav. Stat.
**2010**, 35, 307–335. [Google Scholar] [CrossRef] - Greene, W.H. Econometric Analysis, 5th ed.; Pearson Education: Upper Saddle River, NJ, USA, 2003. [Google Scholar]
- Sohn, K. An expectation-maximization algorithm to estimate the integrated choice and latent variable model. Transp. Sci.
**2017**, 51, 946–967. [Google Scholar] [CrossRef] - Mullis, I.V.; Martin, M.O. TIMSS 2019 Assessment Frameworks; Technical Report; TIMSS & PIRLS International Study Center: Boston, MA, USA, 2017. [Google Scholar]
- Binder, D.A. On the variances of asymptotically normal estimators from complex surveys. Int. Stat. Rev.
**1983**, 51, 279–292. [Google Scholar] [CrossRef] - Wolter, K.M. Intorudction to Variance Estimation, 2nd ed.; Springer: New York, NY, USA, 2007. [Google Scholar]
- Rubin, D.B. Multiple Imputation for Nonresponse in Surveys; Wiley: Hoboken, NJ, USA, 1987. [Google Scholar]
- Wu, M. The role of plausible values in large-scale surveys. Stud. Educ. Eval.
**2005**, 31, 114–128. [Google Scholar] [CrossRef] - Matta, T.; Rutkowski, L.; Rutkowski, D.; Liaw, Y.L.L.; Leoncio, W. lsasim: Functions to Facilitate the Simulation of Large Scale Assessment Data; R Package Version 2.1.3; SpringerOpen: London, UK, 2023. [Google Scholar]
- Luo, Y.; Dimitrov, D.M. A short note on obtaining point estimates of the IRT ability parameter with MCMC estimation in Mplus: How many plausible values are needed? Educ. Psychol. Meas.
**2019**, 79, 272–287. [Google Scholar] [CrossRef] [PubMed] - Meng, X.B.; Tao, J.; Chang, H.H. A conditional joint modeling approach for locally dependent item responses and response times. J. Educ. Meas.
**2015**, 52, 1–27. [Google Scholar] [CrossRef] - De Boeck, P.; Scalise, K. Collaborative problem solving: Processing actions, time, and performance. Front. Psychol.
**2019**, 10, 1280. [Google Scholar] [CrossRef]

**Figure 2.**Impact of model for $\theta $ on student likelihood, evaluated at fixed quadrature points [−4,4]. (

**a**) Student likelihood at different values of $\sigma $, holding $\beta $ fixed at 0. (

**b**) Student likelihood at different values of $\beta $, holding $\sigma $ fixed at 1.

**Figure 3.**3D and 2D density plots of the components of a student’s likelihood, evaluated over a grid of fixed quadrature points over ${\theta}_{1}$ (x-axis) and ${\theta}_{2}$ (y-axis). (

**a**) 3D plot of item response likelihood. (

**b**) 3D plot of latent trait. (

**c**) 3D plot of complete distribution. (

**d**) Contour plot of item response likelihood. (

**e**) Contour plot of latent trait. (

**f**) Contour plot of complete distribution.

**Figure 4.**Bivariate density plots for Algebra and Geometry subscales for a select student, evaluated over a grid of fixed quadrature points over ${\theta}_{1}$ (x-axis) and ${\theta}_{2}$ (y-axis). (

**a**) Density evaluated over the standard grid. (

**b**) Density evaluated over the spline interpolated grid.

**Figure 5.**Predicted U.S. 2019 Grade 8 TIMSS Math scale score (color coded) as a function of screen time on math items (x-axis), screen time on science items (y-axis), for students who were administered math items first (

**left panel**) and science items first (

**right panel**).

**Figure 6.**Plot of regression coefficients (excluding the intercept) for the model shown in Table 3, including 95% confidence intervals for the original plausible values (top, orange) and the plausible values that were generated after fitting our new conditioning model (bottom, blue).

Method | $\widehat{\mathit{\rho}}$ | Log-Likelihood | Computing Time |
---|---|---|---|

Standard grid, Q = 34 | 0.995 | −250,553.4 | 44 s |

Spline interpolated grid, Q = 34 | 0.962 | −252,857.9 | 119 s |

Standard grid, Q = 136 | 0.961 | −252,829.1 | 381 s |

**Table 2.**Summary of model; math achievement regressed on time spent on math, time spent on science, first subject seen, and their two-way interactions. *** p < 0.0001, ** p < 0.001.

Estimate | SE | t-Value | dof | Pr(>|t|) | |
---|---|---|---|---|---|

intercept | 523.01 | 4.60 | 113.62 | 65.0 | <$2.2\times {10}^{-16}$ *** |

$Tim{e}_{M}$ | 13.95 | 3.79 | 3.68 | 23.3 | 0.0012 **$\phantom{*}$ |

$Tim{e}_{S}$ | −2.84 | 4.35 | −0.65 | 7.4 | 0.53 $\phantom{*\phantom{\rule{-0.166667em}{0ex}}*\phantom{\rule{-0.166667em}{0ex}}*}$ |

$Firs{t}_{S}$ | 16.73 | 2.72 | 6.16 | 22.1 | $3.29\times {10}^{-6}$ *** |

$Firs{t}_{S}\times Tim{e}_{M}$ | 41.83 | 5.3 | 7.97 | 28.0 | $1.13\times {10}^{-8}$ *** |

$Firs{t}_{S}\times Tim{e}_{S}$ | −38.31 | 6.0 | −6.42 | 14.1 | $1.55\times {10}^{-5}$ *** |

$Tim{e}_{M}\times Tim{e}_{S}$ | −13.30 | 2.2 | −5.97 | 11.8 | $7.03\times {10}^{-5}$ *** |

**Table 3.**Linear regression models for math achievement using the original PVs (${R}^{2}$ = 0.112) and new PVs (${R}^{2}$ = 0.141). *** p < 0.0001, ** p < 0.001.

Estimate | SE | t-Value | dof | Pr(>|t|) | |
---|---|---|---|---|---|

Original PVs | |||||

intercept | 518.68 | 4.98 | 104.17 | 60.9 | <$2.2\times {10}^{-16}$ *** |

$Tim{e}_{M}$ | 15.36 | 4.90 | 3.14 | 14.5 | 0.007 **$\phantom{*}$ |

$Tim{e}_{S}$ | −2.54 | 4.49 | −0.57 | 15.3 | 0.58 $\phantom{*\phantom{\rule{-0.166667em}{0ex}}*\phantom{\rule{-0.166667em}{0ex}}*}$ |

$Firs{t}_{S}$ | 22.19 | 3.38 | 6.57 | 18.0 | $3.58\times {10}^{-6}$ *** |

$Firs{t}_{S}\times Tim{e}_{M}$ | 31.93 | 5.60 | 5.70 | 36.0 | $1.76\times {10}^{-6}$ *** |

$Firs{t}_{S}\times Tim{e}_{S}$ | −34.89 | 5.97 | −5.84 | 25.2 | $4.17\times {10}^{-6}$ *** |

$Tim{e}_{M}\times Tim{e}_{S}$ | −12.15 | 2.07 | −5.87 | 16.7 | $2.01\times {10}^{-5}$ *** |

New PVs | |||||

intercept | 523.06 | 4.19 | 124.71 | 93.8 | <$2.2\times {10}^{-16}$ *** |

$Tim{e}_{M}$ | 13.85 | 3.84 | 3.61 | 22.3 | 0.002 **$\phantom{*}$ |

$Tim{e}_{S}$ | −3.20 | 3.14 | −1.02 | 32.0 | 0.32 $\phantom{*\phantom{\rule{-0.166667em}{0ex}}*\phantom{\rule{-0.166667em}{0ex}}*}$ |

$Firs{t}_{S}$ | 16.63 | 2.86 | 5.81 | 24.8 | $4.85\times {10}^{-6}$ *** |

$Firs{t}_{S}\times Tim{e}_{M}$ | 42.07 | 4.57 | 9.21 | 63.6 | $2.59\times {10}^{-13}$ *** |

$Firs{t}_{S}\times Tim{e}_{S}$ | −38.07 | 4.99 | −7.63 | 48.6 | $7.45\times {10}^{-10}$ *** |

$Tim{e}_{M}\times Tim{e}_{S}$ | −13.36 | 2.41 | −5.54 | 15.0 | $5.60\times {10}^{-5}$ *** |

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |

© 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Bailey, P.D.; Webb, B.
Expanding NAEP and TIMSS Analysis to Include Additional Variables or a New Scoring Model Using the *R* Package *Dire*. *Psych* **2023**, *5*, 876-895.
https://doi.org/10.3390/psych5030058

**AMA Style**

Bailey PD, Webb B.
Expanding NAEP and TIMSS Analysis to Include Additional Variables or a New Scoring Model Using the *R* Package *Dire*. *Psych*. 2023; 5(3):876-895.
https://doi.org/10.3390/psych5030058

**Chicago/Turabian Style**

Bailey, Paul Dean, and Blue Webb.
2023. "Expanding NAEP and TIMSS Analysis to Include Additional Variables or a New Scoring Model Using the *R* Package *Dire*" *Psych* 5, no. 3: 876-895.
https://doi.org/10.3390/psych5030058