Parameter Estimation of KST-IRT Model under Local Dependence

Abstract

A mantra often repeated in the introductory material to psychometrics and Item Response Theory (IRT) is that a Rasch model is a probabilistic version of a Guttman scale. The idea comes from the observation that a sigmoidal item response function provides a probabilistic version of the characteristic function that models an item response in the Guttman scale. It appears, however, more difficult to reconcile the assumption of local independence, which traditionally accompanies the Rasch model, with the item dependence existing in a Guttman scale. In recent work, an alternative probabilistic version of a Guttman scale was proposed, combining Knowledge Space Theory (KST) with IRT modeling, here referred to as KST-IRT. The present work has, therefore, a two-fold aim. Firstly, the estimation of the parameters involved in KST-IRT models is discussed. More in detail, two estimation methods based on the Expectation Maximization (EM) procedure are suggested, i.e., Marginal Maximum Likelihood (MML) and Gibbs sampling, and are compared on the basis of simulation studies. Secondly, for a Guttman scale, the estimates of the KST-IRT models are compared with those of the traditional combination of the Rasch model plus local independence under the interchange of the data generation processes. Results show that the KST-IRT approach might be more effective in capturing local dependence as it appears to be more robust under misspecification of the data generation process, but it comes with the price of an increased number of parameters.

1. Introduction

Local independence (LI) is a property that serves as a fundamental assumption underlying both parametric [1,2,3] and non-parametric [4,5,6] IRT models. Despite its importance, LI is easily violated in many situations (see, e.g., [7]). Model-wise, one can generally account for two main types of local dependence (LD), one related to misspecifications in the dimensionality of the model and one related to direct and invasive relations between the items in a test (see, e.g., [8]). In the present work, we are interested in modeling the latter by means of the machinery introduced by KST (see, e.g., [9]). More in detail, we are interested in comparing the modeling of a Guttman scale [10] under two different sets of assumptions: on the one hand, the traditional IRT approach of Rasch plus LI; on the other hand, a latent trait-based version of KST models in which the LD between the items is captured by the existence of a partial order (i.e., a knowledge structure in the shape of a chain of items) of latent states. The assumption of LI is indeed often taken for granted, or its violations are ignored, in the IRT framework for various purposes, including parameter estimation and its use in adaptive testing, and consequently, it can undermine the validity of the estimation (see, e.g., [11]). Yet, the deterministic dependence between responses is a prominent feature of the Guttman scale, which is however difficult to fit into real-world data due to its rigid nature, and thus provides a perfect case study.

As it is often stated in the psychometrics literature, to make the Guttman scale more practical, the work of the early proponents, e.g., Rasch [2] and Mokken [4], evolved the Guttman scale into parametric and non-parametric probabilistic scales, respectively, which, based on the notions of invariant measurement, are generally considered to be the probabilistic versions of Guttman (see, e.g., [12]). A relevant example of this stream of literature is provided by Andrich [13], which supported the statement of Rasch being the natural probabilistic model of the Guttman scale by showing the connection between features from both, such as invariance across persons and items. However, the Rasch model’s interpretation as a probabilistic version of the Guttman scale has also been debated on the basis that it is difficult to reconcile the probabilistic nature of the Rasch model with the deterministic nature of a Guttman scale. One of the earliest examples of such critiques can be found in, e.g., Brink [14]. Specifically, the LI assumption that traditionally accompanies Rasch modeling appears to clash with the dependence between the items that are actually at the basis of a Guttman scalogram. As a consequence, it is argued that data patterns that do not conform with a Guttman scale properly fit under Rasch plus LI because the total order established between the difficulties of the items allows modeling such unlikely patterns only because LI is assumed. This suggests that interpreting the Rasch model as a probabilistic version of the Guttman scale might be debatable, since enforcing an order relation among the items by constraining the difficulty parameters is not necessarily the same as setting an order relation between the items themselves. Interestingly, a similar debate can be traced in polytomous IRT models under the name of ‘disordered thresholds’ or ‘reversed deltas’ controversy (see, e.g., [15,16,17]), and originates from the implications of obtaining estimates of the category parameters in a polytomous model which do not follow the order followed by the categories. As, conceptually, a Guttman scalogram is not different to a polytomous IRT model, it is not surprising that the same debate surfaces in both contexts. As it was suggested by Noventa et al. [18], under a KST perspective, there is indeed no formal difference between modeling a set of dependent dichotomous items or a polytomous IRT model. Once the KST concept of knowledge structure is introduced to model the deterministic relation between the basic entities, whether they are items, sub-items, or thresholds capturing the sequence of categories, then probabilistic models can be developed by imposing probability distributions over the states of such structures. Moreover, Noventa et al. [18] suggested that by extending these latter probability distributions to encompass a latent trait, KST models can also be extended to encompass IRT models and provide a generalization of both. We refer here to such an approach as KST-IRT.

The integrated KST-IRT approach is therefore capable of manifesting the definition of Guttman scales using the knowledge structure known as a graded chain (i.e., a chain of items in which the states increase by one item at a time, hence the concept of gradedness). A probabilistic model for the Guttman scale is then obtained by imposing a probability distribution over its states. The probabilities of the response patterns are given by latent class models in which the states of the structure play the role of the latent classes, and their probabilities play the role of the membership probabilities. The conditional error parameters of the model allow for those response patterns which are inconsistent with the assumed knowledge structure. This provides a different approach than the one given by the combination of Rasch plus LI since the latter is not anymore assumed. Most notably, the deterministic response behavior that is induced by the set order relation among items is then decoupled from the total order between the difficulties of the item responses. This also contrasts with the Rasch plus LI approach in which the Rasch model provides a total order between the difficulties of the items.

In Section 2, the IRT, KST, and KST-IRT frameworks are briefly introduced and discussed. The details of model specifications that pertain particularly to KST-IRT models, such as the conditional probability of responses given the knowledge state and the state response functions, are given. In Section 3, frequentist and Bayesian computational approaches are introduced to estimate the item parameters of the KST-IRT models. The setup of the marginalized likelihood for the KST-IRT model and the steps to implement MML-EM and EM-with-Gibbs sampling algorithms are provided. To investigate the performance of the estimation methods and the interplay of IRT and KST-IRT frameworks under local dependence, two simulation studies were conducted and are laid out in Section 4. The first study focuses on the estimation performances of the MML-EM and the EM-with-Gibbs procedures. While the KST-IRT framework allows for specifying the knowledge structures, two structures that each represent LI and LD were used along with varying the state response functions. For these specifications, the data-generating process and the fitted model were fully interchanged to exhibit model misspecifications. In the second study, the response behavior under local dependence is investigated by comparing the model fit of the Rasch model and KST-IRT models when the item parameters were arranged to generate the graded responses as in the Guttman scale. Deriving from the results, the KST-IRT and IRT models do not provide a symmetric interpretation of the response patterns, especially when the two models are interchanged. Section 5 further details the interpretation of the results in regard to the LD.

2. IRT, KST, and KST-IRT Models

This section provides a brief summary of some IRT and KST notions used in the present manuscript. For more detailed introductions, the readers are referred to comprehensive textbooks, e.g., van der Linden [19] for IRT and Falmagne and Doignon [9] for KST.

2.1. Item Response Models

A general IRT model for the response of the i-th individual to the j-th dichotomous items, with $i\in \{1,\dots ,N\}$ and $j\in \{1,\dots ,J\}$, is often given in the form of the four parameters logistic model (4PL, [20]), that is

$$\begin{array}{cc}\hfill P(X_{ij}=1|{\theta }_i,{\mathsf{\Gamma }}_j^4)& =c_j+(1-d_j-c_j)\frac{e^{a_j({\theta }_i-b_j)}}{1+e^{a_j({\theta }_i-b_j)}}\hfill \end{array}$$

(1)

where ${\theta }_i\in \mathbb{R}$ expresses the location of the i-th individual along the latent trait and ${\mathsf{\Gamma }}_j^4=\{a_j,b_j,c_j,d_j\}$ collects the difficulty parameter $b_j\in \mathbb{R}$, the discrimination parameter $a_j\in {\mathbb{R}}^{+}$, the guessing parameter $c_j\in [0,1]$, and the ceiling parameter $d_j\in [0,1]$ associated to the j-th item. By using a ${\mathsf{\Gamma }}_j^n$ notation, in which n is the number of item parameters, the Rasch model (RM, [2], also referred to as the 1PL model) is given here by $P(X_{ij}=1|{\theta }_i,{\mathsf{\Gamma }}_j^1)$, the Birnbaum model (also known as the 2PL) is given by $P(X_{ij}=1|{\theta }_i,{\mathsf{\Gamma }}_j^2)$, and the three parameters logistic model (3PL, [21]) is given by $P(X_{ij}=1|{\theta }_i,{\mathsf{\Gamma }}_j^3)$. Typically, the likelihood of the pattern of responses $X_i$ provided by the i-th individual is expressed by means of an assumption of LI, which captures the idea that, conditional on the value of the latent variable, there is no association between the items. A common parametric definition is the (strong) form

$$P(X_i=x_i|{\theta }_i,{\mathsf{\Gamma }}^n)=\prod _{j=1}^{J}P(X_{ij}=x_{ij}|{\theta }_i,{\mathsf{\Gamma }}_j^n)$$

(2)

where ${\mathsf{\Gamma }}^n=\left\{{\mathsf{\Gamma }}_j^n\right\}$ and the random vector $X_i=[X_{i1},\dots ,X_{iJ}]$ has realizations $x_i=[x_{i1},\dots ,x_{iJ}]$ with $x_{ij}\in \{0,1\}$. A typical IRT choice to model the probabilistic version of a Guttman scale is thus LI (2) in conjunction with a RM $P(X_{ij}=1|{\theta }_i,{\mathsf{\Gamma }}_j^1)$. In such a case, indeed, the existence of a total order between the item difficulties $b_j$ ensures, by means of LI (2), that those patterns $X_i$ that do not conform to a Guttman scalogram have lower values of their likelihood and are therefore less likely to occur.

2.2. Knowledge State Theory

By means of a combinatorial and set-theoretical approach, KST classifies individuals based on the collections of items that they can master in a given domain of knowledge (see, e.g., [9]). More in detail, let Q be a nonempty set of items $q_j$ (i.e., the domain of knowledge), then a knowledge state $K\subseteq Q$ is a collection of problems $q_j\in Q$ that an individual is capable of mastering. A knowledge structure is a pair $(Q,\mathcal{K})$ where $\mathcal{K}$ is a family of subsets of Q that always includes the full domain Q and the empty set ∅. As an example, for the domain $Q=\{q_1,q_2,q_3\}$, a possible knowledge structure is defined by the collection

$$\begin{array}{cc}\hfill \mathcal{K}& =\{\varnothing ,\left\{q_1\right\},\left\{q_2\right\},\{q_1,q_2\},\{q_1,q_3\},Q\}\hfill \end{array}$$

(3)

and represents a situation in which items $q_1$ and $q_2$ can be mastered independently of each other, while item $q_3$ requires item $q_1$ to be mastered before it can be mastered. Although arbitrary structures $\mathcal{K}\subseteq 2^Q$ can be considered, in what follows, we will only focus on two types of structures, the power set, $\mathcal{K}=2^Q$, and the graded chain $\mathcal{C}$. The latter is given by a chain of states that increase one item at a time. As an example, for $Q=\{q_1,q_2,q_3\}$ one has

$$\begin{array}{cc}\hfill \mathcal{C}& =\{\varnothing ,\left\{q_1\right\},\{q_1,q_2\},Q\}.\hfill \end{array}$$

(4)

The reason for such a choice is that, as discussed by Noventa et al. [18], in the context of KST-IRT models, the power set represents local independence (see below) while the graded chain represents local dependence. It is indeed evident that the graded chain closely resembles the structural behavior of the Guttman scale (or of a polytomous item) as the strictest form of local dependence where every item is dependent on the previous ones. In order to discuss the KST-IRT approach, first, we need to introduce probabilities over the deterministic structures just defined and discuss two important KST models.

2.2.1. Basic Local Independence Model (BLIM)

A probabilistic knowledge structure (PKS) is a triple $(Q,\mathcal{K},\pi )$ in which the deterministic knowledge structure $(Q,\mathcal{K})$ is made probabilistic by imposing a probability distribution $\pi :\mathcal{K}\to [0,1]$ over the knowledge states $K\in \mathcal{K}$. In what follows, we deviate from the traditional KST notation in favor of a more IRT-like notation that allows merging the two better. In order to do so, the symbol $X_i$ is both interpreted as a random vector (as in the IRT case) and as a subset $X_i\in 2^Q$ (as in the KST case). Such bijection between set-theoretic and vector notations was, for instance, discussed by Heller et al. [22] and Noventa et al. [18]. Hence, with a slight abuse of notation, an expression such as $q_j\in X_i$ indicates the fact that the random variable $X_{ij}$ associated with the item $q_j$ takes realization $x_{ij}=1$ within the random vector $X_i$, or equivalently that the item $q_j$ belongs to the set $X_i\subseteq Q$. Given this notation, the probability of the response pattern $X_i\in 2^Q$ for the i-th individual is then given by the fundamental equation

$$\begin{array}{cc}\hfill P\left(X_i\right|{\mathsf{\Gamma }}_{\mathcal{K}})& =\sum _{K\in \mathcal{K}}P\left(X_i\right|K)\pi \left(K\right)\hfill \end{array}$$

(5)

where one only needs to set a specific shape for the conditional probabilities $P\left(X_i\right|K)$ in order to specify all parameters collected in ${\mathsf{\Gamma }}_{\mathcal{K}}$ and that must be estimated from the data. The most widely used KST model is the BLIM, in which the conditional probabilities $P\left(X_i\right|K)$ are written as the product of two sets of parameters

$$\begin{array}{cc}\hfill P\left(X_i\right|K)& =\prod _{q\in X_i}{\varphi }_{q,K}\prod _{q^{\prime }\in Q\setminus X_i}(1-{\varphi }_{q^{\prime },K}),\hfill \end{array}$$

(6)

where

$$\begin{array}{cc}\hfill {\varphi }_{q,K}& =\left\{\begin{array}{cc}1-{\beta }_q\hfill & \mathrm{if}q\in K\hfill \\ {\eta }_q\hfill & \mathrm{if}q\notin K.\hfill \end{array}\right.\hfill \end{array}$$

(7)

where lucky guesses $\eta =\left\{{\eta }_q\right\}$ model correct responses even though an item has not been mastered, whereas careless errors $\beta =\left\{{\beta }_q\right\}$ model incorrect responses even though an item has been mastered. Since ${\mathsf{\Gamma }}_{\mathcal{K}}=\{\pi ,\beta ,\eta \}$, the BLIM has therefore a total of $\left|\mathcal{K}\right|-1+2\left|Q\right|$ parameters to be estimated from the data.

2.2.2. Simple Learning Model (SLM)

As mentioned above, the state probabilities $\pi$ in a PKS are typically treated as a parameter to be estimated with the consequence that large domains Q might imply large structures and, therefore, many state probability parameters. One way to reduce their number is to further constrain the probabilities $\pi$. An example is provided by the SLM (see, e.g., [9] (p. 199)), which is used to factorize the probability distributions in learning spaces, which are particularly regular structures that allow for multiple solution strategies while requiring that items are mastered one at a time. The SLM is formally given by

$$\pi \left(K\right)=\prod _{q\in K}{g}_q\prod _{q^{\prime }\in K^{\mathcal{O}}}(1-g_{q^{\prime }})$$

(8)

where $g_q\in ]0,1[$ is a parameter expressing the probability of mastering the item q, and where $K^{\mathcal{O}}$ is called the outer fringe of the knowledge state K and is defined as the set of items that can be learned next (one at a time) when being in the knowledge state K, that is

$$K^{\mathcal{O}}:=\{q\in Q\setminus K:K\cup \left\{q\right\}\in \mathcal{K}\}.$$

(9)

In essence, the SLM factorizes the probability $\pi \left(K\right)$ of each state $\left(K\right)$ into the product of the probabilities of the items that have already been learned (i.e., those in K) and the complementary probabilities of the items that can be learned next (i.e., items in $K^{\mathcal{O}}$). Generalized versions of the SLM have been explored by Noventa et al. [23].

2.2.3. KST-IRT Models

As formally shown by Noventa et al. [18], if both the state probabilities $\pi \left(K\right)$ and the item probabilities $g_q$ in Equations (5) and (8) are extended to encompass some latent variable $\theta \in \mathbb{R}$, so that ${\theta }_i$ expresses the ability of the i-th individual, then the machinery of KST can be used to extend IRT models and to generalize the fundamental Equation (5) and the SLM (8) as

$$\begin{array}{cc}\hfill P\left(X_i\right|{\theta }_i,{\mathsf{\Gamma }}_{\mathcal{K}})& =\sum _{K\in \mathcal{K}}P\left(X_i\right|K)\pi \left(K\right|{\theta }_i)\hfill \end{array}$$

(10)

and

$$\begin{array}{cc}\hfill \pi \left(K\right|{\theta }_i)& =\prod _{q\in K}{g}_q\left({\theta }_i\right)\prod _{q^{\prime }\in K^{\mathcal{O}}}(1-g_{q^{\prime }}\left({\theta }_i\right)),\hfill \end{array}$$

(11)

respectively, where for sake of simplicity within the terms $\pi \left(K\right|{\theta }_i)$ and $g_q\left({\theta }_i\right)$ the parameters vector $\mathsf{\Gamma }$ are omitted. If the $g_q\left({\theta }_i\right)$ functions are identified with the 2PL $P(X_{ij}=1|{\theta }_i,{\mathsf{\Gamma }}_j^2)$ or the RM $P(X_{ij}=1|{\theta }_i,{\mathsf{\Gamma }}_j^1)$, then the state probabilities $\pi \left(K\right|{\theta }_i)$ can be considered state response functions that generalize to the knowledge states $K\in \mathcal{K}$, the IRT idea of an item response function. Most of all, two important consequences follow, which are specific to the power set structure $\mathcal{K}=2^Q$. Firstly, the extended version of the SLM given by Equation (11) provides a generalized version of local stochastic independence in IRT as given by Equation (2) such that, only when the power set $2^Q$ is considered, then LI (2) is returned. Intuitively, (11) generalizes local independence in that it allows for the presence of items that cannot be mastered from a given state because their prerequisites are not satisfied. Secondly, only when the power set $2^Q$ is considered, the combination of the extended versions of the SLM and of the BLIM (given by (10) with conditional probabilities set by (6) as in the BLIM) yields exactly the likelihood of a 4PL IRT model as in Equation (1) in the presence of LI (2) in which the following identifications are performed: $c_j={\eta }_{q_j}$, $d_j={\beta }_{q_j}$, and $g_{q_j}\left({\theta }_i\right)=P(X_{ij}=1|{\theta }_i,{\mathsf{\Gamma }}_j^2)$ so that ${\mathsf{\Gamma }}_{\mathcal{K}}={\mathsf{\Gamma }}^4$. These results show that LI (2) is obtained in the KST-IRT approach only in power set case, while if an arbitrary structure is considered, LI does not hold anymore, and one needs to consider the fundamental Equation (10) (or the generalized version of LI as provided by Equation (11)) to account for the LD captured by the structure. As a consequence, alternatively to the Rasch plus LI approach that is used in IRT, one can capture a Guttman scalogram by considering a graded chain $\mathcal{C}$ as given by Equation (4), together with the latent trait extended fundamental KST equation (10) in conjunction (or not) with the latent trait extended SLM given by Equation (11). Specifically, in what follows, we will compare the IRT approach with two KST-IRT models as provided by Equation (10). The first follows the SLM (11), while the latter assumes a different shape of the state response functions $\pi \left(K\right)|{\theta }_i$ and is discussed in the following subsection.

2.2.4. Logistic Knowledge Structure

A perfect example of a KST-IRT model is provided by a Logistic Knowledge Structure ([24], LKS). If one indeed considers an arbitrary knowledge structure $\mathcal{K}$, one can define an LKS as a 5-tuple $(Q,\mathcal{K},b,P,\pi )$ where $b:Q\to \mathbb{R}$, $\pi :\mathcal{K}\times \mathbb{R}\to (0,1)$ are such that

$$\begin{array}{cc}\hfill P\left(X_i\right|{\theta }_i)& =\sum _{K\in \mathcal{K}}P\left(X_i\right|K)\pi \left(K\right|{\theta }_i)=\sum _{K\in \mathcal{K}}P\left(X_i\right|K)\frac{exp\left[{\displaystyle \sum _{q_j\in K}}({\theta }_i-b_{q_j})\right]}{{\displaystyle \sum _{L\in \mathcal{K}}}exp\left[{\displaystyle \sum _{q_j\in L}}({\theta }_i-b_{q_j})\right]}\hfill \end{array}$$

(12)

where $P\left(X_i\right|K)$ is given by the BLIM (6). The LKS can be seen as a generalization of the partial credit model (PCM) to an arbitrary structure rather than a graded chain [18]. Finally, notice that while the SLM approach of Equation (11) resembles an IRT sequential/step model, the LKS of Equation (12) resembles a polytomous IRT model. For this reason, they provide two different ways of modeling LD that are considered in what follows.

3. Estimation Methods

Since the introduction of the MML [25] and MML-EM [26] that remedy the inconsistent parameter estimates of the joint maximum likelihood estimation (JMLE, [21]), the MML-EM has served as a standard method to estimate the parameters of many psychometric models [27]. In order to separate the item parameters from the ability parameter, the joint distribution is marginalized over a known examinee ability distribution.

One known shortcoming of the MML-EM is that when this distributional assumption is violated, the estimates can produce undesirably deviating results [28,29]. To anticipate the deviation, one can restrict the estimates within a reasonable range by imposing a known distribution for the item parameters. This Bayesian approach [30,31] was further developed by incorporating the Markov chain Monte Carlo (MCMC) simulation with the Bayesian estimation (Metropolis-Hastings, Robbins-Monro, [32,33] blocked Metropolis algorithm [34,35]). MML-EM and the MCMC methods are generally more computationally intensive than the conditional maximum likelihood estimation [36] used for the RM. However, they allow for more flexible and accurate estimation of the parameters in more complex models such as KST-IRT models.

To this end, two estimation methods, the MML-EM, and a Bayesian method, EM-with-Gibbs, were used to recover the item parameters of the KST-IRT models. The item parameters to be estimated are, per each item j, the BLIM parameters, ${\eta }_j$ and ${\beta }_j$, and the item difficulties $b_j$. In the first subsection, we derive the marginalized likelihood for the KST-IRT models and define the necessary parts uniquely facilitated in the MML estimation. In the following subsections, the steps for implementing each MML-EM and the EM-with-Gibbs sampling method were presented.

3.1. Marginalized Likelihood for KST-IRT Models

Following the same notation used in Section 2, we denote by $\mathsf{\Gamma }$ the collection of all item parameters and by $\xi \in \mathsf{\Gamma }$ an arbitrary target parameter. To recover the item parameters from the response patterns utilizing the marginalized likelihood, the conducted estimation procedure resembles that of Bock and Aitkin [26]. Under the KST-IRT model, the conditional probability of the response pattern $X_i$ is given by (10). For an examinee sampled from a population with a known distribution, $f\left(\theta \right)$, the unconditional probability of the response pattern, $X_i$, for the i-th individual is given by:

$$P\left(X_i\right|\mathsf{\Gamma })=\int P\left(X_i\right|{\theta }_i,\mathsf{\Gamma })f\left({\theta }_i\right)d{\theta }_i.$$

(13)

The log-likelihood is denoted $logL\left(X\right|\mathsf{\Gamma })={\sum }_{i}logP\left(X_i\right|\mathsf{\Gamma })$ where $X=[X_1,\dots ,X_N]$ is the full data matrix. Then, to derive the marginal likelihood, we take the first derivative of the log-likelihood with regard to the target parameter $\xi$. By using the Bayes’ rule that $f\left({\theta }_i\right|X_i,\mathsf{\Gamma })=\frac{P\left(X_i\right|{\theta }_i,\mathsf{\Gamma })f\left({\theta }_i\right)}{P\left(X_i\right|\mathsf{\Gamma })}$ and using the relation

$$\frac{\partial }{\partial \xi }P\left(X_i\right|{\theta }_i,\mathsf{\Gamma })=\frac{\partial }{\partial \xi }\left[logP\left(X_i\right|{\theta }_i,\mathsf{\Gamma })\right]P\left(X_i\right|{\theta }_i,\mathsf{\Gamma }),$$

(14)

we have the following convenient form where the posterior distribution, $f\left({\theta }_i\right|X_i,\mathsf{\Gamma })$, replaces the prior distribution of the ability parameter, $f\left({\theta }_i\right)$, in the formula:

$$\begin{array}{cc}\hfill \frac{\partial }{\partial {\xi }_j}logL\left(X\right|\mathsf{\Gamma })& =\sum _i^N\frac{1}{P\left(X_i\right|\mathsf{\Gamma })}\int \left[\frac{\partial }{\partial {\xi }_j}P\left(X_i\right|{\theta }_i,\mathsf{\Gamma })\right]f\left({\theta }_i\right)d{\theta }_i\hfill \\ \hfill & =\sum _i^N\int \left[\frac{\partial }{\partial {\xi }_j}logP\left(X_i\right|{\theta }_i,\mathsf{\Gamma })\right]f\left({\theta }_i\right|X_i,\mathsf{\Gamma })d\theta \hfill \end{array}$$

(15)

The next step is to attain the first derivative of $P\left(X_i\right|{\theta }_i,\mathsf{\Gamma })$. Let $x_{ij}$ denote the dichotomous response of individual i to item j, and let $P\left(x_{ij}\right|{\theta }_i,{\mathsf{\Gamma }}_j^n)$ and $Q\left(x_{ij}\right|{\theta }_i,{\mathsf{\Gamma }}_j^n)$ denote the probabilities of correct and incorrect responses given ${\theta }_i$, then by LI (2) we can write

$$\begin{array}{cc}\hfill \frac{\partial }{\partial {\xi }_j}logP\left(X_i\right|{\theta }_i,\mathsf{\Gamma })& =\frac{\partial }{\partial \xi }log\left[\prod _j^{J}P{\left(x_{ij}\right|{\theta }_i,{\mathsf{\Gamma }}_j^n)}^{x_{ij}}Q{\left(x_{ij}\right|{\theta }_i,{\mathsf{\Gamma }}_j^n)}^{1-x_{ij}}\right]\hfill \\ \hfill & =\sum _j^J\frac{x_{ij}-P\left(x_{ij}\right|{\theta }_i,{\mathsf{\Gamma }}_j^n)}{P\left(x_{ij}\right|{\theta }_i,{\mathsf{\Gamma }}_j^n)Q\left(x_{ij}\right|{\theta }_i,{\mathsf{\Gamma }}_j^n)}\frac{\partial P\left(x_{ij}\right|{\theta }_i,{\mathsf{\Gamma }}_j^n)}{\partial {\xi }_j}.\hfill \end{array}$$

(16)

So far, the above procedure provided by Bock and Lieberman [25] for decomposing the log-likelihood is common in IRT. However, with KST-IRT models, $P\left(X_i\right|{\theta }_i,{\mathsf{\Gamma }}_{\mathcal{K}})$ as given by the fundamental equation (10) expands accordingly to the specified knowledge structure. Therefore, Equation (16) needs to be replaced by the corresponding element of the Jacobian matrix, which now depends on both the choice of a structure $\mathcal{K}$ and state response probability $\pi \left(K\right|{\theta }_i)$. Indeed, in attaining the first derivatives of the difficulty item parameters, one would obtain different results for the SLM and the LKS. Similarly, the Jacobian matrix has different sizes for different structures, i.e., $(J+1)\times \left(3J\right)$ for a graded chain and $\left(2^J\right)\times \left(3J\right)$ for a power set. To illustrate these differences, let us consider a set of two items, $Q=\{q_1,q_2\}$ where graded chain yields $\mathcal{C}=\{\varnothing ,\left\{q_1\right\},Q\}$ and the power set yields $2^Q=\{\varnothing ,\left\{q_1\right\},\left\{q_2\right\},Q\}$. Let then $x_{ij}$ denote, as usual, the binary response for a choice of the item response functions $g_{q_j}\left({\theta }_i\right)=P(X_{ij}=1|{\theta }_i,{\mathsf{\Gamma }}_j^1)$ as in the RM, and let $h_{q_j}\left({\theta }_i\right)=P(X_{ij}=0|{\theta }_i,{\mathsf{\Gamma }}_j^1)$ denote the response function for the incorrect response. Then the first derivatives of each item parameter for item $j=1$, when the state response function is modeled according to the SLM (11), yields for the graded chain $\mathcal{C}$   

$$\frac{\partial }{\partial {\xi }_1}{P}_{\mathrm{SLM}}\left(X_i\right|{\theta }_i,{\mathsf{\Gamma }}_{\mathcal{C}})=\left\{\begin{array}{cc}{(-1)}^{x_{i1}+1}{\eta }_{q_2}^{x_{i2}}{(1-{\eta }_{q_2})}^{1-x_{i2}}{h}_{q_1}\left({\theta }_i\right)\hfill & \xi =\eta \hfill \\ {(-1)}^{x_{i1}}{(1-{\beta }_{q_2})}^{x_{i2}}{\beta }_{q_2}^{1-x_{i2}}{g}_{q_1}\left({\theta }_i\right)g_{q_2}\left({\theta }_i\right)\hfill & \xi =\beta \hfill \\ \begin{array}{c}\left[P\right(X_i|\varnothing )-P\left(X_i\right|\left\{q_1\right\})h_{q_2}\left({\theta }_i\right)\hfill \\ \hspace{1em}-P\left(X_i\right|Q)g_{q_2}\left({\theta }_i\right)]g_{q_1}\left({\theta }_i\right)h_{q_1}\left({\theta }_i\right)\hfill \end{array}\hfill & \xi =b\hfill \end{array}\right.$$

and for the power set $2^Q$

$$\frac{\partial }{\partial {\xi }_1}{P}_{\mathrm{SLM}}\left(X_i\right|{\theta }_i,{\mathsf{\Gamma }}_{2^Q})=\left\{\begin{array}{cc}\begin{array}{c}{(-1)}^{x_{i1}+1}{h}_{q_1}\left({\theta }_i\right)[{\eta }_{q_2}^{x_{i2}}{(1-{\eta }_{q_2})}^{1-x_{i2}}{h}_{q_2}\left({\theta }_i\right)\hfill \\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}+{(1-{\beta }_{q_2})}^{x_{i2}}{\beta }_{q_2}^{1-x_{i2}}]\hfill \end{array}\hfill & \xi =\eta \hfill \\ \begin{array}{c}{(-1)}^{x_{i1}}{g}_{q_1}\left({\theta }_i\right)[{\eta }_{q_2}^{x_{i2}}{(1-{\eta }_2)}^{1-x_{i2}}{h}_{q_2}\left({\theta }_i\right)\hfill \\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}+{(1-{\beta }_{q_2})}^{x_{i2}}{\beta }_{q_2}^{1-x_{i2}}{g}_{q_2}\left({\theta }_i\right)]\hfill \end{array}\hfill & \xi =\beta \hfill \\ \begin{array}{c}{g}_{q_1}\left({\theta }_i\right)h_{q_1}\left({\theta }_i\right)[(P\left(X_i\right|\varnothing )h_{q_2}\left({\theta }_i\right)-P\left(X_i\right|\left\{q_1\right\})g_{q_2}\left({\theta }_i\right))\hfill \\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}-(P\left(X_i\right|\left\{q_1\right\})h_{q_2}\left({\theta }_i\right)-P\left(X_i\right|Q)g_{q_2}\left({\theta }_i\right))]\hfill \end{array}\hfill & \xi =b\hfill \end{array}\right.$$

where $P_{\mathrm{SLM}}\left(X_i\right|{\theta }_i,{\mathsf{\Gamma }}_{\mathcal{C}})$ and $P_{\mathrm{SLM}}\left(X_i\right|{\theta }_i,{\mathsf{\Gamma }}_{2^Q})$ take the form of Equation (10). Specifically, the first term of each sum in Equation (10), $P\left(X_i\right|\mathcal{K})$ contains $\eta$ and $\beta$ in the form of BLIM (6) and the second term corresponding to (11) contains the difficulty parameters, b, where the factorized $g_q$ follows the form of RM.

Similarly, the first derivatives of each item parameter for item $j=1$, when the state response function is modeled according to the LKS (12), yields for the graded chain $\mathcal{C}$

$$\frac{\partial }{\partial {\xi }_1}{P}_{\mathrm{LKS}}\left(X_i\right|{\theta }_i,{\mathsf{\Gamma }}_{\mathcal{C}})=\left\{\begin{array}{cc}{(-1)}^{x_{i1}}{\eta }_{q_2}^{x_{i2}}{(1-{\eta }_{q_2})}^{1-x_{i2}}\pi (\varnothing |{\theta }_i)\hfill & \xi =\eta \hfill \\ {(-1)}^{x_{i1}}{(1-{\beta }_{q_2})}^{x_{i2}}{\beta }_{q_2}^{x_{i2}}\pi \left(Q\right|{\theta }_i)\hfill & \xi =\beta \hfill \\ \begin{array}{c}P(X|\varnothing )\left(\pi (\varnothing |{\theta }_i)(\pi \left(q_1\right|{\theta }_i)+\pi \left(Q\right|{\theta }_i))\right)\hfill \\ \hspace{1em}\hspace{1em}-P\left(X_i\right|q_1)\left(\pi (\varnothing |{\theta }_i)\pi \left(q_1\right|{\theta }_i)\right)\hfill \\ \hspace{1em}\hspace{1em}-P\left(X_i\right|Q)\left(\pi (\varnothing |{\theta }_i)\pi \left(Q\right|{\theta }_i)\right)\hfill \end{array}\hfill & \xi =b\hfill \end{array}\right.$$

and for the power set $2^Q$

$$\frac{\partial }{\partial {\xi }_1}{P}_{\mathrm{LKS}}\left(X_i\right|{\theta }_i,{\mathsf{\Gamma }}_{2^Q})=\left\{\begin{array}{cc}{(-1)}^{x_{i1}+1}\left[{\eta }_2^{x_{i2}}{(1-{\eta }_2)}^{1-x_{i2}}\pi (\varnothing |{\theta }_i)+{(1-{\beta }_2)}^{x_{i2}}{\beta }_2^{1-x_{i2}}\pi \left(q_2\right|{\theta }_i)\right]\hfill & \xi =\eta \hfill \\ {(-1)}^{x_{i1}}\left[{\eta }_2^{x_{i2}}{(1-{\eta }_2)}^{1-x_{i2}}\pi (\varnothing |{\theta }_i)+{(1-{\beta }_2)}^{x_{i2}}{\beta }_2^{1-x_{i2}}\pi \left(Q\right|{\theta }_i)\right]\hfill & \xi =\beta \hfill \\ \begin{array}{c}[(P\left(X_i\right|\varnothing )\pi (\varnothing |{\theta }_i)+P\left(X_i\right|q_2)\pi \left(q_2\right|{\theta }_i))\left(\pi \left(q_1\right|{\theta }_i)\pi \left(Q\right|{\theta }_i)\right)]\hfill \\ \hspace{1em}-\left[(P\left(X_i\right|q_1)\pi \left(q_1\right|{\theta }_i)+P\left(X_i\right|Q)\pi \left(Q\right|{\theta }_i))(\pi (\varnothing |{\theta }_i)+\pi \left(q_2\right|{\theta }_i))\right]\hfill \end{array}\hfill & \xi =b\hfill \end{array}\right.$$

where the second term of each sum in $P_{\mathrm{LKS}}\left(X_i\right|{\theta }_i,{\mathsf{\Gamma }}_{\mathcal{C}})$ and $P_{\mathrm{LKS}}\left(X_i\right|{\theta }_i,{\mathsf{\Gamma }}_{2^Q})$ now correspond to the divide-by-total approach (12).

As a final step, for marginalizing the likelihood over the ability parameter, the integral in (16) is replaced with a summation. Bock and Lieberman [25] used numerical quadrature as an approximation for the integration. The assumed continuous density for the ability parameter is filled with rectangles of equal widths to be summed over, and the midpoints of the rectangles are nodes denoted as $Y_k$, $k\in \{1,...,s\}$, and each node is associated with a quadrature weight, $A\left(Y_k\right)$. Typically, a normal density is applied for the distribution of the individual ability parameter. We can now compute the posterior distribution of the finite partitions of $\theta$ at each node,

$$P\left(Y_k\right|X)=\frac{{\sum }_{K}P\left(X\right|K)\pi \left(K\right|\theta )A\left(Y_k\right)}{{\sum }_k^s{\sum }_{K}P\left(X\right|K)\pi \left(K\right|\theta )A\left(Y_k\right)}.$$

(17)

Similarly to Bock and Lieberman [25], $x_{ij}$ and $P\left(x_{ij}\right|{\theta }_i,{\mathsf{\Gamma }}_j^n)$ in Equation (16) are replaced by

$${\overline{n}}_k=\sum _i^{N}P\left(Y_k\right|X_i)=\sum _i^N\left[\frac{{\sum }_{K}P\left(X_i\right|K)\pi \left(K\right|{\theta }_i)A\left(Y_k\right)}{{\sum }_k^q{\sum }_{K}P\left(X_i\right|K)\pi \left(K\right|{\theta }_i)A\left(Y_k\right)}\right]$$

(18)

and

$${\overline{r}}_{jk}=\sum _i^N{x}_{ij}P\left(Y_k\right|X_i)=\sum _i^N\left[\frac{x_{ij}{\sum }_{K}P\left(X_i\right|K)\pi \left(K\right|\theta )A\left(Y_k\right)}{{\sum }_k^q{\sum }_{K}P\left(X_i\right|K)\pi \left(K\right|\theta )A\left(Y_k\right)}\right],$$

(19)

respectively. These two new artificial measurements given by the observed responses act as sufficient statistics as they represent the expected number of individuals at ability level $Y_k$ and the expected number of correct responses at ability level $Y_k$ for item j, respectively.

3.2. MML-EM

Estimating the item parameters with the MML-EM involves iterative computation of expectation and maximization. The following two-step procedure is repeated until a predetermined convergence criterion is met:

• E-step: with the response patterns X, let ${\eta }^{\left(0\right)}$, ${\beta }^{\left(0\right)}$, and $b^{\left(0\right)}$ be the collections of initial parameters, we compute on each node $Y_k$ the expected number ${\overline{n}}_k$ of individuals and the expected number of correct responses ${\overline{r}}_{jk}$ for item j.
• M-step: we solve the first derivatives for each item via a numerical method such as Newton-Raphson’s.

In the expectation step, the initial parameters for the BLIM, ${\eta }^{\left(0\right)}$ and ${\beta }^{\left(0\right)}$, can be drawn from a uniform distribution with a range of $]0.05,0.4[$. The initial values for the difficulty parameters $b_j^{\left(0\right)}$ can be drawn from a uniform distribution with a range of $]-3,3[$. For the prior distribution of the ability parameter, we consider a standard normal distribution.

In the maximization step, the parameters are updated to maximize the expected log likelihood. Once the difficulty parameters are estimated, the solutions to the first derivatives can be attained for each item parameter independently as the artificial quantities ${\overline{n}}_k$ and ${\overline{r}}_{jk}$ provide sufficient statistics for the parameter estimates. As each iteration of the MML-EM can be solved quickly with the closed-form first derivatives of the marginalized likelihoods, the short estimation time is the advantage of this method.

3.3. EM-with-Gibbs

The Gibbs sampling technique is an MCMC algorithm that draws samples from full conditional distributions. As a special case of Metropolis-Hastings, it leverages known prior distributions to approximate the posterior distribution. This approach can be viewed as a generalized EM algorithm, where the standard expectation step is replaced by drawing samples from conditional distributions. In particular, when the distribution of item parameters is known, the Gibbs sampling procedure can be effectively combined with the EM algorithm.

The present study proposes an approach for estimating the parameters of the KST-IRT models by combining the Gibbs sampling method with the EM algorithm. This approach differs from the MML-EM in that the BLIM parameters are drawn from the full conditional distributions. The log-likelihoods of the response patterns are computed with the drawn sample instead of using the more rigidly computed artificial measurements as (18) and (19) as in the MML-EM. The resulting parameter samples are subsequently used to update the joint distribution in the next iteration of the algorithm.

The difficulty parameter is constrained to reflect the response behavior of the graded chain in the IRT case, meaning that the difficulty parameters are strictly increasing. To enforce this constraint, a multivariate normal distribution is used with means that impose the order of increasing difficulty parameters. For the BLIM parameters, which are supported on the interval $]0,1[$, a Beta (1,1) flat prior can be assumed. Once the frequencies of guessing and careless error incidences are incorporated into the prior distribution, the resulting posterior distribution with updated parameters can be used to draw estimates of $\eta$ and $\beta$. In the likelihoods that follow, ${\widehat{K}}_i^{\max }$ represents the knowledge state that maximizes $\pi \left(K\right|{\theta }_i)$ for the i-th individual given the response pattern X:

$$\begin{array}{cc}\hfill & L\left(\eta \right|X_i,\beta ,b)\propto {\eta }_{q_j}^{{\sum }_i^N{x}_{ij}[q_j\notin {\widehat{K}}_i^{\max }]}{(1-{\eta }_{q_j})}^{{\sum }_i^N(1-x_{ij})[q_j\notin {\widehat{K}}_i^{\max }]}\hfill \\ \hfill & L\left(\beta \right|X_i,\eta ,b)\propto {(1-{\beta }_{q_j})}^{{\sum }_i^N{x}_{ij}[q_j\in {\widehat{K}}_i^{\max }]}{\beta }_{q_j}^{{\sum }_i^N(1-x_{ij})[q_j\in {\widehat{K}}_i^{\max }]}\hfill \end{array}$$

(20)

where $[q_j\notin {\widehat{K}}_i^{\max }]$ is the Iverson brackets that attain value one when the logical condition is satisfied and zero otherwise. Normalizing the conditional likelihoods in (20), we attain the Beta distribution where the BLIM parameters can be drawn from, that is

$$\begin{array}{cc}\hfill & {\eta }_{q_j}\sim Beta(\sum _i^N{x}_{ij}[q_j\notin {\widehat{K}}_i^{\max }],\sum _i^N[q_j\notin {\widehat{K}}_i^{\max }])\hfill \\ \hfill & {\beta }_{q_j}\sim Beta(\sum _i^N(1-x_{ij})[q_j\in {\widehat{K}}_i^{\max }],\sum _i^N[q_j\in {\widehat{K}}_i^{\max }]).\hfill \end{array}$$

(21)

Due to the sampling nature of the Gibbs method, the iterations are not stopped by convergence criteria. Instead, a fixed number of iterations are repeated as follows:

• Step 1: with the response patterns, X, values ${\theta }_i$ are sampled from the posterior distribution of the ability parameter given by the computed nodes $A\left(Y_k\right)$ for some fine grid $k\in \{1,...,s\}$ with the vectors of initial item parameters ${\eta }^{\left(0\right)}$, ${\beta }^{\left(0\right)}$, and $b^{\left(0\right)}$.
• Step 2: draw a predetermined batch size of difficulties b from a multivariate normal distribution with a mean vector $\mu =[{\mu }_1,...,{\mu }_J]$ that previously maximized the joint likelihood.
• Step 3: draw from the conditional likelihood of the BLIM parameters and update the current estimate.

In the first step, the use of the grid of $\theta$ and sampling of the node that maximizes the likelihood of the response patterns act as the computation of the expected number of individuals at each node in the MML-EM algorithm. For the difficulty parameter, the sampling procedure differs from those of the BLIM parameters that are drawn from a full conditional distribution as suggested by the traditional Gibbs sampling. Specifically, difficulty parameters are not drawn from each one from their own distribution, but they are instead drawn from a multivariate distribution so that they are used as an entirety of J-tuple vector for computing the state response function. Given an informative prior, it is more sensible to draw a batch of J-tuple vectors and select the one that returns the maximum likelihood at each draw. Then, the BLIM parameters are drawn with the standard Gibbs sampling method, where the Beta distribution parameters are updated at each iteration. Finally, in MCMC simulations, the burn-in period is the number of initial iterations that are discarded to ensure that the chain has converged to the stationary distribution. During the burn-in period, the chain may be in a transient state, which means that it has not yet reached its equilibrium distribution. In this application, 1000 iterations were used to determine the burn-in period.

4. Simulation Studies

In the first simulation study, we evaluate the efficacy of the parameter estimation methods by comparing the recovery performance of KST-IRT model parameters. Specifically, we compare the estimates and the root mean squared error (RMSE) of the KST-IRT model parameters using different knowledge structures and state response functions. A further comparison of the estimation performance is demonstrated by misspecifying the assumed knowledge structure (i.e., either a graded chain or a power set) and/or the state response function (i.e., either a SLM or a LKS). In the second simulation study, we examine the model fit of the RM and the KST-IRT models when the data-generating process and fitted model are interchanged. In other words, the response pattern generated from RM is fitted with KST-IRT models and vice versa. As RM and KST-IRT models, as discussed in Section 2, assume completely different processes underlying the generation of unlikely data patterns (i.e., non-Guttman patterns of responses), this latter simulation study allows for assessing the models’ ability to capture the underlying structure of the data and to understand which form of modeling of a Guttman scale is more robust to violations of its own assumptions.

The true item parameters were established to mimic a Guttman scalogram. In both simulation studies, the difficulty parameters of the items are arranged in increasing order. This arrangement was chosen to generate responses that are consistent with the assumed characteristic of the Guttman scale according to the IRT perspective, where each successive item is expected to be more difficult than the previous one. In comparison, the KST-IRT model explicitly accounts for local dependence of the items through the knowledge structure (i.e., the graded chain), and for such a reason, it would not need to a priori establish an increasing order for the difficulties. Nonetheless, the assumption was taken to meet the usual requirements of IRT specifications. It should also be stressed that, since the condition of LI is obtained in KST-IRT models when the knowledge structure is a power set, the comparison between RM plus LI vs SLM plus power set actually corresponds to comparing a Rasch model as it is traditionally applied in IRT with a 4PL model in which the Birnbaum parameter is set to one.

In the two simulation studies, a total of five models were considered: one IRT model, the Rasch Model (RM), and four KST-IRT models. The KST-IRT models are identified by their respective knowledge structures and state response functions. Each KST-IRT model is labeled by the assumed knowledge structure: either a graded chain denoted as $\mathcal{C}$ or a power set denoted as $2^Q$ where Q represents the full domain. Additionally, the choice of the state response function is indicated as either SLM or LKS. In order to simplify the interpretations of the models for the reader, we summarize here which different combinations actually correspond.

• $RM$: this is the traditional assumption of IRT models in which a Rasch model is considered in conjunction with the assumption of LI.
• $\mathcal{C}+SLM$: this is the first KST-IRT version of a Guttman scale in which a graded chain is used to model the scalogram, and the state response functions are factorized by means of the generalized version of LI provided by Equation (11). The model is structurally similar to a sequential/step IRT model. The conditional error parameters provide the noise generating the non-Guttman pattern of responses.
• $\mathcal{C}+LKS$: this is a second KST-IRT version of a Guttman scale in which a graded chain is used to model the scalogram, but this time the state response functions are not factorized. Since a LKS (12) is used, the state response function rather takes the form of a polytomous IRT model applied to the Guttman scale. As previously, the conditional error parameters provide the noise generating the non-Guttman pattern of responses.
• $2^Q+SLM$: this is the KST-IRT version of a 4PL model plus LI (but the discrimination parameters are set to one).
• $2^Q+LKS$: by construction, this model is equivalent to the former, so it was not reported in the results of the second simulation study since the estimates are essentially the same, and the models generate essentially the same data.

In both simulation studies, each design was repeated $N_{rep}=50$ times to compute the average of the estimates and the RMSE to evaluate the dispersion of the estimate distribution for a target parameter ${\xi }_j$. Preliminary experiments with smaller and larger set of replications indicated that augmenting the number of replications beyond 50 did not yield a substantial reduction in their respective RMSE or significantly enhance the precision of the estimates. Consequently, $N_{rep}=50$ was adopted as a reasonable compromise between acquiring reliable results and managing computational burden. The RMSE was calculated with the following formula:

$$RMSE=\sqrt{\frac{{\sum }_k^{N_{rep}}{({\xi }_j-{\widehat{\xi }}_{jk})}^2}{N_{rep}}}$$

(22)

where ${\widehat{\xi }}_{jk}$ represents the k-th replication of the parameter estimate.

The statistical computations in this study were executed using the programming language R and its extension, Rcpp, which utilizes C++. To compute the difficulty parameters for the RM estimates, we employed the rasch function from the ltm (version 1.2-0) package. For the estimation of the KST-IRT parameters, b, $\eta$, and $\beta$, the two introduced estimation procedures, MML-EM and EM-with-Gibbs, were implemented in R. The simulations and computations were performed on a computing system with an Intel Core i7-10610U CPU, operating at a speed of 1.80 GHz to 2.30 GHz and 32 GB of random access memory (RAM).

The simulation time for both MML-EM and EM-with-Gibbs is heavily influenced by the length of the assessment and the knowledge structure. It is vital to note that computation time can escalate rapidly as the number of items increases. Among the dimensions of experimentation in the simulation, the most significant contrast in computation time was observed between the estimation methods. The computation time of MML-EM is primarily contingent upon the duration of Newton-Raphson iterations, while EM-with-Gibbs is contingent upon the specified run length of the chain. Specifically, for the simulation, the average computation times of MML-EM under SLM and LKS were 0.87 and 0.74 h, respectively. Correspondingly, the average computation times of EM-with-Gibbs under SLM and LKS were 1.24 and 1.23 h.

4.1. Simulation Study 1

The objective of this study is to assess the accuracy of item parameter recovery by means of the MML-EM and EM-with-Gibbs sampling methods under correct and incorrect model specifications. To this end, we generated response patterns for $N=10,000$ individuals with $\theta$ randomly sampled from a standard normal distribution. Each individual responded to $J=5$ items, with item parameters assumed as follows: for the difficulty parameters, $b=(-2,-1,0,1,2)$, for the lucky guess parameter, $\eta =(0.129,0.100,0.074,0.115,0.096)$ and careless error parameters, $\beta =(0.158,0.112,0.063,0.148,0.147)$. The state response function, $\pi \left(K\right|{\theta }_i)$, was computed based on two knowledge structures: power set, $2^Q$, and graded chain, $\mathcal{C}$, where ${\eta }_{q_j}$ and ${\beta }_{q_j}$ correspond to the guessing and slipping parameters. For the power set $2^Q$, Q denotes the full domain where the considered domain for this simulation is $Q=\{q_1,q_2,q_3,q_4,q_5\}$ for $J=5$.

A $2\times 2\times 2$ design was conducted using two computational methods, MML-EM and EM-with-Gibbs, two state response functions, SLM and LKS, and two knowledge structures, $\mathcal{C}$ and $2^Q$, where each of the five items has three item parameters, i.e., ${\mathsf{\Gamma }}_j=\{{\eta }_{q_j},{\beta }_{q_j},b_j\}$.

Parameter recovery under correctly specified knowledge structure and state response probability is presented in

Table 1. Parameter estimation of KST-IRT models under correctly specified structure $\mathcal{K}$ and state response probability $\pi \left(K\right|\theta )$.

			MML-EM					EM-with-Gibbs
			$\mathit{j}$					$\mathit{j}$
${\mathit{\xi }}_{\mathit{j}}$	$\mathit{\pi }\left(\mathit{K}\right|\mathit{\theta })$	$\mathcal{K}$	1	2	3	4	5	1	2	3	4	5
$b_j$	SLM	$\mathcal{C}$	−1.94 (0.07)	−1.05 (0.06)	0.05 (0.06)	1.03 (0.04)	1.94 (0.05)	−2.03 (0.06)	−1.04 (0.05)	0.02 (0.03)	1.03 (0.05)	2.04 (0.06)
	$2^Q$	−1.97 (0.08)	−1.05 (0.06)	0.02 (0.04)	1.02 (0.04)	1.95 (0.07)	−2.03 (0.04)	−1.02 (0.03)	0.03 (0.03)	1.03 (0.06)	2.03 (0.04)
	LKS	$\mathcal{C}$	−1.95 (0.06)	−0.99 (0.02)	−0.02 (0.04)	1.04 (0.05)	1.96 (0.06)	−1.96 (0.07)	−0.98 (0.05)	0.03 (0.04)	0.96 (0.05)	1.98 (0.03)
	$2^Q$	−1.95 (0.08)	−1.05 (0.08)	0.05 (0.06)	1.04 (0.05)	1.98 (0.05)	−1.97 (0.06)	−1.05 (0.07)	0.02 (0.03)	1.04 (0.05)	1.96 (0.04)
${\eta }_{q_j}$	SLM	$\mathcal{C}$	0.10 (0.06)	0.13 (0.05)	0.10 (0.03)	0.14 (0.04)	0.12 (0.05)	0.15 (0.04)	0.12 (0.04)	0.06 (0.03)	0.10 (0.03)	0.07 (0.02)
	$2^Q$	0.16 (0.07)	0.13 (0.05)	0.09 (0.06)	0.09 (0.04)	0.13 (0.05)	0.11 (0.05)	0.08 (0.04)	0.05 (0.04)	0.14 (0.05)	0.08 (0.02)
	LKS	$\mathcal{C}$	0.11 (0.06)	0.12 (0.05)	0.09 (0.05)	0.10 (0.04)	0.13 (0.05)	0.11 (0.04)	0.12 (0.03)	0.06 (0.03)	0.10 (0.04)	0.08 (0.02)
	$2^Q$	0.11 (0.03)	0.08 (0.02)	0.10 (0.03)	0.13 (0.05)	0.11 (0.04)	0.12 (0.05)	0.09 (0.05)	0.09 (0.02)	0.10 (0.04)	0.11 (0.02)
${\beta }_{q_j}$	SLM	$\mathcal{C}$	0.12 (0.06)	0.13 (0.04)	0.08 (0.05)	0.13 (0.03)	0.12 (0.03)	0.19 (0.03)	0.08 (0.01)	0.09 (0.02)	0.12 (0.03)	0.14 (0.04)
	$2^Q$	0.13 (0.03)	0.09 (0.02)	0.04 (0.02)	0.18 (0.04)	0.17 (0.04)	0.18 (0.03)	0.10 (0.02)	0.05 (0.02)	0.13 (0.03)	0.16 (0.03)
	LKS	$\mathcal{C}$	0.13 (0.02)	0.09 (0.04)	0.04 (0.01)	0.15 (0.02)	0.16 (0.03)	0.18 (0.02)	0.13 (0.02)	0.05 (0.01)	0.15 (0.02)	0.13 (0.02)
	$2^Q$	0.13 (0.03)	0.09 (0.02)	0.09 (0.04)	0.17 (0.04)	0.17 (0.05)	0.13 (0.03)	0.09 (0.03)	0.05 (0.01)	0.17 (0.02)	0.13 (0.02)

Note. Each cell contains the average of the simulated estimates and the RMSE of the estimates in the parenthesis. $b_j$, ${\eta }_{q_j}$, and ${\beta }_{q_j}$ represent the difficulty, lucky guess, and careless error parameters for item $q_j$.

. The first column indicates the target parameter ${\xi }_j$ for which we provide the mean estimated values, the second column specifies the state response probability $\pi \left(K\right|\theta )$, and the third column specifies the knowledge structure $\mathcal{K}$ which is paired with the state response probability. Then, every block of five columns contains the mean estimates and RMSE of each parameter ${\widehat{b}}_j$, ${\widehat{\eta }}_{q_j}$, and ${\widehat{\beta }}_{q_j}$, for all the items $j\in \{1,\dots ,5\}$. The two blocks refer to the two estimation methods, MML-EM and EM-with-Gibbs, respectively. In each cell, the first row is the arithmetic mean of the estimates for each condition, and the second in parentheses is the RMSE.

In Table 1, the performance of the MML-EM and the EM-with-Gibbs can be compared in terms of bias and the RMSE. A general finding was that the MML-EM yielded higher absolute bias and RMSE than the EM-with-Gibbs for all parameters. Between SLM and LKS, using the LKS for the state probabilities showed an overall smaller bias than those of the SLM. However, the difference is not too apparent. The absolute bias of b, $\eta$, and $\beta$ were, in general, less than 0.06, 0.035, and 0.04, respectively. The bias and the RMSE behaved similarly between the two knowledge structures, $\mathcal{C}$ and $2^Q$.

Table 2. Parameter estimation of KST-IRT models under fully misspecified structure $\mathcal{K}$ (data-generating (DG) model and fitted model (FM) and state response probability $\pi \left(K\right|\theta )$ with MML-EM estimation.

					MML-EM
${\mathit{\xi }}_{\mathit{j}}$	$\mathit{\pi }\left(\mathit{K}\right|\mathit{\theta })$		$\mathcal{K}$		$\mathit{j}$
	DG	FM	DG	FM	1	2	3	4	5
$b_j$	SLM	LKS	$2^Q$	$\mathcal{C}$	−1.90 (0.19)	−1.10 (0.13)	0.04 (0.05)	1.13 (0.15)	1.83 (0.21)
			$\mathcal{C}$	$2^Q$	−2.21 (0.31)	−1.15 (0.18)	0.03 (0.04)	0.85 (0.16)	1.99 (0.12)
	LKS	SLM	$2^Q$	$\mathcal{C}$	−1.88 (0.21)	−1.16 (0.13)	0.07 (0.06)	1.14 (0.20)	1.80 (0.22)
			$\mathcal{C}$	$2^Q$	−2.24 (0.38)	−1.22 (0.21)	0.06 (0.08)	0.84 (0.16)	2.12 (0.19)
${\eta }_{q_j}$	SLM	LKS	$2^Q$	$\mathcal{C}$	0.14 (0.04)	0.10 (0.03)	0.05 (0.02)	0.13 (0.04)	0.09 (0.03)
			$\mathcal{C}$	$2^Q$	0.15 (0.05)	0.12 (0.04)	0.06 (0.04)	0.06 (0.03)	0.09 (0.03)
	LKS	SLM	$2^Q$	$\mathcal{C}$	0.15 (0.06)	0.09 (0.04)	0.09 (0.03)	0.09 (0.04)	0.09 (0.04)
			$\mathcal{C}$	$2^Q$	0.13 (0.05)	0.12 (0.04)	0.07 (0.02)	0.10 (0.02)	0.11 (0.04)
${\beta }_{q_j}$	SLM	LKS	$2^Q$	$\mathcal{C}$	0.18 (0.05)	0.14 (0.04)	0.03 (0.02)	0.15 (0.03)	0.14 (0.03)
			$\mathcal{C}$	$2^Q$	0.15 (0.04)	0.15 (0.06)	0.10 (0.03)	0.18 (0.05)	0.13 (0.05)
	LKS	SLM	$2^Q$	$\mathcal{C}$	0.17 (0.03)	0.08 (0.03)	0.03 (0.01)	0.12 (0.04)	0.15 (0.02)
			$\mathcal{C}$	$2^Q$	0.14 (0.03)	0.10 (0.03)	0.04 (0.02)	0.13 (0.04)	0.16 (0.04)

Note. Each cell contains the average of the simulated estimates and the RMSE of the estimates in the parenthesis. $b_j$, ${\eta }_{q_j}$, and ${\beta }_{q_j}$ represent the difficulty, lucky guess, and careless error parameters for item $q_j$.

and

Table 3. Parameter estimation of KST-IRT models under fully misspecified structure $\mathcal{K}$ (data-generating (DG) model and fitted model (FM) and state response probability $\pi \left(K\right|\theta )$ with Gibbs sampling estimation.

					EM-with-Gibbs
${\mathit{\xi }}_{\mathit{j}}$	$\mathit{\pi }\left(\mathit{K}\right|\mathit{\theta })$		$\mathcal{K}$		$\mathit{j}$
	DG	FM	DG	FM	1	2	3	4	5
$b_j$	SLM	LKS	$2^Q$	$\mathcal{C}$	−1.94 (0.16)	−1.09 (0.11)	0.02 (0.02)	1.08 (0.07)	1.84 (0.13)
			$\mathcal{C}$	$2^Q$	−2.13 (0.15)	−1.19 (0.12)	0.05 (0.05)	0.91 (0.09)	2.06 (0.08)
	LKS	SLM	$2^Q$	$\mathcal{C}$	−1.90 (0.12)	−1.12 (0.14)	0.05 (0.04)	1.08 (0.11)	1.85 (0.13)
			$\mathcal{C}$	$2^Q$	−2.31 (0.27)	−1.16 (0.14)	0.06 (0.08)	0.96 (0.11)	2.10 (0.09)
${\eta }_{q_j}$	SLM	LKS	$2^Q$	$\mathcal{C}$	0.14 (0.02)	0.11 (0.02)	0.06 (0.01)	0.12 (0.01)	0.09 (0.01)
			$\mathcal{C}$	$2^Q$	0.14 (0.02)	0.13 (0.03)	0.05 (0.01)	0.09 (0.01)	0.09 (0.03)
	LKS	SLM	$2^Q$	$\mathcal{C}$	0.13 (0.02)	0.09 (0.01)	0.09 (0.02)	0.12 (0.03)	0.10 (0.04)
			$\mathcal{C}$	$2^Q$	0.14 (0.03)	0.11 (0.01)	0.05 (0.02)	0.11 (0.01)	0.10 (0.02)
${\beta }_{q_j}$	SLM	LKS	$2^Q$	$\mathcal{C}$	0.16 (0.02)	0.13 (0.02)	0.06 (0.02)	0.15 (0.01)	0.14 (0.03)
			$\mathcal{C}$	$2^Q$	0.15 (0.01)	0.13 (0.02)	0.08 (0.02)	0.16 (0.05)	0.17 (0.02)
	LKS	SLM	$2^Q$	$\mathcal{C}$	0.16 (0.02)	0.10 (0.01)	0.05 (0.01)	0.14 (0.02)	0.15 (0.01)
			$\mathcal{C}$	$2^Q$	0.15 (0.02)	0.10 (0.02)	0.05 (0.01)	0.13 (0.03)	0.15 (0.03)

Note. Each cell contains the average of the simulated estimates and the RMSE of the estimates in the parenthesis. $b_j$, ${\eta }_{q_j}$, and ${\beta }_{q_j}$ represent the difficulty, lucky guess, and careless error parameters for item $q_j$.

display instead the parameter estimation under misspecification of both knowledge structure $\mathcal{K}$ and state response function $\pi \left(K\right|\theta )$, each for MML-EM and EM-with-Gibbs sampling, respectively. Misspecifying both the knowledge structure and the state response function, this simulation investigated the full misspecification of the two components of the KST-IRT model. The first column indicates the target parameter ${\xi }_j$ being estimated. The second and third columns specify the state response probability $\pi \left(K\right|\theta )$ used for data generation (DG) and fitted model (FM). The subsequent fourth and fifth columns indicate the knowledge structure $\mathcal{K}$ used for data generation and the fitted model. Starting from the first row, every two consecutive rows of two-way mismatched knowledge structures pertain to a paired mismatch of the state response function.

Comparing Table 2 and Table 3, the estimates given by EM-with-Gibbs approximate closer to the true parameter values with relatively lower RMSE than those of MML-EM. It is, however, difficult to conclude that the estimates of the EM-with-Gibbs are superior to those of MML-EM in all cases. In terms of bias for the difficulty parameter for all four designs, 16 cells out of 20 of Table 3 have a smaller bias magnitude than those of Table 2 when we compute the between-item RMSE for the difficulty parameter, for instance, Table 2 and Table 3 each yield 0.14 and 0.12. The overall bias and RMSE are greater than the case of correctly specified state response probability $\pi \left(K\right|\theta )$ and knowledge structure $\mathcal{K}$ for both estimation methods. The Gibbs sampling approach consistently performs better than the MML-EM with relatively smaller bias and RMSE, while there are a few instances where MML-EM produces estimates closer to the true value, for instance, the estimate of $b_5$ when the data is generated under $\mathcal{C}+SLM$. Nonetheless, most of these cases appear to fall within the uncertainty range expressed by the RMSE.

It is noteworthy to observe that response patterns generated from a graded chain $\mathcal{C}$ and fitted with a power set $2^Q$ appear to inflate the range of the difficulty parameters in all four designs, whereas response patterns generated from a power $2^Q$ and fitted with a graded chain $\mathcal{C}$ appear to shrink the range of the difficulty parameters. This pattern only shows with the difficulty parameters but not with the conditional error parameters. This appears to be due to the structural role that the difficulty parameters play in the KST-IRT models. Indeed, since they determine the state response probabilities, they also establish whether these will be inflated or deflated. Contrary to the conditional error parameters, which simply cause low-level noise that reshuffles data among the different patterns of responses, difficulties have more leverage in shaping the values of the state probabilities and, therefore, of the final probability of a response pattern. This reflects variations in the range of the difficulty parameters when the data-generating structure shows discrepancies from the data-fitting one. Specifically, data generated under a graded chain would tend to have higher response probabilities for the Guttman patterns (as they come from higher values of the state probabilities of these patterns) than for the non-Guttman ones (as they only come from the noise generated by the conditional error parameters). As soon as such data is fitted by a power set, which instead expects to find higher values of the response probabilities for the non-Guttman patterns, the range of the difficulties is inflated in order to make the easiest items even easier, and the most difficult items even more difficult. This amounts to reducing the state response probabilities of the non-Guttman patterns, which the power set believes exist but are only very unlikely, in favor of the Guttman ones.

On the converse, data generated under a power set would tend to have more comparable values of the response probabilities for both Guttman and non-Guttman patterns. As soon as such data is fitted by a graded chain, which instead expects to find lower values of response probabilities of the non-Guttman patterns that are only due to noise generated by the conditional error parameters, the range of the difficulties is deflated in order to make the easiest items more difficult and the most difficult ones easier. This amounts to deflating the state response probabilities of the Guttman patterns, which according to the graded chains, should be the only existing ones, in favor of the non-Guttman ones so that the latter can have comparable response probabilities when data is reshuffled by the error parameters.

4.2. Simulation Study 2

The objective of this study is to compare the fit of the model between the RM and the KST-IRT models under misspecifications. The data-generating process and the fitted models are interchanged between RM and the KST-IRT models with two knowledge structures, $\mathcal{C}$ and $2^Q$, and two state response functions, SLM and LKS. As previously mentioned, however, the case $2^Q+LKS$ has been left out since it is formally equivalent to the case $2^Q+SLM$.

Similarly to the previous simulation study, for N = 10,000, response patterns were generated with the true parameter values from the previous simulation study. For the difficulty parameters, it was set to a vector of increasing difficulty parameters, $b=(-2,-1,0,1,2)$, while the guessing and slipping parameters were set to the values $\eta =(0.129,0.100,0.074,0.115,0.096)$ and $\beta =(0.158,0.112,0.063,0.148,0.147)$. It should be noted that the KST-IRT models use all three parameters while the RM uses only the difficulty parameter. As highlighted in Section 2, this stresses that the two families of models assume very different data-generating processes.

For the computational method, though EM-with-Gibbs sampling outperforms MML-EM, the MML-EM is chosen as a method of parameter estimation as there are fewer assumptions involved. We use the MML-EM to estimate the parameters of the KST-IRT model and rasch function from ltm (version 1.2-0) package in R to estimate the RM difficulty parameters. Each MML-EM consumes, on average, 85.3 s to converge.

In

Table 4. Difficulty parameter estimation of Rasch and KST-IRT models under misspecified data-generating (DG) model and fitted model (FM).

		${\mathit{b}}_{\mathit{j}}$
		$\mathit{j}$
DG	FM	1	2	3	4	5
$RM$	$\mathcal{C}+SLM$	−1.86 (0.38)	−0.92 (0.30)	−0.02 (0.23)	0.93 (0.29)	1.80 (0.46)
$RM$	$\mathcal{C}+LKS$	−1.89 (0.28)	−0.95 (0.36)	−0.00 (0.30)	0.95 (0.41)	1.87 (0.50)
$RM$	$2^Q+SLM$	−1.79 (0.41)	−1.07 (0.37)	−0.04 (0.31)	0.90 (0.34)	1.87 (0.36)
$\mathcal{C}+SLM$	$RM$	−1.62 (0.38)	−0.52 (0.48)	0.86 (0.86)	2.04 (1.04)	3.10 (1.11)
$\mathcal{C}+LKS$	$RM$	−1.90 (0.11)	−1.22 (0.22)	−0.00 (0.02)	1.31 (0.31)	2.45 (0.48)
$2^Q+SLM$	$RM$	−1.60 (0.45)	−1.05 (0.14)	−0.05 (0.07)	1.13 (0.20)	2.05 (0.30)

Note. Each cell contains the average of the simulated estimates and the RMSE of the estimates in the parenthesis. $b_j$, ${\eta }_{q_j}$, and ${\beta }_{q_j}$ represent the difficulty, lucky guess, and careless error parameters for item $q_j$.

, the first two columns indicate the data-generating (DG) model and fitted model (FM), respectively. As expected, the increasing trend of the difficulty parameters is captured in all six comparisons. When the KST-IRT models are fitted to RM-generated data, the estimates are closer to the true value than in the converse case. The fit of KST-IRT models onto the RM-generated data provided estimates that display consistent trends with only small differences. The highest absolute difference between the mean estimates of the KST-IRT model fit and the true value is 0.21, whereas that of the RM is as high as 1.10 at $j=5$ for fitting RM to $\mathcal{C}+SLM$. However, the stability of the bias varies significantly when RM is used to fit the responses generated from KST-IRT models. As to the magnitudes of RMSEs, these are quite consistent across the model fit of the KST-IRT models, whereas fitting the RM to the KST-IRT responses is inconsistent across the items. For instance, the RMSE of fitting RM increases gradually to 1.11 when RM is fitted with $\mathcal{C}+SLM$. As a consequence, there appears to be two major concerns with the estimates of fitting RM to the assumed structure of the graded chain. Firstly, most of the estimates are overestimated compared to the true values. Secondly, the RMSE quickly rises as j increases. It does appear then that KST-IRT models, by virtue of their higher number of parameters, are more robust to violations of the data-generating process, while the traditional IRT approach, which assumes RM plus LI is more susceptible to violations of the LI assumption as it is implied by a data-generating process of the KST-IRT type in which a graded chain is considered. Notice that, instead, when the data is generated using a power set, the RM is less biased in its estimates since the combination of SLM and power set is equivalent to a 4PL model of IRT (with the discrimination parameter set to 1). Hence, the data generated from $2^Q+SLM$ can be considered as generated by a Rasch model with left-side added guessing and ceiling parameter, and where local independence is assumed.

5. Discussion

The present contribution had a two-fold aim. On the one hand, to test and compare estimation methods for KST-IRT models. On the other hand, to compare the performance of these models to the more traditional IRT approach of using the Rasch model in conjunction with the assumption of LI to model a probabilistic version of a Guttman scale. This second question can be considered part of a larger investigation on the possible benefits of using a KST-IRT approach to model LD as suggested by Noventa et al. [18].

As to the first aim, two computational methods, MML-EM and EM-with-Gibbs, were presented. The steps to derive the marginalized likelihood of how closely the KST-IRT models follow the guidelines given by Bock and Lieberman [25] and Bock and Aitkin [26]. A noteworthy difference in using the KST-IRT models for the MML approach is that the joint likelihood of the response pattern does not assume local independence among items, so the joint likelihood does not take the form of a product of factorized item response functions. This difference, however, does not hinder the use of the MML-EM. The artificial measurements such as ${\overline{n}}_k$ and ${\overline{r}}_{jk}$ depend on the observed responses, not necessarily the likelihood built under LI. For the Gibbs sampling approach, selecting the proper prior distribution for the KST-IRT item parameters showed that the response data can update the posterior distributions, yielding comparable estimates to those of MML-EM.

The parameter estimates were consistent with the correct model specification, and the estimates deviated while capturing the trend when misspecified models were fit. EM-with-Gibbs sampling outperforms MML-EM overall by small margins. However, it must be noted that implementing Gibbs sampling involves more assumptions on the item parameters. The MML-EM also is not completely frequentist in that matter, and the estimates can suffer deviating estimates when the prior distribution of the ability parameter is misspecified. However, with the Gibbs sampling, as the posterior distributions are only updated accordingly to the predetermined distributions, misspecification can lead to undesirable results entirely.

The computational burden for the presented method increases with the incremental number of items, mainly due to the growing complexity of knowledge structures and the increasing number of item parameters associated with each knowledge state. Efficiently implementing the MML-EM with larger assessments necessitates obtaining the marginalized likelihood and its essential components. For instance, obtaining derivatives of the item response functions with respect to each item parameter can bear a more computational burden, particularly with arbitrary knowledge structures beyond the graded chain and the power set. There is a need to generalize the parameter estimation method to handle larger assessments and accommodate arbitrary knowledge structures. This expansion, in the future, will offer a broader range of options for modeling responses, especially when the knowledge structure is capable of specifying the dependence structure.

Additionally, for further practical use, it is crucial to have model diagnostics to evaluate the model fit between the variations of the KST-IRT models. Beyond achieving reliable estimates with or without misspecification, model diagnostics to identify the better fit of the knowledge structure or the state response probability. Investigating model diagnostics will be essential in ensuring the appropriateness of the chosen model and enhancing the overall quality of the parameter process.

As to the second aim, it appears that KST-IRT models perform well in modeling a probabilistic version of a Guttman scale by capturing the essence of the scalogram by means of a graded chain, and by implementing additional noise in the form of conditional error parameters, to generate a non-Guttman pattern of responses. As such, they appear to provide an alternative to the more traditional IRT modeling of a probabilistic Guttman scale achieved by means of Rasch models under the assumption of local independence. As the two approaches are radically different in their fundamental assumptions, they were tested in terms of bias and robustness under the interchange of their respective generating processes. Generally, KST-IRT models appear to be less biased and more robust to violations of the data-generating process itself than the IRT traditional counterpart. Nonetheless, without having the true form of the Guttman scale or clear guidance on modeling the local dependence with the Rasch model, the direct comparison with the KST-IRT models that can inherently incorporate the local dependence and also allow for a larger number of parameters may not be fair. Nonetheless, for future consideration of more sophisticated IRT models and further acknowledging their prevalence in practice, a better understanding of the role of the noise parameters such as guessing and careless error probabilities, which in IRT applications are often debated or even neglected and yet in a KST-IRT approach are instead needed to recover those pattern of responses that do not conform with the knowledge structure, appears to be vital.