A Deterministic Learning Algorithm Estimating the Q-Matrix for Cognitive Diagnosis Models

Abstract

The goal of an exam in cognitive diagnostic assessment is to uncover whether an examinee has mastered certain attributes. Different cognitive diagnosis models (CDMs) have been developed for this purpose. The core of these CDMs is the Q-matrix, which is an item-to-attribute mapping, traditionally designed by domain experts. An expert designed Q-matrix is not without issues. For example, domain experts might neglect some attributes or have different opinions about the inclusion of some entries in the Q-matrix. It is therefore of practical importance to develop an automated method to estimate the Q-matrix. This research proposes a deterministic learning algorithm for estimating the Q-matrix. To obtain a sensible binary Q-matrix, a dichotomizing method is also devised. Results from the simulation study shows that the proposed method for estimating the Q-matrix is useful. The empirical study analyzes the ECPE data. The estimated Q-matrix is compared with the expert-designed one. All analyses in this research are carried out in R.

1. Introduction

1.1. Cognitive Diagnosis Models

Cognitive diagnostic assessment (CDA) is a framework that intends to evaluate an examinee’s mastery of a specific cognitive skill called an attribute [1]. A few cognitive diagnosis models (CDMs) have been developed, such as the deterministic input, noisy “and” gate (DINA) model [2], and the reduced reparameterized unified model (RRUM) [3,4].

The core of different CDMs is the Q-matrix [5].

Table 1. A Q-matrix example.

Item	Attribute
	Addition	Subtraction
(1). $4+5$	1	0
(2). $7-2$	0	1
(3). $6+3-1$	1	1

shows a simple example of the Q-matrix, which assesses whether an examinee has acquired addition and subtraction attributes. An examinee must possess an addition attribute in order to correctly answer the first item. Suppose that the attribute status of the examinee is $(1,0)$. Without possessing the subtraction attribute, the examinee is expected to only answer item 1 correctly since items 2 and 3 require a subtraction attribute.

The DINA model and RRUM are two popular models based on the Q-matrix. The DINA model and RRUM are non-compensatory, assuming that examinees must have mastered all the required attributes to correctly answer an item.

Suppose the data is comprised of item responses from I examinees to J items that measure K attributes. In the DINA model, ${\eta }_{ij}$ indicates whether examinee i ($i=1,\cdots ,I$) can correctly answer item j ($j=1,\cdots ,J)$,

$${\eta }_{ij}=\prod _{k=1}^K{\alpha }_{ik}^{q_{jk}},$$

where ${\alpha }_{ik}$ is the mastery of attribute k ($k=1,\cdots ,K)$ for examinee i and $q_{jk}$ is the state of the jth item and kth attribute in the Q-matrix. The item response function (IRF) for the DINA model is:

$$P(X_{ij}=1|{\alpha }_i)={(1-s_j)}^{{\eta }_{ij}}{g}_j^{1-{\eta }_{ij}},$$

where $g_j$ and $s_j$ are guess and slip parameters for item j, ${\alpha }_i$ is the attribute status of examinee i. The guess parameter $g_j$ represents the probability of $X_{ij}=1$ when at least one required attribute is lacking, and the slip parameter $s_j$ denotes the probability of $X_{ij}=0$ when all required attributes are present.

Suppose the guess and slip parameters for item 1 in Table 1 are $g_1=s_1=0.2$. If the attribute status of an examinee i is ${\alpha }_i=\left(1,0\right)$, then ${\eta }_{i1}=1$. Therefore, the examinee’s probability of correctly answering item 1 is $P\left(X_{ij}=1|{\alpha }_i=\left(1,0\right)\right)={\left(1-0.2\right)}^{1}0.2^0=0.8$.

The RRUM is another popular CDM, especially in language assessment. The IRF of the RRUM is:

$$P(X_{ij}=1|{\alpha }_i)={\pi }_j^{*}\prod _{k=1}^K{\left(r_{jk}^{{*}^{(1-{\alpha }_{ik})}}\right)}^{q_{jk}},$$

(1)

where ${\pi }_j^{*}={\displaystyle \prod _{k=1}^K}{(1-s_{jk})}^{q_{jk}}$ and $r_{jk}^{*}=g_{jk}/1-s_{jk}.$ Note that $g_{jk}$ and $s_{jk}$ are guess and slip parameters of item j for attribute k. As in the DINA model, ${\alpha }_{ik}$ is the mastery of attribute k for examinee i and $q_{jk}$ is the state of the jth item and kth attribute in the Q-matrix.

1.2. Model Based Estimation of the Q-Matrix

Most of the Q-matrices are not specified during exam development and are conventionally designed by domain experts after the fact. A more objective means to assign the Q-matrix is needed because domain experts might neglect some attributes or have different opinions. For instance, different modifications of the Q-matrix for the fraction subtraction data [6] have been suggested (e.g., Tatsuoka [6]; de la Torre [7]; de la Torre and Douglas [8]; de la Torre and Douglas [9]; Henson et al. [10]; Tatsuoka [11]) over the past three decades.

It is therefore of practical importance to develop an automated method for searching the Q-matrix. The purpose of this research is to develop a deterministic learning algorithm that estimates the Q-matrix for CDMs. It should be noted that this research attempts to develop a method to estimate the whole Q-matrix rather than to validate an existing one.

Estimating the whole Q-matrix is NP-hard [12]. The sample space of this discrete optimization can be huge. For example, simulation studies of this research try to recover a 15 by 4 Q-matrix (see

Table 2. Q-matrices for simulations.

Item	Complete Q-Matrix				Incomplete Q-Matrix
	Attributes				Attributes
	1	2	3	4	1	2	3	4
1	1	0	0	0	1	0	0	1
2	0	1	0	0	0	1	1	0
3	0	0	1	0	0	1	1	0
4	0	0	0	1	1	0	0	1
5	1	1	0	0	1	1	0	0
6	1	0	1	0	1	0	1	0
7	1	0	0	1	1	0	0	1
8	0	1	1	0	0	1	1	0
9	0	1	0	1	0	1	0	1
10	0	0	1	1	0	0	1	1
11	1	1	1	0	1	1	1	0
12	1	1	0	1	1	1	0	1
13	1	0	1	1	1	0	1	1
14	0	1	1	1	0	1	1	1
15	1	1	1	1	1	1	1	1

), which gives $2^{60}/4!$ possible Q-matrices when column permutation is considered. Since the optimization problem can be simplified using model-based methods, most of the studies attempting to estimate the Q-matrix are CDM based.

For example, Chen et al. [13], Liu et al. [14], Liu et al. [15], and Xu and Shang [16] use the DINA model to estimate the Q-matrix. Chung [17] and Chung [18] show that both the DINA model and RRUM are both feasible models. Treating the Q-matrix as a parameter to estimate these studies assume that the true model is the DINA model or the RRUM. Although simulations in these studies suggest promising Q-matrix recovery rates, it is desirable to find non-model-based methods.

1.3. NMF Estimation of the Q-Matrix

Winters [19] was the first to apply non-negative matrix factorization (NMF) to explore the structure of the Q-matrix. Desmarais and his colleagues have also published a few studies (e.g., Desmarais [20]; Desmarais [21]; Desmarais and Naceur [22]; Desmarais et al. [23]) that apply NMF to estimating the Q-matrix. These studies applying NMF are inspiring, showing that Q-matrix estimation is in essence a matrix factorization.

While their findings are promising, some issues are worth considering. The primary concern is that the inclusion of an entry (1 or 0) is counted solely on visual inspection using heatmaps. While using visual inspection is desirable in some conditions, we would expect a more decisive means to avoid ambiguity. Another issue is that the way the initial values were retrieved is unclear, especially considering that NMF is very sensitive to initial values (Cichocki et al. [24]; Gond and Nadi [25]; Zheng et al. [26]). A more recent work by Casalino et al. [12] has shown that using the constraint alternating least square is a practical solution to a stable estimation.

Other issues are related to their simulation studies. No information is available for the different sample sizes and the correlations between attributes. In addition, each item measures at most two attributes in their Q-matrix for simulations when in reality it is not uncommon to have an items testing more than two attributes. We are also interested in whether NMF can recognize items measuring all attributes.

Inspired by Desmarais and his colleagues’ research, this research offers a factorization algorithm for the Q-matrix, as well as refining simulation designs. Specifically, we use a maximum likelihood estimation and enforce a dichotomizing scheme in the estimation. We define a recovery rate to investigate the effect of the method on different sample sizes and correlations under complete and incomplete Q-matrix designs. In addition to the DINA model, the RRUM is also adopted in the simulation study.

2. A Deterministic Learning Algorithm for the Q-Matrix

2.1. Maximum Likelihood Estimation

Barnes et al. [27] and Desmarais [20] give the following equation that infers the item response ($\mathit{X}$) as the product of the Q-matrix ($\mathit{Q}$) and the attribute mastery matrix ($\mathit{\alpha }$):

$$\mathit{X}=\mathit{Q}{\mathit{\alpha }}^T,$$

(2)

where $\mathit{X}$, $\mathit{Q}$, and $\mathit{\alpha }$ are binary with each element either 0 or 1. Estimating the Q-matrix turns into a factorization problem, which can be simplified if we temporarily treat $\mathit{Q}$ as continuous.

This research proposes to freely derive $\mathit{Q}$ by maximum likelihood estimation and then dichotomize the estimate to produce a binary Q-matrix. The advanced procedure is explained as the following. Suppose that a corresponding continuous version of $\mathit{Q}$ is $\mathit{R}$. The problem becomes to find a valid factorization of $\mathit{X}$ that could provide an estimate of $\mathit{R}$:

$$\mathit{X}=\mathit{R}{\mathit{\alpha }}^T.$$

(3)

If the transformation in (3) is linear and nonsingular, then ${\mathit{\alpha }}^T={\mathit{R}}^{-1}\mathit{X}$. Using a Jacobian transformation, we can derive the pdf of $\mathit{X}$:

$$p_X\left(\mathit{X}\right)=\frac{1}{\left|\det \mathit{R}\right|}{p}_{\alpha }\left({\mathit{\alpha }}^T\right).$$

(4)

Let $\mathit{M}={\mathit{R}}^{-1}$, and let $p_i$ denote the pdf of ${\mathit{\alpha }}_i$. The pdf of $\mathit{X}$ in (4) becomes:

$$p_X\left(\mathit{X}\right)=\left|\det \mathit{M}\right|p_{\alpha }\left({\mathit{\alpha }}^T\right)=\left|\det \mathit{M}\right|\prod _i{p}_i\left({\mathit{\alpha }}_i^T\right).$$

Suppose the Q-matrix measures K attributes. That is, $\mathit{M}={\left(m_1,m_2,\cdots ,m_K\right)}^T$, and therefore:

$$p_X\left(\mathit{X}\right)=\left|\det \mathit{M}\right|\prod _{i=1}^K{p}_i\left(m_i^T\mathit{X}\right).$$

(5)

Suppose that we have N observations, denoted ${\mathit{X}}_1,{\mathit{X}}_2,\cdots {\mathit{X}}_N$. From the pdf shown in (5), the likelihood $L\left(\mathit{M}\right)$ is therefore:

$$L\left(\mathit{M}\right)=\prod _{n=1}^N\prod _{i=1}^K{p}_i\left(m_i^T{\mathit{X}}_n\right)\left|\det \mathit{M}\right|,$$

and therefore the log-likelihood is:

$$logL\left(\mathit{M}\right)=\sum _{n=1}^N\sum _{i=1}^{K}log{p}_i\left(m_i^T{\mathit{X}}_n\right)+Nlog\left|\det \mathit{M}\right|.$$

(6)

Dividing the log-likelihood of (6) by N yields:

$$\frac{1}{N}logL\left(\mathit{M}\right)=E\left(\sum _{i=1}^{K}log{p}_i\left(m_i^T\mathit{X}\right)+log\left|\det \mathit{M}\right|\right).$$

(7)

It should be noted that the expectation will eventually be replaced by sample averages.

To find the learning algorithm for $\mathit{M}$, we can maximize (7) by taking partial derivative with respect to $\mathit{M}$:

$$\frac{1}{N}\frac{\partial logL\left(\mathit{M}\right)}{\partial \mathit{M}}=\frac{\partial E\left({\sum }_{i=1}^{K}log{p}_i\left(m_i^T\mathit{X}\right)\right)}{\partial \mathit{M}}+\frac{\partial E\left(log\left|\det \mathit{M}\right|\right)}{\partial \mathit{M}}.$$

To simplify notation, we let $u_i=m_i^T\mathit{X}$ so that $\frac{\partial E\left({\sum }_{i=1}^{K}log{p}_i\left(m_i^T\mathit{X}\right)\right)}{\partial \mathit{M}}=\frac{\partial {\sum }_{i=1}^{K}log{p}_i\left(u_i\right)}{\partial \mathit{M}}$. If we let $p\left(\mathit{u}\right)={\displaystyle \underset{i=1}{\overset{n}{\Pi }}}{p}_i\left(u_i\right)$ and define $\phi \left(\mathit{u}\right)=-\frac{\frac{\partial p\left(\mathit{u}\right)}{\partial \mathit{u}}}{p\left(\mathit{u}\right)}$ where $\phi \left(·\right)$ is the negative score function, Lee et al. [28] shows that:

$$\frac{\partial {\sum }_{i=1}^{K}log{p}_i\left(u_i\right)}{\partial \mathit{M}}=-\phi \left(\mathit{u}\right){\mathit{X}}^T.$$

We now evaluate the other term, $\frac{\partial log\left|\det \mathit{M}\right|}{\partial \mathit{M}}$. From Grossman [29], we know that:

$$\frac{1}{\mathit{M}}=\frac{1}{\det \mathit{M}}\mathrm{adj}\left(\mathit{M}\right),$$

where $\mathrm{adj}\left(\mathit{M}\right)$ is the adjoint of $\mathit{M}$. We can express $\det \mathit{M}$ in terms of cofactors,

$$\det \mathit{M}=\sum _{k=1}^n{b}_{ik}{M}_{ik}.$$

(8)

Taking partial derivative of (8) with respect to $m_{ij}$ gives $\frac{\partial \det \mathit{M}}{\partial m_{ij}}=m_{ij},$ which suggests that:

$$\frac{\partial \det \mathit{M}}{\partial b_{ij}}=\mathrm{adj}{\left(\mathit{M}\right)}^T.$$

As $\mathrm{adj}{\left(\mathit{M}\right)}^T=\left(\det \mathit{M}\right){\left({\mathit{M}}^T\right)}^{-1}$, we get:

$$\frac{\partial \left|\det \mathit{M}\right|}{\partial \mathit{M}}=\frac{\det \mathit{M}}{{\mathit{M}}^T},$$

so that:

$$\frac{\partial log\left|\det \mathit{M}\right|}{\partial \mathit{M}}=\frac{1}{det\left|\mathit{M}\right|}\frac{\partial \left|\det \mathit{M}\right|}{\partial \mathit{M}}=\frac{1}{{\mathit{M}}^T}.$$

The gradient of the log-likelihood in (7) is therefore:

$$\begin{array}{cc}\hfill \frac{1}{N}\frac{\partial logL\left(\mathit{M}\right)}{\partial \mathit{M}}& =\frac{\partial E\left({\sum }_{i=1}^{n}log{p}_i\left(m_i^T\mathit{X}\right)\right)}{\partial \mathit{M}}+\frac{\partial E\left(log\left|\det \mathit{M}\right|\right)}{\partial \mathit{M}}.\hfill \\ \hfill & =-E\left(\phi \left(\mathit{M}\mathit{X}\right){\mathit{X}}^T\right)+\frac{1}{{\mathit{M}}^T}\hfill \end{array}$$

For only one data point, the expectation is omitted. The learning algorithm is given by:

$$\Delta \mathit{M}=-\phi \left(\mathit{M}\mathit{X}\right){\mathit{X}}^T+\frac{1}{{\mathit{M}}^T}.$$

2.2. Rotation and Dichotomization

Inverse $\mathit{M}$ in each iteration, and we can obtain $\mathit{R}$. To enhance the interpretability, $\mathit{R}$ entails an orthogonal rotation by varimax. The rotated $\mathit{R}$ is denoted as $\mathit{Q}$. As the Q-matrix is binary, a method to dichotomize $\mathit{Q}$ is imperative. We devise the following dichotomizing scheme for converting $\mathit{Q}$ to the binary Q-matrix.

Suppose the Q-matrix uses J items to measure K attributes. Each row in ${\mathit{Q}}_{J\times K}$ is an item. Item j can be presented as $q_j=(q_{j1},q_{j2},\cdots ,q_{jK})$. Let the entry with the highest value in item j be $q_{jmax}$, namely, $q_{jmax}=max(q_{j1},q_{j2},\cdots ,q_{jK})$. The value of each attribute in the item is then decided by the relative magnitude of its value to the highest one. That is, $q_j/q_{jmax}=(q_{j1}/q_{jmax},q_{j2}/q_{jmax},\cdots ,q_{jK}/q_{jmax})$. For entry $q_{jk}$ in item j, if $q_{jk}/q_{jmax}\ge a,$ set $q_{jk}$ to 1. If $q_{jk}/q_{jmax}<a,$ then set $q_{jk}$ to 0.

For example, suppose a Q-matrix measures 4 attributes, and values for certain item j is $q_j=(0.111,0.222,0.333,0.444)$. From $q_j$, we find $q_{jmax}=0.444$ and therefore $q_j/q_{jmax}=(0.111/0.444,0.222/0.444,0.333/0.444,0.444/0.444)=(0.25,0.5,0.75,1)$. If a is set to 0.6, then the derived Q-matrix state for item j is $(0,0,1,1)$.

Different values of a (i.e., $0.4$, $0.5$, $0.6$) are tested in this research. After this procedure is applied to every item, the whole Q-matrix is derived because ${\mathit{Q}}_{J\times K}$ is now binary.

3. Simulation Study

We examine whether our proposed algorithm can recover the Q-matrix from simulated data. Settings for the simulation are the same as those in Chung [18]. The following simulation study is carried out using customized R codes.

3.1. Q-Matrix for Simulation

This simulation study adopts the artificial Q-matrix (the complete Q-matrix in Table 2) acquired from Rupp and Templin [30]. Fifteen items that measure four attributes comprise the Q-matrix, which is constructed such that each attribute appears alone from items 1 to 4, in a pair from items 5 to 10, or in a triad from items 11 to 14. It should be noted that item 15 measures all 4 attributes.

The complete Q-matrix in Table 2 contains at least one item devoted solely to each attribute [31]. That is, each attribute in the complete Q matrix in Table 2 is individually measured by at least one item (e.g., attribute 1 is individually tested by item 1).

This research also examines the effectiveness of our proposed algorithm in recovering an incomplete Q-matrix (the incomplete Q-matrix in Table 2), in which an item measures more than one attribute. Each item in the incomplete Q-matrix in Table 2 tests at least two attributes (e.g., item 1 tests attributes 1 and 4).

3.2. Generating Correlated Attributes

Chung [18] suggests generating correlated attributes using a copula with the Choleski decomposition. Suppose the Q-matrix (Q) uses J items to measure K attributes for I examinees. That is, $Q_{J\times K}={\left(q_{jk}\right)}_{J\times K}$ and ${\mathit{\alpha }}_{I\times K}={\left({\alpha }_{ik}\right)}_{I\times K}$. Let $\mathit{\vartheta }$ be the I by K underlying probability matrix of $\mathit{\alpha }$, and let column k of $\mathit{\vartheta }$ be vector ${\mathit{\vartheta }}_k$, $k=1,\dots ,K$. That is, $\mathit{\vartheta }=(\mathit{\vartheta }{}_1,\dots ,{\mathit{\vartheta }}_K)$. The correlation coefficient for each pair of columns in $\mathit{\vartheta }$ takes the constant value $\rho$, and the correlation matrix is represented as $\mathit{\Sigma }$. Each entry in $\mathit{\Sigma }$ corresponds to the correlation coefficient between two columns in $\mathit{\vartheta }$. $\mathit{\Sigma }$ can be further decomposed as $\mathit{\Sigma }={\mathit{\nu }}^T\mathit{\nu }$ using the Choleski decomposition, where $\mathit{\nu }$ is an upper triangular matrix.

After $\mathit{\nu }$ is derived, create an $I\times K$ matrix $\mathit{\tau }$, in which each entry is generated from $\mathit{N}(0,1)$. $\mathit{\tau }$ is then transformed to $\gamma$ by using $\gamma =\mathit{\tau }\mathit{\nu }$, so that $\mathbf{fl}$ and $\mathit{\Sigma }$ will have the same correlation structure. Set $\Phi \left(\gamma \right)=\mathit{\vartheta }$, where $\Phi (·)$ is the CDF of the standard normal distribution. $\mathit{\alpha }$ is generated using inverse transform sampling. Create a matrix ${\Theta }_{I\times K}={\left({\theta }_{ik}\right)}_{I\times K}$, where each element is generated from $\mathit{Uniform}(0,1)$. If ${\vartheta }_{ik}\ge {\theta }_{ik}$, set ${\alpha }_{ik}$ to 1, and if ${\vartheta }_{ik}<{\theta }_{ik}$, set ${\alpha }_{ik}$ to 0.

3.3. Generating Data from the DINA Model and RRUM

With the Q-matrix and dependent attributes, data were simulated using the DINA model and the RRUM. Values for all guess and slip parameters are set to 0.2 for both the DINA model ($s_j=g_j=0.2$) and RRUM ($s_{jk}=g_{jk}=0.2$), and the data are then created using the inverse transform sampling from two points, in which the probability is obtained from the IRF of the DINA model and RRUM. Note that from Equation (1) for the RRUM, we simulate data using $s_{jk}$ and $g_{jk}$ instead of using reparameterized parameters ${\pi }_j^{*}$ and $r_{jk}^{*}$.

We follow the following settings that appear in Chung [18]. Examinees in groups of 500, 1000, and 2000 were simulated with the correlation between each pair of attributes set to 0.1, 0.3, and 0.5 for both the DINA model and RRUM. One hundred datasets were simulated for each combination of sample size and correlation.

3.4. Evaluation

Evaluating how well the proposed method recovers the true Q-matrix $q$, the recovery rate $\Delta$ suggested by [14] is defined as:

$$\Delta =\frac{1}{M}\sum _{m=1}^M\left(1-\frac{\left|{\widehat{\mathit{q}}}^{\left(m\right)}-\mathit{q}\right|}{JK}\right),$$

where $M=100$, $|·|$ takes the absolute value and ${\widehat{\mathit{q}}}^{\left(m\right)}$ stands for the estimated Q-matrix for mth dataset.

3.5. Results from Simulation

For both the DINA model and RRUM, the optimal cutoff value a is 0.5 (see

Table 3. Average $\Delta$ from different a.

DINA		RRUM
$\mathit{a}$	$\Delta$	$\mathit{a}$	$\Delta$
0.4	0.983	0.4	0.944
0.5	0.994	0.5	0.962
0.6	0.991	0.6	0.954

). It should be noted that 0.5 is not an arbitrary cutoff. The value of 0.5 here means a half of the highest value. The following has stated Section 2.2: If $q_{jk}/q_{jmax}\ge a,$ set $q_{jk}$ to 1. If $q_{jk}/q_{jmax}<a,$ then set $q_{jk}$ to 0. The value of an attribute has to be more than a half of the highest value to be regarded as 1 in the Q-matrix.

As $a=0.5$ is optimal,

Table 4. Recovery rates of complete Q-matrix when $a=0.5$.

N	DINA			RRUM
	$\mathbf{\rho }$			$\mathbf{\rho }$
	0.1	0.3	0.5	0.1	0.3	0.5
500	0.997	0.990	0.975	0.942	0.932	0.913
1000	0.999	0.998	0.992	0.985	0.974	0.954
2000	1.000	1.000	0.998	0.995	0.988	0.976

Note: N and ρ respectively stand for sample size and correlation coefficient.

shows the recovery rate for the complete Q-matrix when $a=0.5$. For the complete Q-matrix with $a=0.5$, average $\Delta$ from all combinations in the DINA model is 0.994, suggesting an adequate result when the Q-matrix is complete although average $\Delta$ drops to 0.962 for the more complicated RRUM.

In terms of different sample sizes and correlations, $\Delta$ rises along with the increase of the sample size and drops when the correlation between attributes rises. Note that the number of attributes is supposed to be known in advance.

For different combinations of sample size and correlation, $\Delta$ declines when the correlation between attributes goes up, while it increases along with the increase of the sample size. $\Delta$ is higher when the data are simulated from the RRUM.

When the Q-matrix is incomplete, $\Delta$ is unsatisfactory (see

Table 5. Recovery rates of incomplete Q-matrix when $a=0.5$.

N	DINA			RRUM
	$\mathbf{\rho }$			$\mathbf{\rho }$
	0.1	0.3	0.5	0.1	0.3	0.5
500	0.792	0.772	0.751	0.781	0.761	0.740
1000	0.838	0.813	0.772	0.825	0.804	0.762
2000	0.849	0.831	0.797	0.834	0.822	0.789

Note: N and ρ respectively stand for sample size and correlation coefficient.

), ranging from 0.754 to 0.772. Such low recovery rates from an incomplete Q-matrix are similar to the findings in Chung [18].

4. Empirical Study

Obtained from the CDM package in R [32], the Examination for the Certificate of Proficiency in English (ECPE) data that consists of 2922 examinees is analyzed using the proposed method described in Section 2. The ECPE is a test developed and scored by the English Language Institute of the University of Michigan [33].

The ECPE data have appeared in different studies, such as Buck and Tatsuoka [34], Henson and Templin [35], Templin and Hoffman [33], Feng, Habing, and Huebner [36], and Templin and Bradshaw [37].

Buck and Tatsuoka [34] suggest a Q-matrix consisting of 28 items that measure 3 attributes: Morphosyntactic rules, Cohesive rules, and Lexical rules (expert Q-matrix in Table 5). In addition, a parallel analysis was tentatively used to determine the number of attributes in the Q-matrix. The result also suggests 3 attributes.

We consequently decided to evaluate a 3-attribute Q-matrix solution. Exhibited in

Table 6. Q-matrix estimate from the empirical study.

Item	Expert Q-Matrix (${\mathit{Q}}_{\mathit{X}}$)			Estimated Q-Matrix (${\mathit{Q}}_{\mathit{E}}$)
	Attribute			Attribute
	1	2	3	1	2	3
1	1	1	0	1	1	1
2	0	1	0	0	1	1
3	1	0	1	1	0	0
4	0	0	1	0	0	1
5	0	0	1	0	1	1
6	0	0	1	0	0	1
7	1	0	1	1	0	1
8	0	1	0	1	1	1
9	0	0	1	0	0	1
10	1	0	0	1	0	0
11	1	0	1	1	0	1
12	1	0	1	0	0	1
13	1	0	0	0	0	1
14	1	0	0	0	0	1
15	0	0	1	0	1	1
16	1	0	1	0	0	1
17	0	1	1	0	0	1
18	0	0	1	0	1	1
19	0	0	1	0	0	1
20	1	0	1	1	0	1
21	1	0	1	0	1	1
22	0	0	1	0	0	1
23	0	1	0	0	1	1
24	0	1	0	0	1	0
25	1	0	0	1	0	0
26	0	0	1	0	1	1
27	1	0	0	1	0	1
28	0	0	1	0	0	1

, the expert-designed and estimated Q-matrices are denoted $Q_X$ and $Q_E$ respectively. An inspection of $Q_X$ and $Q_E$ shows that each of them is a complete Q-matrix, and no two columns are identical. Comparing $Q_X$ with $Q_E$, we find that 11 items are identical. Overall, 76.2% of the entries are identical. Like the expert designed Q-matrix, the estimated Q-matrix is complete. As for model fit, AICs for $Q_X$ and $Q_E$ are respectively 85,812.92 and 85,709.47, suggesting that the estimated Q-matrix ($Q_E$) better fits the data.

5. Discussion

The last 10 years have seen the development of a few CDM-based methods for extracting the Q-matrix, whereas non-CDM based approaches have been rarely seen. One of the non-CDM based methods makes use of NMF. Desmarais and his colleagues revealed that NMF is a useful method in deriving the Q-matrix. From the result of NMF, Q-matrix estimation is a matrix factorization. Inspired by NMF, this study demonstrates the practicality of our proposed deterministic learning algorithm in generating the Q-matrix.

As the Q-matrix is regarded as a factor model with binary factor loadings, a rotation using varimax after the estimation is necessary to aid interpretation. After a rotation, an objective way to dichotomize continuous estimates is needed to derive the final Q-matrix. Arbitrarily using a fixed value such as 0.5 might generate fallacious estimates. For example, if the factor loadings for an item j is (0.111, 0.222, 0.333, 0.444), then a cutoff 0.5 will result in (0, 0, 0, 0). In this research, we enforce a dichotomizing scheme using a proportion of the highest loading (a) on the estimates after they are rotated. Results from the simulation study suggests that setting a to 0.5 is optimal.

The simulation study demonstrates that the proposed deterministic learning algorithm is capable of extracting the Q-matrix from data, with the results showing that recovery rates from different conditions are all above 0.9. Both sample sizes and correlations between attributes are influential in the Q-matrix recovery. When the Q-matrix is incomplete, the results are not as good as the settings with a complete Q-matrix. The low recovery rate is due to the non-identifiability of model parameters under the incomplete Q-matrix [38].

One limitation in this research is that the number of attributes is assumed to be known in the estimation. Future research could use a parallel analysis to deal with the issue. Another limitation is that data were simulated only from the DINA model and RRUM. It is desirable to test the effectiveness of the proposed method on data generated from more general CDMs, such as the GDINA model. Further research using the proposed method is also recommended to examine the recovery rate from data simulated from non-compensatory models, such as the DINO model.

More extensive research is needed to compare the result from factor models with the result from CDM-based methods. A further comparison could investigate the effect of model misspecification on guess and slip parameters in the DINA model and RRUM.

In the empirical study, our Q-matrix estimate is very different from the expert designed Q-matrix, although 76.2% of the Q-matrix entries are identical. It is suggested that further research applies validation approach, setting those identical entries as fixed and estimating the rest of the entries.

In conclusion, this research demonstrated that the Q-matrix is intrinsically a factor model. Different dimensionality reduction techniques, such as principal component analysis and singular value decomposition, incorporated with the dichotomizing scheme advanced in this research should be able to derive the Q-matrix.