cdcatR: An R Package for Cognitive Diagnostic Computerized Adaptive Testing
Round 1
Reviewer 1 Report
Cognitive diagnosis models are very important psychometric tools in education, psychology and many other social sciences. This article introduces the first R package that combines CDM with adaptive testing and thus has important practical implications. I think the paper is well written, and I only have a few minor suggestions:
it would be great if the authors could present some general background information on diagnostic testing before the first section.
It would be better if the authors could add a few sentences to discuss why CAT is important.
Page 3 line 95: Where -> where
Page 13 Figure 7: while this graph is elegant, I'm not clear about the meaning of the big red dot.
Page 15 line 560: I'm not sure why maxitr is set to be 0 here.
Author Response
Please see the attachment.
Author Response File: 
Author Response.docx
Reviewer 2 Report
Summary of the Manuscript
The focus of this manuscript was to illustrate the various functions in a newly developed packed called cdcatR. Overall, this manuscript’s writing and readability were good; however, the descriptions of the methods and the results still have room for improvement. In the following section of this review, I will offer suggestions that may potentially improve the merit of this manuscript.
Manuscript Comments:
1. Please adding two paragraphs at the begging of the article. One paragraph gives a general introduction to the CDM and CAT. Another paragraph to list the reasons why we need to combine the CDM and CAT as one piece.
2. Please send the R package file to run on my local computer to test the performance of the package. For other journals regarding reviewing software, it is necessary to test the codes by running on the reviewers’ laptops. Without doing so, it is hard to comment on the validity of the package.
Author Response
Please see the attachment.
Author Response File: Author Response.docx
Reviewer 3 Report
This paper describes the usage of the cdcatR package containing tools to conduct analyses for cognitive diagnostic computer adaptive testing. The paper is dense with information, terminology, and code, but I think this is unavoidable, given the aims of the paper. However, I think the paper can be made more readable, especially with some edits to the introduction. I have included my specific comments below:
*Page 2, lines 66-67: The authors state they developed the package to facilitate research and empirical applications in CDMs, but it would be helpful in the introduction to give a summary of the types of analyses that can be done with the package. I understand that this information is given in subsequent sections, but it would be helpful to have more of an overview at the outset.
*Page 3, line 89: Why was this set of models chosen over other available CDMs?
*Page 14, line 502: “ For this reason, the cdcatR package was developed in a manner analogous…”—I think this sentence should be moved to the introduction. Also, please elaborate upon how the cdcatR package is analogous to other IRT packages.
Author Response
Please see the attachment.
Author Response File: Author Response.docx
Round 2
Reviewer 2 Report
The author addressed my previous comments.
A minor comment:
Please adding one potential extension of current package by mentioning the newly developed methods:
1) Zhan P, Jiao H, Man K, Wang L. Using JAGS for Bayesian Cognitive Diagnosis Modeling: A Tutorial. Journal of Educational and Behavioral Statistics. 2019;44(4):473-503. doi:10.3102/1076998619826040
Thanks
Author Response
Following the reviewer’s suggestion, we have included a reference to Zhang et al. (2019) on the use of Bayesian methods in CDM (lines 626-631):
“Finally, in this paper, we have adopted as a starting point a maximum likelihood estimation of item parameters. Recently, efforts have been made to make the estimation with the Bayesian Markov chain Monte Carlo algorithm more accessible [56]. This approach can be particularly useful in the case of complex models. Work will be done to allow the adoption of the results of a calibration obtained in this way as input in the cdcat function.”
