Dynamic Assessment to Assess Mathematical Problem Solving of Students with Disabilities

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

The article is well structured, interesting and clear. However, I would point out some minor points: the use of terms (problem solving, word problems, realistic situations) needs to be uniform. We distinguish between problem solving and word problems. With regard to the KK and GP problems presented, I would say that you are dealing with contextual problems that are related to physics. 
I would suggest you avoid naming the tablet and otherwise the iPad to avoid commercials.
I would also suggest avoiding the term traditional because it is dichotomous (negative connotation - I think term traditional is irrelevant), Similarly, I would like you to consider the notation in lines 595-596. I also miss the discussion of the gap (in the theoretical part you wrote about the gap: what the learner does with and without assistance).
The article would certainly have been enriched by some concrete dialogue with the learner in a concrete example of the task. In defining the contexts discussed, mathematical objectives could also have been mentioned, which would certainly have made the understanding of mathematics in the paper more meaningful.

 

Author Response

Comments 1: The article is well structured, interesting and clear. However, I would point out some minor points: the use of terms (problem solving, word problems, realistic situations) needs to be uniform. We distinguish between problem solving and word problems. 

Response 1: Thank you for pointing out these points. We have revised the terms to be consistent throughout the manuscript (as highlighted in yellow). In most cases, we used “mathematical problem solving” or “MPS”. In some cases, we described mathematical “word” problem to provide specific context. We revised realistic situation to problems in “real-world situations” as well. While we made revisions to ensure the consist use of terms throughout the manuscript, also found that we used examiner and evaluator interchangeably, so we revised this term as well (consistent use of examiner) as highlighted.

Comments 2: With regard to the KK and GP problems presented, I would say that you are dealing with contextual problems that are related to physics. 

Response 2: As briefly mentioned, all units are closely aligned with the CCSS-M that are used/adapted by 41 states in the United States. The math standards of the contextual problems of the KK and GP problems (e.g., calculating speed, draw and interpret an line of best fit) are 8.EE.B.5-6, which are 8^th grade math standards in the United States.

Comments 3: I would suggest you avoid naming the tablet and otherwise the iPad to avoid commercials.

Response 3: We have removed the term iPad as suggested and replaced by “tablet computers”.

Comments 4: I would also suggest avoiding the term traditional because it is dichotomous (negative connotation - I think term traditional is irrelevant), Similarly, I would like you to consider the notation in lines 595-596.

Response 4: We have removed the term “traditional test” and replaced by “written test” throughout the manuscript. As well, we revised the notation in lines 595-596 by using “perseverance” instead of “positive learning attitude”.

Comments 5: I also miss the discussion of the gap (in the theoretical part you wrote about the gap: what the learner does with and without assistance).

Response 5: We really appreciate your detailed review. This is a great point and something we have missed. We clarified that DA is assisted testing in discussion (lines 617-618) and added (and clarified) that WT is unassisted test and DA is assisted test (lines 613-615). We also added that the combined testing (both assisted DA and unassisted WT) would help teachers identify the gap and ensure the validity of testing (lines 703-705).

Comments 6: The article would certainly have been enriched by some concrete dialogue with the learner in a concrete example of the task. In defining the contexts discussed, mathematical objectives could also have been mentioned, which would certainly have made the understanding of mathematics in the paper more meaningful.

Response 6: Thank you for the suggestion, and we definitely agree that having more dialogue and concrete examples of the task would provide richer information. Due to the page/word limit, we had to be selective with what to include and not to. The current examples already included in the manuscript would provide some context of the nature of DA tasks. However, if you suggest any specific area, we would be certainly happy to add a couple of more dialogues/examples. To clarify the objectives of EAI units, we re-worded lines 244-245 to provide concrete example of the EAI unit objectives.

 

Reviewer 2 Report

Comments and Suggestions for Authors

The study includes a mixed-method design. However, the quantitative component was not described in detail. Did you use Hierarchical Liner Modeling? The study does not explain how the authors determine the sample for hierarchical linear modeling, teachers in schools or students in classrooms. 

How does quantitative part relate to qualitative part? What is the methodology followed through qualitative design? Which data collection methods are collected through quantitative and qualitative methods? Without clear explanation about the research design and methods, the results cannot be manageable. 

Additionally, in the literature review part, the authors should include whether there are similar studies in this context or methodology. The previous findings should be well represented to address the gaps in the literature. These gaps should be related to the discussion part to answer the research questions. 

Author Response

Comments 1: [The study includes a mixed-method design. However, the quantitative component was not described in detail. Did you use Hierarchical Liner Modeling? The study does not explain how the authors determine the sample for hierarchical linear modeling, teachers in schools or students in classrooms.]

Response 1: [Yes, we used hierarchical Linear Modeling (HLM), also called multilevel models (MLMs). We used the two terminologies interchangeably as researchers from different fields might be familiar with only one of them. We added more details about why we used a two-level MLM with outcome measurements nested within students. Given that a very small number of classrooms (i.e., three classrooms in total) was included in our study, we did not treat the classroom as a separate level. Instead, we dummy-coded Class as student-level predictors. We also understood that we did not provide justification of sample size for HLM because the current study was originally designed and intended for smaller group instruction int the classroom and by nature our study is an exploratory investigation. We acknowledged this limitation for using HLM in our discussion part as we could have low power to detect significant effects and lack of generalizability of our findings to a broader population of students and educators. Hence, we proposed to conduct future studies using large and diverse samples to address these limitations.]

Comments 2: [How does quantitative part relate to qualitative part?]

Response 2: [Qualitative parts (RQs 2 and 3) provide conceptual directions of DA in the MPS domain as explained in lines 167-187. To answer the qualitative questions, we believe the current method is most relevant.]

Comments 3: [What is the methodology followed through qualitative design?]

Response 3: [Please see the section 2.7.2. Qualitative Analysis (lines 448-478) for the methodology followed through qualitative design.]

Comments 4: [Which data collection methods are collected through quantitative and qualitative methods? Without clear explanation about the research design and methods, the results cannot be manageable.]

Response 4: [We have described data collection (administrating and scoring written test and DA) in lines 300-425. Research design and methods are in section 2.1 (lines 189-200). If there’s any specific areas, we will be happy to clarify and provide more information.]

Comments 5: [Additionally, in the literature review part, the authors should include whether there are similar studies in this context or methodology. The previous findings should be well represented to address the gaps in the literature. These gaps should be related to the discussion part to answer the research questions.]

Response 5: [As indicated in lines 167-169, the contextual gap we identified was the paucity of DA to assess MPS for students with disabilities. We have explained that our finding provides the empirical evidence of using the assessment approach to assess MPS skills of SWD (lines 620-621). As well, we included the literature review in lines 116-169 with previous findings of DA in the field.]

Reviewer 3 Report

Comments and Suggestions for Authors

Well done. It was easy to read and understand. I wonder if the DA would also support learning of students who struggle with mathematics, but not necessarily have a disability. 

The only comment I have concerns the part where you talk about the DA administration (lines 321-356). I maybe read this incorrectly, but in line 347 you state: "Students took the DA posttests followed by the WT posttest via Zoom". Then in lines 354-356 you state: "There were six DA posttest sessions over the course of three days immediately following the WT posttest after the last day of the GP uni". Which are contradicting in my opinion, the first states that the student first completed DA then WT, and the second states that the students first completed WT then DA.

Author Response

Comments 1: [Well done. It was easy to read and understand. I wonder if the DA would also support learning of students who struggle with mathematics, but not necessarily have a disability.]

Response 1: [Thank you. I agree that the DA would certainly support learning of struggling students without identified disability. In the current study, all subjects had disabilities, but we look forward to future opportunities to work with students who are “at risk” for learning disabilities in math.]

Comments 2: [The only comment I have concerns the part where you talk about the DA administration (lines 321-356). I maybe read this incorrectly, but in line 347 you state: "Students took the DA posttests followed by the WT posttest via Zoom". Then in lines 354-356 you state: "There were six DA posttest sessions over the course of three days immediately following the WT posttest after the last day of the GP uni". Which are contradicting in my opinion, the first states that the student first completed DA then WT, and the second states that the students first completed WT then DA.]

Response 2: [Thank you so much for catching the error, and our sincere apology that it was mis-written. We revised the sentence (line 347) to clarify that “the DA posttests after the WT posttests”.]

Round 2

Reviewer 2 Report

Comments and Suggestions for Authors

Thank you very much for revisions. I have a few points to ask:

1) How do you describe a mixed-method study? In your study, do qualitative and quantitative parts have equal weight, or is it qualitative dominant or quantitative dominant? 

2) Did you have quantitative data first and then collect qualitative data? What is the sequence of the data collection?

3) Could you please explain or address the implications of this research for curriculum development?

Author Response

Comments 1: [1) How do you describe a mixed-method study? In your study, do qualitative and quantitative parts have equal weight, or is it qualitative dominant or quantitative dominant?]

Response 1: [Thank you for the question! Please see the section 2.1 Research design (especially lines 208-218) for our response. We added an additional description of our mixed-method with the approach of “explanatory sequential design” (Cresswell et al., 2003; Ivankova et al., 2006). We also specified that the quantitative and qualitative data carried the equal weight but each had different focus.]

Comments 2: [2) Did you have quantitative data first and then collect qualitative data? What is the sequence of the data collection?]

Response 2: [In the same section of 2.1 Research Design, we explained our sequential approach: quantitative first and qualitative second (please see lines 208-210).]

Comments 3: [3) Could you please explain or address the implications of this research for curriculum development?]

Response 3: [Because the current study focused on “assessment”, we are afraid of adding “curriculum development” perspective. However, we addressed that further development of detailed guides and resources to implement the DA protocol used in the study as a suggestion for future research (please see lines 738-741).]