Review Reports - Perceptions about the Use of Educational Robotics in the Initial Training of Future Teachers: A Study on STEAM Sustainability among Female Teachers

Round 1

Reviewer 1 Report

The paper presents an interesting research question and results from an extensive survey. If some points are complemented and modified first, it could be a valuable publishable paper. Please enhance or work on the following aspects:

The figures do not provide any additional and relevant information to the paper. Please delete them.
Instead, the robotic kits used during the evaluated intervention are not described at all. The intervention, including the intended competences to be trained and the content they dealt with, should be described in detail in a separate section of the paper.
Results from the questionnaire are only shown in terms of average scores. Differences among the pre and post-answers are not very high. Thus, the standard deviation should be shown for each of the scores in order to get a clearer idea of the influence of the intervention. The standard deviation would give a much clearer picture of the differences achieved due to the course or to different personal attitudes of certain students.
A variance analysis should be performed to the pre- and post- results in order to check the statistical significance in the differences due to the intervention itself.

Comments for author File: Comments.pdf

Author Response

Dear Reviewer:

I have analyzed and reviewed the changes you have suggested to me. They have been very helpful in improving the manuscript.

For a better check, please allow me to copy and paste your improvement observations and my argumentation to each of them.

I hope and wish that with these changes you can verify that I have followed your instructions.

Sincerely.

1) Suggestion for improvement 1.

The figures do not provide any additional and relevant information to the paper. They should be deleted.

Answer 1 (in red in the word document).

The mentioned figures have been removed and new figures have been added that clarify the robotic kits used in the investigation. Lines 86, 96, 114 and 138.

The references in the text to these have also been removed figures as a suggestion of the second reviewer.

2) Suggestion for improvement 2.

Instead, the robotic kits used during the evaluated intervention are not described at all.

Answer 2 (in red in the word document).

Added improvement suggestions: description of robotic kits, from lines 82 to 45.

3) Suggestion for improvement 3.

The intervention, including the intended competences to be trained and the content they dealt with, should be described in detail in a separate section of the paper.

Answer 3 (in red in the word document).

Added suggestions for improvement related to the intervention with robotic kits, it has been included in a separate section, along with the description of robotic kits and what skills are developed with them, from lines 49 to 152.

4) Suggestion for improvement 4.

Results from the questionnaire are only shown in terms of average scores. Differences among the pre and post-answers are not very high. Thus, the standard deviation should be shown for each of the scores in order to get a clearer idea of the influence of the intervention.

Answer 4 (in red in the word document).

Added suggestions for improvement, which refer to the inclusion of the standard deviations obtained in each of the items, both in the pre-test and post-test scores. The average of the standard deviations obtained for all the items has also been calculated, from lines 234 to 353.

5) Suggestion for improvement 5.

A variance analysis should be performed to the pre- and post- results in order to check the statistical significance in the differences due to the intervention itself.

Answer 5 (in red in the word document).

Added suggestions for improvement, which refer to the inclusion of the variance obtained in each of the items, both in the pre-test and post-test scores, to verify the statistical significance of the results. The average of the variances obtained by all the items has also been calculated, from the lines 234 to 367.

We have also corrected and supervised your suggestions marked in yellow in the submitted pdf document. They have also been of great help to us.

Thank you very much for your review.

Author Response File: Author Response.docx

Reviewer 2 Report

The topic is interesting. However, the title of the paper and the abstract do not illustrate the content of the paper. The paper does not illustrate any connection between the STEAM disciplines and the gender inequality (as stated in the abstract). The paper seems to have nothing to do with any of the two, but only with the perception of education degree students on the use of robotics in education.

The paper need serious improvements before being considered for publication.

Some suggestions for improvement are:

clarify first for yourself what is the purpose of the present study and make sure that the research and its presentation follow the purpose and this is clear for the reader
shortly describe the four robotic kits used for the experiment (purpose, activity and applicability in education)

figures are too large, they should be much smaller and maybe put together (where the four kits are presented). The references in the text to the existing figures make no sense.
the two research instruments (the pre and post questionnaires) are to be included with all the items and scales in the appendix
title of table 1 is not appropriate for the content (Sample size ...)
if gender is an issue for you, the gender structure of the sample is to be included
shortly describe the two educational programs of the selected participants
clarify what is the relationship (if any) between the students who study at the two educational degree programs and the STEAM disciplines
results with the averages for each item are to be included in a synthetic form as a table
analysis of the results to be presented on groups of items (with the reference to the figures in the table) and to include the interpretation of the results, not descriptive presentation of the figures
results of the study are to be connected to the literature (similarities, differences between your study and other studies, etc)
the conclusions are in fact a summary of the results - they would rather be appropriate for the results section (without including the actual question in the comment), but to also include the interpretation
the research does not present anything to address the gender breach or inequality
English language needs to be revised (The main purpose of this goal is ...., etc)

Author Response

Dear Reviewer:

I have analyzed and reviewed the changes you have suggested to me. They have been very helpful in improving the manuscript.

For a better check, please allow me to copy and paste your improvement observations and my argumentation to each of them.

I hope and wish that with these changes you can verify that I have followed your instructions.

Sincerely.

1) Suggestion for improvement 1.

Shortly describe the four robotic kits used for the experiment (purpose, activity and applicability in education)

Answer 1 (in red in the word document).

Added improvement suggestion: add the description of the robotic kits used during the experience, together with their purpose and applicability in education.

Lines 84 to 147.

2) Suggestion for improvement 2.

Figures are too large, they should be much smaller and maybe put together (where the four kits are presented). The references in the text to the existing figures make no sense.

Answer 2 (in red in the word document).

Added improvement suggestion: reduce the size of the figures. References to figures in the text have also been removed.

Lines 88, 98, 116 and 139.

3) Suggestion for improvement 3.

Title of table 1 is not appropriate for the content (Sample size ...).

Answer 3 (in red in the word document).

Modified the table title as suggested by the reviewer.

Line 221.

4) Suggestion for improvement 4.

Shortly describe the two educational programs of the selected participants.

Answer 4 (in red in the word document).

Added the suggestion for improvement: describe the two educational programs where the research has been carried out and explain in which subjects and time of the course in which the intervention has been carried out.

Lines 57 to 81.

5) Suggestion for improvement 5.

Clarify what is the relationship (if any) between the students who study at the two educational degree programs and the STEAM disciplines.

Answer 5 (in red in the word document).

Añadida la sugerencia de mejora: describir las razones por las que se ha realizado esta investigación con ambos grados.

Lines 77 to 81.

6) Suggestion for improvement 6.

Results of the study are to be connected to the literature (similarities, differences between your study and other studies, etc).

Answer 6 (in red in the word document).

Added the suggestion for improvement: describe the coincidences or issues for which this type of work should be considered, to eliminate the gender gap that occurs in nursery and primary schools.

Lines 635 to 659.

7) Suggestion for improvement 7.

The research does not present anything to address the gender breach or inequality.

Answer 7 (in red in the word document).

Added the suggestion for improvement: the research aims to raise awareness that this type of basic training in educational robotics must be taught in the Faculties of Educational Sciences of the Universities, since it is the only way that future female teachers can raise awareness female students in schools that they are also capable of learning programming, robotics and other engineering related specialties. One of the reasons why there are few programmers, engineers, and computer scientists is because no one has told them that they can too.

Lines 635 to 639.

8) Suggestion for improvement 8.

English language needs to be revised (The main purpose of this goal is ...., etc).

Answer 8 (in red in the word document).

English language corrected at the suggestion of the reviewer and revised the rest of the document.

Lines 25 to 26.

In addition, more changes have been made than have been suggested by reviewer 1.

Thank you very much for your review.

Author Response File: Author Response.docx

Round 2

Reviewer 1 Report

Line 89: "...basic programming skills and more" - please be concise and use accurate scientific language: Which competences are being addressed and majorly promoted by the Mouse Kit?

The competences and skills addressed with the interventions are not mentioned or addressed in the questionnaires and the shown results. Please show the structure of the questionnaire and main items investigated there: e.g. motivation, programming skills in Scratch, etc...

Now the standard deviation is shown and argued on its basis on the results and conclusion, which increases the quality of the analysis. However, a pre- post-design allows as well to investigate the significance in the answers based on a variance analysis (i.e. with significance test from inference-statistics). Please perform such tests and argue based on the statistical significance of the results obtained, as this is a necessary extension to the scientific and experimental method you applied (pre/post test).

Some of the results could (or should) be shown in graphs with the averages, and standard deviations and significance. This would make the analysis more visual and clearer.

Author Response

Dear Reviewer:

I have analyzed and reviewed the changes you have suggested to me. They have been very helpful in improving the manuscript.

For a better check, please allow me to copy and paste your improvement observations and my argumentation to each of them.

I hope and wish that with these changes you can verify that I have followed your instructions.

Sincerely.

1) Suggestion for improvement 1.

T Line 89: "...basic programming skills and more" - please be concise and use accurate scientific language: Which competences are being addressed and majorly promoted by the Mouse Kit?

Answer 1 (in red in the word document).

We have removed that colloquial expression and specified the skills that are developed and enhanced using the Robot Mouse Kit. Lines 95-99.

2) Suggestion for improvement 2.

Answer 2 (in red in the word document).

We have compiled a list of the competences (Table 1) that are developed when using robotic educational kits (lines 163-168). We have also included a table (Table 2) with the questions in the questionnaire (lines 194-196).

3) Suggestion for improvement 3.

Answer 3 (in red in the word document).

We have applied different statistical tests, as the reviewer has suggested. To compare the mean values obtained in the pretest and posttest, we have applied the t-test. This comparison provides an inferential statistic to assess whether the difference between the two means obtained in the pretest and posttest is statistically significant. In our case, we have applied the independent samples t test. But to perform the T tests, the data must be distributed normally. To find out if they are, we have first applied the Kolmogorov-Smirnov (Ks) test (Table 7). Once we know that the distribution is normal, we have applied the T test for samples (Table 8). Levene's test for equality of variances tells us whether or not we can assume equal variances. Therefore, if the probability associated with the Levene statistic is >0.05, we assume equal variations, and if it is <0.05, we assume different variations. After assuming equal variations, we observe the t statistic with its level of bilateral significance, this value informs us about the degree of compatibility between the hypothesis of equality of means and the difference between the observed population means; in our case it is less than 0.05. Lines 365-391.

4) Suggestion for improvement 4.

Some of the results could (or should) be shown in graphs with the averages, and standard deviations and significance. This would make the analysis more visual and clearer.

Answer 4 (in red in the word document).

As suggested by the reviewer, we have included a stacked bar graph (Figure 5) showing the items that have obtained a high perception score on educational robotics kits. Standard deviations and their variances have also been included, in order to make the results obtained more visual and clear. Lines 353-355.

Thank you very much for your review.

Author Response File: Author Response.docx

Reviewer 2 Report

The authors put some effort and included some of the revisions that they have been recommended. The paper has improved from the initial version, but still needs further improvements:

not clear how is it handled the gender inequality issue: are these robotics classes about attracting more females to become technologically prepared teachers? OR are they about attracting more female early age pupils to the technical and IT subjects? OR both OR anything else? This needs to be specifically addressed in the paper.
what would be the contribution of the robotic kits to the development of the IT skills/programming skills for children? The connection between the use of robotic kits in university classrooms/intended to be used with young children AND the improvement of ICT competences of women to study/teach/work in robotics/programming/technical fields. The relationship is not obvious, is not self-explanatory, you have to point it out, to express it in the paper, to make it clear what leads to what. Otherwise it is not clear how the use of robotic kits influence the diminishing of gender inequality
more details need to be included about the competences to be developed by each of the robotics kits. Include them in a comparative table and comment if they are complementary, if they support the development of the same competences. What is the age of children they address (each of them)
the questionnaire (with all questions and scales) needs to be included in the appendix OR/AND
include all results for all the questions in a table. Instead of describing as text the results/figures include them in a centralizing table and in the text discuss
- which aspects score the highest/the lowest and why? Try to interpret.
- for which aspects there were differences between pre and post answers. Why? Try to interpret/explain.
- explain/comment on (not describe) means, standard deviations and variances
one main weakness of the paper is that it is far too descriptive. you need to explain/interpret the results. Not only announce the numbers for each question/ Why these answers? What do we get from them?
again, the conclusion section is too descriptive (a simple list of the questions). You need to interpret: what do results mean? how do you explain them? why did they answer that way? Are there personal, institutional, cultural, national influencing factors and explanations for this? what are those?
there a lot of unclear statements in the paper. For instance:
- rows 284-286 : what does it mean? how do you interpret?
- row 300 "female teacher training programme" Not clear! Is there a female as opposed to a male or mixed teaching program? re-formulate/ clarify!
- row 310 "could have answered randomly". The advise is to revise/eliminate this comment. It questions the reliability of all answers. Find a different explanation.
- rows 394-395 "In the case of girls, this perception is clearly conditioned by the feeling of capacity". Not clear what feeling of capacity are you talking about. Re-phrase!
- take out the paragraph rows 312-321. It is a simple list.
Still English and editing mistakes: r. 296-297; r 124; r 56-57 (names of degrees with capital letters); r.73
How is your study and its findings similar or different from other studies on the topic. Listing what other studies content is, does not make the connection with your study. You have to comment on the similarities and differences between different studies and yours.
Comment on the contributions of the study to literature/practice/educational policy if any.

Author Response

Dear Reviewer:

I have analyzed and reviewed the changes you have suggested to me. They have been very helpful in improving the manuscript.

For a better check, please allow me to copy and paste your improvement observations and my argumentation to each of them.

I hope and wish that with these changes you can verify that I have followed your instructions.

Sincerely.

1) Suggestion for improvement 1.

Not clear how is it handled the gender inequality issue: are these robotics classes about attracting more females to become technologically prepared teachers? OR are they about attracting more female early age pupils to the technical and IT subjects? OR both OR anything else? This needs to be specifically addressed in the paper..

Answer 1 (in red in the word document).

One of the priority objectives of this study was to attract more women to technical and scientific careers, especially those related to programming. You only have to see a classroom in Spain for software engineering or computer engineering to verify the small number of women who are studying them. Increasing this number of female students can be achieved by making future female teachers wake up their future student girls the idea of studying this type of university studies and that they can do it just like men. Lines 82-87.

2) Suggestion for improvement 2.

What would be the contribution of the robotic kits to the development of the IT skills/programming skills for children? The connection between the use of robotic kits in university classrooms/intended to be used with young children AND the improvement of ICT competences of women to study/teach/work in robotics/programming/technical fields. The relationship is not obvious, is not self-explanatory, you have to point it out, to express it in the paper, to make it clear what leads to what. Otherwise it is not clear how the use of robotic kits influence the diminishing of gender inequality.

Answer 2 (in red in the word document).

The kits used are complementary and develop almost the same skills, three of them are considered "ground robots" (Robot Mouse Colby, mBot and Ozobot) and one of them is a programmable board (Makey-Makey). Lines 166-168.

Table 1 shows the different competencies and abilities that each of the educational robotic kits develops. Lines 163-165.

Increasing this number of female students can be achieved by making future female teachers wake up their future student girls the idea of studying this type of university studies and that they can do it just like men. Lines 82-87.

3) Suggestion for improvement 3.

More details need to be included about the competences to be developed by each of the robotics kits. Include them in a comparative table and comment if they are complementary, if they support the development of the same competences. What is the age of children they address (each of them).

Answer 3 (in red in the word document).

Table 1 shows the different competencies and abilities that each of the educational robotic kits develops. Lines 163-165.

4) Suggestion for improvement 4.

The questionnaire (with all questions and scales) needs to be included in the appendix OR/AND.

Answer 4 (in red in the word document).

The table 2 shows the complete list of questions and what they develop, they are all related to digital competence. Lines 191-196.

5) Suggestion for improvement 5.

Include all results for all the questions in a table. Instead of describing as text the results/figures include them in a centralizing table and in the text discuss.

Answer 5 (in red in the word document).

The table 6 show the results obtained in each of the questions, indicating the average scores, the standard deviations and the variances obtained, which measure the level of perception that female students have about the convenience of the use of educational robotics in schools. Lines 340-345.

The other reviewer has not suggested removing the paragraphs with the data and has suggested adding a chart with the highest scoring questions. Lines 346-353.

To compare the mean values obtained in the pretest and in the posttest, the t test has been applied. This comparison provides an inferencial statistic to assess whether the difference between the two means obtained in the pretest and in the posttest is statistically significant. In our case we have applied the independent samples t-test. But to perform the T-Tests, the data must be distributed in a normal way. To know if they are, we applied the Kolmogorov-Smirnov (Ks) test, and we check the significance level, if it is less than 0.05 the distribution is not normal, if it is greater than 0.05 the distribution is normal. In our case the distribution was normal (significance level 0.060) (see Table 7 and 8). Lines 355-389.

6) Suggestion for improvement 6.

Which aspects score the highest/the lowest and why? Try to interpret.

Answer 6 (in red in the word document).

The same happens with the global variance, in the pre-test: 0.749 and 0.759 in the post-test, there is no variability in the dispersion of the answers, only in the items that asked about the degree of knowledge they had about the languages of Scratch and mBlock programming. The reason for these low scores can be explained because these students did not attend any of those two practical seminars that we organized with them (mBot kit and Makey-Makey kit) and only attended the sessions of the other two educational robotic kits (Robot Mouse Colby and Ozobot). It may also be because the students were distracted when we have been explaining in class that this programming language was called Scratch (for the Makey-Makey kit) and mMblock (for the mBot kit), programming with this language was essential to be able to move the mBot kit and to interact with the Makey-Makey board. Lines 401-410.

The items with the lowest scores before carrying out the practical sessions with the robotic kits (ordered from lower to higher score, up to a maximum of 2.00 points), were those related to having basic knowledge about robotics, robots and programming language, as well as the items that were enunciated negatively to prevent the participants from answering randomly. This is logical because in Spain the training in programming and robotics that students who enter the University have is very little or nil. Currently, the only training in educational robotics and precomputational thinking is being developed in the last years of secondary education (about 15 and 16 years old). We believe that training in educational robotics should start from the age of 4 and extend to all secondary education. Lines 413-431.

7) Suggestion for improvement 7.

One main weakness of the paper is that it is far too descriptive. You need to explain/interpret the results. Not only announce the numbers for each question/ Why these answers? What do we get from them?.

Answer 7 (in red in the word document).

To expand the study and not make it so descriptive, the t test has been applied and compare the mean values obtained in the pretest and in the posttest. This comparison provides an inferencial statistic to assess whether the difference between the two means obtained in the pretest and in the posttest is statistically significant. In our case we have applied the independent samples t-test. But to perform the T-Tests, the data must be distributed in a normal way. To know if they are, we applied the Kolmogorov-Smirnov (Ks) test, and we check the significance level, if it is less than 0.05 the distribution is not normal, if it is greater than 0.05 the distribution is normal. In our case the distribution was normal (significance level 0.060) (see Table 7 and 8). Lines 355-389.

8) Suggestion for improvement 8.

Again, the conclusion section is too descriptive (a simple list of the questions). You need to interpret: what do results mean? How do you explain them? Why did they answer that way? Are there personal, institutional, cultural, national influencing factors and explanations for this? What are those?.

Answer 8 (in red in the word document).

Comments have been included that try to explain the reasons for the students' responses, and to understand what the educational robotics training system currently is in Spain and why this has to change. Lines 404-410, lines 427-431 and lines 504-507.

9) Suggestion for improvement 9.

Rows 284-286 : what does it mean? How do you interpret?.

Answer 9 (in red in the word document).

This means that students do want to receive training in educational robotics, they are interested and perceive that it is important for their comprehensive training as teachers. Lines 295-297.

10) Suggestion for improvement 10.

Row 310 "could have answered randomly". The advice is to revise/eliminate this comment. It questions the reliability of all answers. Find a different explanation.

Answer 10 (in red in the word document).

Comment deleted by the authors. The reason for these low scores can be explained because these students did not attend any of those two practical seminars that we organized with them (mBot kit and Makey-Makey kit) and only attended the sessions of the other two educational robotic kits (Robot Mouse Colby and Ozobot). It may also be because the students were distracted when we have been explaining in class that this programming language was called Scratch (for the Makey-Makey kit) and mMblock (for the mBot kit), programming with this language was essential to be able to move the mBot kit and to interact with the Makey-Makey board. Lines 406-412.

11) Suggestion for improvement 11.

Rows 394-395 "In the case of girls, this perception is clearly conditioned by the feeling of capacity". Not clear what feeling of capacity are you talking about. Re-phrase!.

Answer 11 (in red in the word document).

Reformulated text: In the case of girls, this perception is highly conditioned by their feeling of capacity; women often believe that they are not prepared to program, even without trying. During the practical sessions they stated that they found it very easy to program and believed that it was much more difficult. Lines 506-509.

12) Suggestion for improvement 12.

Take out the paragraph rows 312-321. It is a simple list.

Answer 12 (in red in the word document).

Text reformulated and converted to a simple list. Lines 419-418.

13) Suggestion for improvement 13.

Still English and editing mistakes: r. 296-297; r 124; r 56-57 (names of degrees with capital letters); r.73.

Answer 13 (in red in the word document).

Corrected the names of the degrees in uppercase and converted to lowercase. Lines 56-58, 60-62, 67, table 5, and lines 317-318.

Thank you very much for your review.

Author Response File: Author Response.docx

Round 3

Reviewer 2 Report

Authors clearly tried to incorporate the feed back they received.

The paper improved as compared to its first version.

Some editing recommendations:

all tables to be written with a smaller line space, so that to become more concentrated on the page
in tables 2 and 6 to either number the items or include lines in the tables. As they are no, are difficult to read
raw 353 replace inferencial with inferential
check for some other minor editing inexactities

Good luck!

Author Response

Dear Reviewer:

I have analyzed and reviewed the changes you have suggested to me. They have been very helpful in improving the manuscript.

For a better check, please allow me to copy and paste your improvement observations and my argumentation to each of them.

I hope and wish that with these changes you can verify that I have followed your instructions.

Sincerely.

1) Suggestion for improvement 1.

All tables to be written with a smaller line space, so that to become more concentrated on the page.

Answer 1 (in red in the word document).

We have modified the line space, making it smaller: to 1. Lines 161-163, lines 190-193, lines 340-341, and lines 365-366.

2) Suggestion for improvement 2.

In tables 2 and 6 to either number the items or include lines in the tables. As they are no, are difficult to read.

Answer 2 (in red in the word document).

We have added broken lines for a better reading of tables 2 and 6. We do not know if the publisher will remove them in the final version, since it is a table format not supported by the magazine. Lines 190-192 and lines 340-341.

3) Suggestion for improvement 3.

Raw 353 replace inferencial with inferential.

Answer 3 (in red in the word document).

We have corrected the error found in the inferential word. Line 351.

4) Suggestion for improvement 4.

Check for some other minor editing inexactities.

Answer 4 (in red in the word document).

We have corrected different minor editing errors and inaccuracies that we have detected in the text. Lines 36, 41, 171, 344, 413 to 422, 490, and 491.

Thank you very much for your review.

Author Response File: Author Response.docx