Non-Digital Games That Promote Mathematical Learning in Primary Years Students: A Systematic Review

Round 1

Reviewer 1 Report

Dear Authors,

Thank you for an interesting focus to this manuscript. A systematic review in this area would be a welcome contribution to the field. I hope that you pursue revising the manuscript and resubmitting it in the future.

Since major revisions are warranted in my view, I will provide some general ideas of how to approach these revisions and hope you apply them throughout. Here are the broad areas to consider:

(1) The introduction would benefit from connection to a range of researchers who do work in the area.

(2) The background needs much more elaboration. There are many different kinds of games and the overview provided here ends up impacting the results and discussion later on. A paragraph-long explanation of different types of games (e.g., instructional, commercial, cultural/ancient, etc.) is needed. As well, an exploration of the difference between gamed-based learning and gamification is necessary -- these two approaches are grounded in different epistemological stance or learning theories that the reader needs to understand. Finally, the quality of the sources in this section needs to be raised (i.e., more academic articles compared to proceedings, professional articles, etc.).

(3) In the methodology section, there are details that need more elaboration (e.g., line 163 on page 4; line 197 on page 4). As well, the searches need to be redone as I am aware of articles that fit directly within your criteria that were missed. For example, I alone have 7 articles and 3 proceedings published within your selection criteria and I think only 1 is present in this manuscript. As well, I am aware of work from my country that is within the selection criteria that were not included. An appendix with the 34 pieces would add in both the review process and for the eventual audience's reading.

(4) The first half of the section on RQ1 in the Findings could be condensed by being purely descriptive and not speculative on why certain categories have more or less publications. The second half of the section on RQ1 in the Findings could then be expanded -- more details on "game/intervention", "definition of a game," and "mathematical focus" in particular is needed. Since it was difficult to evaluate the Findings without a discrete list of the 34 articles reviewed and because the selection process missed articles, I concluded my review here.

Again, I would encourage you to continue to pursue this manuscript. A systematic review in this area would be a very welcome contribution to the field. Please examine your selection process to be sure to capture more of the excellent work being done in the area.

Thank you.

Author Response

Dear reviewer

Thank you for taking the time to read and review our manuscript. We note that we received your review after we had been already received an invitation to revise and resubmit our manuscript, and feedback from the first three reviewers (Reviewer 2, Reviewer 3 and Reviewer 4). Consequently, we made some additional changes to the manuscript in response to your valuable feedback. Each of the points raised in your review, and our response to each of them, are outlined below.

(1) The introduction would benefit from connection to a range of researchers who do work in the area.

We acknowledge that there is always more literature we could incorporate into a paper, particularly one as broad in potential scope as a systematic review in educational games. However, we believe that our literature review sufficiently covers much of the key literature in the area. Reviewer 3 concurs, stating: " I found the review of the literature good. It is a detailed and carefully done work with many current references, English articles from periodicals. Using and abundant (67) review of literature, the theoretical part of the dissertation is detailed, accurate and demanding." 

(2) The background needs much more elaboration. There are many different kinds of games and the overview provided here ends up impacting the results and discussion later on. A paragraph-long explanation of different types of games (e.g., instructional, commercial, cultural/ancient, etc.) is needed. As well, an exploration of the difference between gamed-based learning and gamification is necessary -- these two approaches are grounded in different epistemological stance or learning theories that the reader needs to understand. Finally, the quality of the sources in this section needs to be raised (i.e., more academic articles compared to proceedings, professional articles, etc.).

We have elaborated on the background in response to this comment. In particular, we have now more carefully distinguished between mathematical games, and gamified mathematical classrooms/ the process of gamification. To achieve this, we have drawn on Gough's expression of 'pseudo games' to describe those activities which have some but not all features of mathematical games. We note that these need to be carefully distinguished from the notion of gamification, and appreciate the reviewer highlighting this oversight in the initial draft. The revised text now reads: "Using games in the mathematics classroom needs to be distinguished from the broader process of educational gamification, which refers to the process of incorporating game mechanics and game design techniques into educational settings to modify student motivations and behavior [10]. Whereas a mathematical game refers to a discrete activity with specific mathematical learning objectives, gamification effectively involves transforming the mathematics classroom into the game space."

(3) In the methodology section, there are details that need more elaboration (e.g., line 163 on page 4; line 197 on page 4). As well, the searches need to be redone as I am aware of articles that fit directly within your criteria that were missed. For example, I alone have 7 articles and 3 proceedings published within your selection criteria and I think only 1 is present in this manuscript. As well, I am aware of work from my country that is within the selection criteria that were not included. An appendix with the 34 pieces would add in both the review process and for the eventual audience's reading.

Thank you for the comment. We have elaborated the methodology and description of the included studies in line with Reviewer 2's comments. Furthermore, we note that Table 1 in the appendix did in fact document the 34 studies included in the review (although we understand that this table was perhaps not made available to reviewers). With regards to your queries around the included studies, we believe the process we have followed is extremely rigorous. Although it is possible that studies that one might have expected to be in scope for our review were not ultimately included, we believe that our search strategy was comprehensive and is sufficiently transparent to be replicable by other researchers. Likewise, our study selection process and our PRISMA Flow Diagram clearly document the decisions we made to exclude studies at each point in the process.   

(4) The first half of the section on RQ1 in the Findings could be condensed by being purely descriptive and not speculative on why certain categories have more or less publications. The second half of the section on RQ1 in the Findings could then be expanded -- more details on "game/intervention", "definition of a game," and "mathematical focus" in particular is needed. Since it was difficult to evaluate the Findings without a discrete list of the 34 articles reviewed and because the selection process missed articles, I concluded my review here.

We have made refinements to how the results are presented, and in particular, endeavoured to be more descriptive as to which particular studies have been included (in response also to Reviewer 2's feedback). As noted previously, we have included the 34 studies and their characteristics as an appendix to the current review.

Again, I would encourage you to continue to pursue this manuscript. A systematic review in this area would be a very welcome contribution to the field. Please examine your selection process to be sure to capture more of the excellent work being done in the area.

Thank you for the comment. As noted, we believe our selection process is rigorous. Very importantly, we believe the transparency enables it to be replicated by other researchers pursuing systematic reviews in this area in the future. 

Reviewer 2 Report

this manuscripts reports an systematical review with PRISMA. In sum, this work has a potential  to contribute to the literature on mathematical learning with games. However, the manuscript contains a series of problems, that need to be resolved.

Theoretical Background

-      Line 78f Reference is needed

-      L82: the research question could be derived even more strongly from the theoretical background

-      Line 118f:

-      Mixing of theory, method and results. In L118f, criteria points are introduced that would define a mathematical game – at the same time, aspects are already mentioned that belong more in the methods and results section: e.g. the presentation of the study by Casey et al. (16). In addition, it is unclear to the reader why games are included in the systematic review that only partially fulfill or contradict the above criteria: for example, one by Casey et a. (16) is listed that is based on luck alone. It is also unclear how many other games also fulfill other criteria. Is it just this one?

-      In addition, lack of justification of the criteria for a mathematical game

Method

-      L 198f: What is the justification for the fact that two papers are not studies and are nevertheless included in the analyses? The inclusion criteria should be justified in more detail.

-      L 200: Listing of all included studies would be desirable at this point. The list should contain the following information: Outcomes, N (Pupils, Teachers), country, SES etc., grade level, definition of game, type of effects, how many experimental and control groups, type of data

Results and Discussion

-      APA standards when reporting samples – in the text samples were never italicized, e.g. n=7 (line 241), but also throughout the lines and paragraphs thereafter

-      L 236f: how were the study characteristics determined?

-      L 240-244 (years of publication): confusing enumeration of the number of publications when viewed simultaneously with the graph (while explicitly referring to Figure 2) -> instead: Figure with each year from 2003-2022

-      L 249f The generalization in the discussion seems speculative. It should be mentioned that this only refers to a very selective selection of papers.

-      L278: The justification seems inappropriate, as only studies up to 2023 were included.

-      L291: Categorization seems arbitrary. Why have these been grouped together? Please insert the reason for the categorization.

-      L 309: Were all studies included in this analysis of SES? If not, please specify this. The selection of studies seems arbitrary.

-      Line 318 of the Game/Intervention Details paragraph: To what extent do animated games count as non-digital games here? Or what is meant by animated games if they are counted as non-digital games in the paper? Intuitively, I understand this to mean digital games.

-      L 324f: Presentation depending on the target group would be appropriate.

-      L329: Duration of intervention: more stringent presentation required. Breakdown by kindergarten G1-2, grades 3-4 would be desirable. In addition, the presentation gives the impression that studies were selected at random. Please state how many studies of the 34 papers are present.

-      L 336f: Assertion. Evidence of the analysis would be desirable.

-      L339: The terms "more criteria/several studies" are imprecise. Please describe precisely or state the number of studies of the 32 studies.

-      Lines 368-373: Passage described inaccurately. It is reported that four studies can each be assigned to 2 content areas. However, some studies are listed with more than 2 content areas, e.g. Study 48. Explain more clearly how the content areas were formed and how they were applied. In addition, not all studies are discussed. It should be explained why a new selection was made.

-      L374. Incomplete sentence. Unclear what is meant with the two studies.

-      L376f/400f: Establish reference to NCTM.

-      L 383: often imprecise; more precise

-      Line 417: the research question is "How does playing non-digital games impact student and/or teacher outcomes in primary school setting" - however, the answer to the question does not address teacher outcomes, which are also part of the research question. Nor does it explain the extent to which no results were found on teacher outcomes. It is also unclear why studies from the pre-school sector were taken into account (e.g. Skillen).

-      L 420: unclear what is meant by learning outcomes

-      L423f: Classification seems arbitrary. It might be possible to differentiate between short-term and long-term effects in the studies.

-      L424: "several studies" is imprecise. How many studies out of 32? Why are these listed?

-      L 429: why "however" if this sentence supports the previous one in its statement

-      Line 444: Was the experimental group compared with a control group? Or with another experimental group? Unclear

-      L465f: How are long-term effects defined? Doesn't the Skillen study also have a follow-up?  Didn't this study also examine long-term effects? Write more explicitly.

-      L471/477/491/501/511: how many of 32 studies were included in this analysis? Please report. The selection seems arbitrary

-      - L 477: Why is a different term used here than in L 420f?

-      L491: shouldn't this section rather be called Emotion and Motivation? How is it justified that enjoyment is a motivation? Stringency: The use of different levels is not comprehensible. Either everything at one level (motivation/emotion) or everything at construct level (enjoyment)

-      L539: "included publication" shouldn't it be called paper?

-      L543: are the 18 studies included in the 22? If not, please describe them in more detail.

-      L551: how can the stars be distinguished from each other? What are the criteria? Please add. The gradations are not clear from the description

-      L564: unclear which studies are involved

-      Explain ranking from table 2

Figures and Tables

-      Table 1 not found? Makes it difficult to understand the text (According to the text, it can be found in the appendix, but Table 2 is there)

-      Figure 2, 4, 5 and 6: no labeled y-axis & no y-axis generally included

-      In contrast to the other illustrations, Table 2 is not flush with the text, but clearly shifted to the left

References

-      Literature 1 (line 661) and literature 7 (line 674) are given with an invalid doi link

-      The titles of the references / journal names were never written in italics -> but this would be required according to both citation standards "ACS reference style" and "Chicago reference style"

Author Response

Please see attached document

Author Response File: Author Response.pdf

Reviewer 3 Report

The theme of the paper is important, topical and exciting. The work is original regarding its theme and due to the fact that it focuses on non-digital, games-based, empirical studies in the primary mathematics classroom over the past two decades from 2003-2022. 

The review presents an analysis and synthesis of 34 manuscripts, representing 32 distinct studies, highlighting the similarities and the differences. It is topical, recent and up-to-date. The questions examined are good. I found the review of the literature good. It is a detailed and carefully done work with many current references, English articles from periodicals. Using and abundant (67) review of literature, the theoretical part of the dissertation is detailed, accurate and demanding.

The author’s main objective was to examine the impact of non-digital mathematical games. They find many interesting factors in this review.  This manuscript is written very well. The use of English language is very good, and the sections can be easily followed. This is a high-quality paper related to Mathematics Education, to Didactics of Mathematics and Teaching Mathematics. It meets the expectations and profile of the Education Sciences journal.

Author Response

Thank you for the positive feedback on our article. We are also excited about our research, and believe that the systematic reviews addresses a genuine gap in the research literature. 

Reviewer 4 Report

The author's understanding of research methodology and grouping of relevant studies is very good. However, I recommend providing a more in-depth analysis of the weaknesses and potential biases in the included studies. Further explanation of the inclusion and exclusion criteria could also clarify the selection of included studies

The quality of the English in this article is quite good, but some sentences appear unclear and may confuse readers. I suggest revising it so that the expressions are more concise and easier to understand. "Word selection and sentence structure need further attention to improve communication clarity."

Author Response

Please see attached document.

Author Response File: Author Response.docx

Round 2

Reviewer 1 Report

Dear authors,

Thank you for attending to some of the revisions. There are elements of the manuscript that have been improved. I appreciate your comments on the choices of revisions made or not made. Thank you for the note on the Appendix -- it seems to still not be included in the review materials.

However, there were items in my earlier review (statements in italics) that were not addressed in your comments that still require revision:

1) There are many different kinds of games and the overview provided here ends up impacting the results and discussion later on. A paragraph-long explanation of different types of games (e.g., instructional, commercial, cultural/ancient, etc.) is needed. These types of games impact why they are used in research/classrooms, how they are used in research/classrooms, the impact on students' learning and teachers' teaching, and the impact on research results.

2) Finally, the quality of the sources in this section needs to be raised (i.e., more academic articles compared to proceedings, professional articles, etc.). While extra explanations were provided, no new citations were incorporated.

3) As well, the searches need to be redone as I am aware of articles that fit directly within your criteria that were missed. This request for revision is substantial and critical as it affects the entire manuscript. If there were just a few pieces that were missed, I would concur with your perspective that search could be repeated and the pieces included are sufficient. However, there are just too large a volume of published research that is missing for this report to be trustworthy. The literature search needs to be redone with parameters and/or search terms that will capture an accurate set of articles that fit the scope of the systematic review.

I hope that you will continue to revise this manuscript as a systematic review in the area would be a great contribution to the field.

Thank you.

Author Response

Dear reviewer

Thank you for your continuing interest in our paper. We address your points in turn:

1) We agree that the type of game used is an important consideration, however we feel that our discussion of the 'definition of a game', currently at over 500 words, which includes 8 references, references to commercial games/ board games/ gamification/ criteria for a mathematical game, adequately contextualises the topic. We note that the 'game/ intervention' details are included in our appendix describing the studies in scope for the review (i.e., Table 1), and we are uncertain how an additional descriptive paragraph would add to this section. 

2) We believe that our mix of references contains a sufficient number of journal articles (as opposed to conference proceedings/ professional papers) in this section. Given that the purpose is to describe how games are used in mathematics education contexts, and what constitutes a game, we think that including professional journals (almost all of which are peer reviewed), speaks to what teachers actually do in classrooms, and are particularly relevant for this section. However, if there are particular research papers that you would like us to consider to broaden our description of the background, we are open to considering incorporating them.  

3) Beyond reassuring the reviewer that our research process was rigorous, transparent etc., we are not sure what else we can do to address this criticism. The search was checked and double checked by multiple authors, the Cochrane Handbook for systematic reviews etc. was followed. As the reviewer and editor knows, establishing a search strategy and criteria for a systematic review is a significant undertaking, and cannot be easily changed or modified at the last stage easily. We understand you might have expected particular papers to be included that were not there, however without a list of proposed overlooked papers being provided by the reviewer, we are unable to cross-check these with our initial search to see whether they were identified but ruled out due to being out of scope etc. 

Kind regards

The authors