Examining the Design Characteristics of Mnemonics Serious Games on the App Stores: A Systematic Heuristic Review

Abstract

Research shows mnemonics promote knowledge retention in different contexts; hence, they are increasingly being used in serious games aimed to support long-term learning while providing “edutainment.” However, there is limited research on their effectiveness. As such, we conducted a systematic review of 32 mnemonics mobile apps and evaluated them using two established frameworks from the literature. Our analysis revealed that most of the games teach language or medicine, take the form of puzzles or quizzes, and feature acronyms and/or images, with players rating them at least three out of five stars on average. All 32 apps supported feedback, interactivity, and challenge. A few supported agency, identity and self-presence, while many did not support key characteristics such as social and spatial presences. The overall finding indicates a need to create a mnemonics-based and tailored framework to guide the design of mnemonics games in the future to make them more effective.

1. Introduction

1.1. Motivation

Nowadays, students at various education levels are having more difficulty learning and remembering material taught in classrooms, particularly in memory-intensive subjects and courses due to numerous factors [1,2,3,4]. One major factor is the way memory-intensive subjects such as biology [5], psychology [6], history [7], social science [8,9], humanities [10,11], and English/foreign languages [12] are taught, as well as the very nature of the subjects themselves making them difficult to learn [5]. Another factor is that students in the modern era are more susceptible to the increasingly distracting influence of addictive technologies, especially social media networks, which can negatively impact their memory [13,14]. Also, many students have to work to earn money to pay for their education, which leaves them with less time and energy to study [15]. In addition, the fact that there are constantly new discoveries being made means there is more and more that students are required to study. Moreover, students do not have many chances to study together since most study methods focus on a single individual rather than a team [4]. Bloom’s taxonomy regards remembering as a low-order cognitive process; however, that does not make it any less important. Instead it serves as the foundation that higher-order cognitive processes, such as analysis and evaluation, build upon [16]. After all, it is very difficult to analyze, evaluate, or otherwise discuss something if we cannot remember details about it. According to Jensen, “learning and memory are two sides of a coin. You can’t talk about one without the other” [17]. This fact, along with all the aforementioned negative factors, means that the development and use of effective learning strategies to improve learning and content retention has become a greater priority. Fortunately, there are already numerous techniques meant to improve learning and long-term memory; one major example is the use of mnemonics [1,18].

Mnemonics are centered around the use of various types of connections that can be made with the information that needs to be remembered, such as cues, concreteness, and associations. Research has proven that mnemonics are effective in enhancing long-term memory. For example, in 1967, Miller found that students who regularly used mnemonic devices to support their study increased their test scores up to 77% [19]. Regarding memorization-intensive courses and subjects in particular, mnemonics have been shown to help in many of them, including but not limited to mathematics/statistics [20], economics [13] biology [21,22], chemistry [23], psychology [8,9], and engineering [24]. Not only do mnemonics help to recall information, they come with several “bonuses”, too; they free up more cognitive resources for higher-order thinking and help reduce stress and anxiety during tests [20]. Of course, mnemonics need to be applied right to be effective, but how do we know they are “done right”? According to Seay and McAlum, the four main properties that mnemonics depend on to be effective are meaningfulness, elaboration, association, and visualization [13]. These properties are already being utilized by both knowledgeable instructors and students to develop better mnemonics that can help them learn and remember what they learned for longer periods of time.

Though we have mentioned how some modern technologies can be distracting [13,14], others can serve as tools and have a positive effect instead. One such type of tool is serious games, which are games designed to educate rather than solely entertain [25], hence the word “serious”. While entertainment is certainly not the primary goal, making a serious game entertaining likely increases the effectiveness it has on players to take action both in and out of the game. Indeed, Ravyse et al.’s systematic review, which gathered articles about interactive games, concluded that it is necessary to create a fun learning process so that players choose to keep learning [26]. They also point out that elements such as stories not related to the learning content, overwhelming text, severe time limits, and complex interfaces detract from the purpose of the game due to increasing cognitive load and reducing learning efficacy. The game already gives players so much to handle with its learning content, after all. Regardless, serious games have been shown to be helpful in various fields, such as health education [27], medical care [28], chemistry [29], and cybersecurity [30].

Mnemonics are effective, and serious games are effective. As such, it is unsurprising that there have become an increasing number of serious games that use mnemonics to teach memorization-intensive subjects. What is surprising is that some people still tend to disregard mnemonics; they believe that mnemonics are difficult to implement in certain contexts, and that they only benefit students in certain contexts and for a limited time [1]. This, along with the fact that serious games in general are becoming more and more common in education due to enhancing long-term memory [31,32,33], makes it important to research mnemonics serious games and discover how effective they are, what types of such games there are, and other important traits about them.

Fung and Oyibo [34] have conducted a scoping review of papers that detailed studies involving mnemonics serious games mostly created for these studies and presented several important findings. The majority of these games were for personal use rather than collaborative, and they were shelved after their studies instead of being released to the public; as such, the games were not able to spread their influence. Also, mnemonics in these games were created by the game designers rather than the players, denying the latter the chance to come up with their own mnemonics. Moreover, these games were generally successful at improving users’ memory, showing the effectiveness of both mnemonics and serious games. They were also generally found enjoyable, but not particularly memorable, which may need to be worked on. Despite the review’s contributions to the body of knowledge, the paper mainly involved other papers; as stated before, most of these papers described studies involving games created just for them and not released to the public. Research is required for games created for public use.

Ejaz and Oyibo [35] conducted a scoping review of papers that detailed studies about mnemonics creation tools meant to guide users on creating mnemonics themselves and found several important findings. There are already a handful of these tools, particularly for pedagogical purposes, and the number is rising. These tools have potential to improve both short- and long-term memory, though more research on long-term effects needs to be conducted. Regardless, the potential suggests they could assist well in the classroom, yet they have not been incorporated much in or outside the classroom other than language learning. The authors recommend personalizing the tools to suit users’ preferences and learning styles, as well as a feedback system that evaluates how good a particular mnemonic is. They also recommend that students and instructors of memory-intensive subjects receive training in mnemonic creation so as to bring out the full potential of mnemonics. Despite the review’s contributions to the body of knowledge, it has not discussed the effects of using mnemonics creation tools and mnemonics serious games together.

Unfortunately, other than the two papers detailed above, there has not been much research into tools meant for either creating or practicing mnemonics. In other words, to the best of our knowledge, there is not much prior work we can discuss. As such, we decided to perform our own systematic heuristic review that first gathers mnemonics in erious games that take the form of apps and then synthesize and summarize important traits about them. We hope that the synthesized and summarized findings will close several research gaps and answer questions that currently lack an answer, and we hope this knowledge can also be used to create better mnemonics serious games and improve existing ones.

1.2. Objectives

The objective of this systematic heuristic review is to analyze, summarize, and synthesize the characteristics of mnemonics serious games on the App Store and Google Play by answering the following research questions:

• What types of mnemonics serious games are available on the app stores?
• What are the game elements employed in mnemonics serious games?
• How well do they foster experiential learning?
• Are there relationships between the types, general UX, and experiential learning characteristics of the available mnemonics serious games?

2. Materials and Methods

2.1. Overview

Here, we discuss the methods we used in our systematic review, first by providing an overview of the order of our methods, then by describing each one in greater detail. To conduct the systematic review, we followed a 3-stage approach proposed by Tranfield et al. [36]. We first planned the review by including a preliminary search of the available apps. This stage, carried out by the senior researcher (KO), was aimed at identifying and refining the objectives of the review and developing a particular protocol which includes the search strings for retrieving apps from the App Store and Google Play, the inclusion criteria for including apps in the review, and the method of conducting the analysis of the included apps. Next, we conducted the review by having 1 researcher (OI) systematically search the App Store and Google Play using our search strings, then having 2 researchers (KF and OI) screen the retrieved apps by their name and description, and further screen those that passed by commencing a full-app review. KO oversaw both phases of the screening process. Finally, we tabulated, analyzed and reported the characteristics of the apps that remained. Then, 2 researchers (KF and OI) analyzed the games individually, and a 3rd researcher (SE) resolved disagreements.

2.2. Eligibility Criteria

We used the eligibility criteria shown in

Table 1. Eligibility criteria.

Inclusion Criteria	Exclusion Criteria
The app must have English as a language option.	The app does not have English as a language option.
The app must be a mnemonics serious game.	The app is not about mnemonics, or it is not a serious game.
The article can relate to any subject matter, provided the game is aimed at promoting memory and retention.	The article is not related to education or learning.

for the apps during our screening:

2.3. Information Sources and Search Strategy

To find relevant apps, we used the npm packages “google-play-scraper” [37] and “app-store-scraper” [38] to scrape apps from Google Play and the App Store, respectively. We used the search strings “mnemonic game”, “mnemonic games”, “mnemonics game”, “mnemonics games”, “acronym game”, “acronym games”, “acrostic”, “acrostics”, all one by one. We compiled the data into a single .csv file. Duplicates were dealt with by manually removing the second row and changing the first row’s “Platform” column value to “iOS, Android”. We then transferred the data into a Google Sheet. We had numerous columns, which is discussed later. As the data were now free of duplicates, each row had its featured app screened. Both sources were searched on 15 May 2024, and apps were not filtered by date.

2.4. Selection of Sources of Evidence

We used the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) flowchart [39] to screen the apps that we found from Google Play and the App Store. Two (2) researchers (KF and OI), guided and supervised by the senior researcher (KO), screened all unique apps independently by working on their own Google Sheet before putting their decisions together in another Google Sheet. The apps were first screened based on their name and description, and those that passed the screening were screened again based on a full-app review. To maximize inter-rater reliability, KO thoroughly explained the inclusion criteria to KF and OI beforehand, and kept their descriptions in a Google Sheet, so that there would be no misinterpretations. During both phases of screening, for each app that KF and OI disagreed on in terms of passing the inclusion criteria, the two discussed their rationale for their decision and sought to convince the other. Only when both agreed on a decision was the disagreement considered resolved. It was never necessary to ask KO to make the final decision; regardless, KO oversaw both phases of the screening process.

2.5. Data Charting Process and Items

To evaluate games effectively, Wilson et al. [40] recommended that the evaluation be carried out on the basis of four criteria, which they referred to as groundings. They include technical, empirical, external and theoretical groundings. Technical grounding entails considering the technical and operational feasibility constraints and adapting them accordingly as practically feasible. External grounding requires that the evaluation be conducted in real-world settings to increase its relevance. Empirical grounding entails using multiple evaluation methods (e.g., quantitative and qualitative) to increase generalizability and avoid basing conclusions on a single method. Theoretical grounding requires that the evaluation be informed by relevant established theories.

Given the authors’ primary objective of writing the paper (heuristic evaluation) and technical constraints, we chose one author-determined generic set of criteria, two established theoretical evaluation frameworks, and two experts to evaluate the apps retrieved from the app stores. The two experts included one user experience (UX) Master’s student (KF) and one third-year UX undergraduate student (OI), both of whom were guided and supervised by a human–computer interaction (HCI) professor and researcher (KO). The generic assessment criteria are shown in

Table 2. Assessed characteristics of apps from app stores.

Characteristic	Description	Values
App Name	The app’s name	Miscellaneous
URL	The URL used to access the app	Miscellaneous
Price	The price of the app (in Canadian dollars)	Miscellaneous
Age Group	The age group that the app targets (in years)	Miscellaneous
Score (/5)	The average number of stars the app has received, out of 5 (a score of 0 means no ratings)	Miscellaneous
Publication Year	What year the app was first published	Miscellaneous
Received Updates	Whether the game has received updates since first release	Yes, No
Platform	Game operating system, which corresponds to the app stores from which the game was retrieved	iOS, Android, Both
Subject	The subject matter the app is about	Language, Medicine, etc. (Various means several subjects take turns receiving focus, Generic means there is no clear subject)
Mnemonic Type	The type of mnemonics the app makes use of	Acrostic, Acronym, Image, Keyword, Note Organization
Game Type	The type of game that the app is	Puzzle, Quiz, Card, Cognitive (generalized approach)
Purpose	What purpose the app serves	Educational (formal and methodical teaching), Non-Educational (more casual)

. The theoretical frameworks include serious-game characteristics (

Table 3. Assessed serious game characteristics (IN-3PACIFIC).

Characteristic	Description
Immersion	Player’s feeling present in the game
Narrative	A sense of story progression
Presence (Self)	Experience of the player’s persona in the game as their actual self
Presence (Social)	Allowing players to interact with virtual actors in the game as though they were real
Presence (Spatial)	Players feeling physically embodied in the game’s environment
Agency/Control	Allowing players to control aspects of the game including tailoring it to their preferences
Identity	Allowing players to create characters or profiles representing them
Feedback	Information that players receive from the game relating to their performance, progress, rewards, what actions they can take and the consequences of their choices
Interactivity	Allowing players to initiate and receive feedback for actions in the game, as if the game is an entity that can be communicated with
Challenge	The various degrees of complexity that are possible in a game based on the player’s actions

) and experiential-learning characteristics (

Table 4. Assessed experiential learning characteristics (C-SHARP).

Category	Characteristic	Description
Critical Reflection	Investigation-Like	Allowing the player to act as an investigator who knows that there is room for improvement and refinement of solutions to the posed problem in the light of new knowledge
	Multisolution-Based	Providing room to solve problems in multiple ways due to their open-structured nature and context specificity.
Situated in Context	Place	The setting of the game including its location, community, culture, etc.
	Time	Depicting a time period (past, present or future) in the game setting
Hands-on Activity/Participation	Learner Empowerment	Support of players’ agency or control over aspects of the game including tailoring it to their preferences
	Collaboration With Others	Player being able to work with other players, e.g., through cooperation or competition
Risk/Novel Problems	Novel Experience	Throwing player into new and/or unfamiliar situations
	Challenge	The various degrees of complexity that are possible in the game based on the player’s actions
	Spontaneity/Unpredictability	Periodical introduction of unexpected elements that the player must quickly adapt to
Pragmatic/Purposeful	Inquiry-Based	Allowing and encouraging player to ask themselves questions or create their own problems to solve
	Co-Construction of Meaning	Allowing players to cooperate, build knowledge collaboratively, and learn from each other

). The three sets of criteria address the first, second, and third research questions, respectively. General app characteristics include app name, cost, publication year, subject the game is about, and purpose. The serious game characteristics, which foster motivation, engagement, and learning, are based on Blumberg et al.’s framework [41], which, for easy recall, we christened “IN-3PACIFIC” (Immersion, Narrative, 3 Presences, Agency or Control, Identity, Feedback, Interactivity, Challenge). While this list of interrelated game elements is not definitive or exhaustive, Blumberg et al. stated it reflects “how serious game designers, and game designers in general, have structured game development... and how researchers have examined aspects of game play that contribute to engagement and learning” (p. 337) [41]. Moreover, the experiential learning characteristics are based on Morris’s [42] revised Kolb’s [43] framework of experiential learning. Experiential learning is a “learning by doing” method of instruction, in which learners are involved, engaged, and actively participate in the learning process. Informed by Dewey’s [44] constructivist theory, it is a “hands-on” task-oriented learning process based on direct experience. Kolb’s [43] experiential learning cycle, comprising four stages, constitutes one of the most influential and cited theories of experiential learning. Morris [42] identified five broad themes that characterize experiential learning, which we christened “C-SHARP,” for easy recall (see Table 4). They include Critical Reflection, Situated in Context, Hands-on Activity and Participation, Risk/Novel Experience, and Pragmatic/Purposeful. We chose these two frameworks (IN-3PACIFIC and C-SHARP) for the evaluation of the mnemonics apps because they address the critical aspects of games and experiential learning, such as interactivity, feedback, challenge [41], user participation, and reflection [42]. KF and OI each noted in their own Google Sheet which apps fulfill which IN-3PACIFIC and C-SHARP characteristics, before putting their decisions together in another Google Sheet. To maximize inter-rater reliability, KO thoroughly explained each characteristic to the two beforehand and kept the descriptions in a Google Sheet to ensure there were no misinterpretations.

2.6. Synthesis Methods

We conducted three syntheses of results and made sure to include every app in each of them because all apps were evaluated with the same set of characteristics. First, because Subject, Mnemonic Type, Type of Game, and Purpose all have rather few possible values and are focused more on the gameplay details than the technical details, we saw fit to create a table showing which apps have which value for each of the four stated characteristics. Second, we created tables showing which IN-3PACIFIC and C-SHARP characteristics each app demonstrates, and for each such characteristic, we listed the percentage of apps that show it, along with the percentage of apps (and Cohen’s Kappa) where two researchers (KF and OI) had an immediate agreement on whether the app shows the characteristic. Third, we used the ggstatsplot package (version 0.12.2) from R (version 4.3.0) to conduct a Chi-square test to find relationships between any general, IN-3PACIFIC, and C-SHARP characteristics, and we used the ggplot2 package (version 3.4.4) to create graphs for each pairwise comparison. Because there are so many characteristics, we took the opportunity to group some of them if they share the same value for any given app. We also divided a characteristic’s values into groups if it has many possible values. Our groupings are described in further detail when we discuss the Chi-square test in the Section 3.

3. Results

In this section, we present the results of the systematic review, including the number of included apps and summary tables showing the main characteristics of the apps, which relate to the four main research questions. Results of our screening process are shown in Table 5 and Figure 1. Individual apps’ characteristics are shown in Table 6. Synthesis results are shown in Table 7, Table 8 and Table 9, as well as Figure 2, Figure 3 and Figure 4.

3.1. Selection of Sources of Evidence

The review followed the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) flowchart for the screening and inclusion of apps in the final analysis (Figure 1). A total of 257 apps were retrieved from the App Store and Google Play. A total of 183 were retrieved from the former, and 74 were retrieved from the latter. Note that these numbers are non-indicative of whether an app is in both; a given app could have been in both, but perhaps our search only found it in one. Removing 0.0545% (14/257) of them for being duplicates, we had 243 left. Next, we screened them by app name and description. As shown in Table 5, in the name/description screening phase, there were 12 disagreements between the 2 researchers (KF and OI), in which 7 were only accepted by the former and 5 by the latter. There were 231 agreements, in which 65 were accepted by both and 166 were rejected by both. This resulted in an agreement rate of around 95.1% (231/243) and a Cohen’s Kappa of around 0.881. We screened out around 71.6% (174/243) of them by app name and description, which meant we had 69 remaining. In the full-app review screening phase, there were three disagreements between two researchers (KF and OI), all of which involved only the former accepting. There were 66 agreements, in which 49 were accepted by both and 17 were rejected by both. This resulted in an agreement rate of around 95.7% (66/69) and a Cohen’s Kappa of around 0.889. We initially screened out around 29.0% (20/69) of them by full-app review, which meant we had 49 remaining. A double-check by KF and OI later found that two apps were unavailable and fifteen apps were not games. As such, around 34.7% (17/49) more apps were rejected, and we had 32 apps remaining for the final analysis.

Table 5. Inter-rater agreement between researchers during app screening.

	Name/Description Screening	Full-App Screening
Both Accepted	65	49
Both Rejected	166	17
Only KF Accepted	7	3
Only OI Accepted	5	0
% Agreement	95.1	95.7
Cohen’s Kappa	0.881	0.889

Table 5. Inter-rater agreement between researchers during app screening.

	Name/Description Screening	Full-App Screening
Both Accepted	65	49
Both Rejected	166	17
Only KF Accepted	7	3
Only OI Accepted	5	0
% Agreement	95.1	95.7
Cohen’s Kappa	0.881	0.889

Figure 1. PRISMA flowchart for the screening and inclusion of apps in the review.

Figure 1. PRISMA flowchart for the screening and inclusion of apps in the review.

3.2. Synthesis of Results

3.2.1. Overview

We present value distributions of Subject, Mnemonic Type, Type of Game, and Purpose in Table 7. We also present checklists of each app’s demonstration or lack thereof of the IN-3PACIFIC and C-SHARP characteristics in Table 8 and Table 9, respectively. Moreover, we present the results of a Chi-square test to find relationships between the aforementioned characteristics.

3.2.2. Games, Types, and Purposes

Table 7 shows the distributions for Subject, Mnemonic Type, and Type of Game. In particular, most (12/32, 37.5%) mnemonics serious games do not focus on any subject. Images are the most common (18/32, 56.3%) mnemonic device. Most (14/32, 43.8%) such games are quizzes, but cognitive games are not far behind (11/32, 34.4%). Such games are either educational (18/32, 56.3%) or they are neither educational nor professional (14/32, 43.8%).

3.3. Characteristics and Results of Individual Sources of Evidence

Each app’s App Name (and Serial Number created by us), Publication Year, Received Updates, Subject Area, Type of Mnemonic, Type of Game, Purpose, Price, Age Group, Score (/5), and Platform as described in Table 2 as shown in Table 6.

Table 6. Summary of included apps and findings. A: Android, AC: Acronym, ACS: Acrostic, AG: Age Group, ED: Educational, EV: Everyone, SCR: Score (/5), I: iOS, IM: Image, KW: Keyword, NO: Note Organization, NE: Non-Educational, PL: Platform, PU: Purpose, PY: Publication Year, RU: Received Updates SN: Serial Number, TM: Type of Mnemonics, N/A: Not Applicable.

SN	App Name	PY	RU	Subject	Game Type	TM	PU	Price	AG	SCR	PL
A1	ACRACU [45]	N/A	No	Various	Card	AC	ED	$0.00	4+	0	I
A2	Acromania [46]	2023	Yes	Generic	Puzzle	AC	NE	$0.00	12+	0	I
A3	Alpha-Guess [47]	2021	Yes	Various	Puzzle	AC	NE	$0.00	9+	0	I
A4	Anatnomics: Anatomy mnemonics [48]	2020	Yes	Anatomy	Quiz	AC	ED	$8.99	12+	0	I
A5	Dr. Moku’s Hiragana Mnemonics [49]	2010	Yes	Language	Quiz	IM	ED	$4.99	4+	4.11	I
A6	Dr. Moku’s Katakana Mnemonics [50]	2011	Yes	Language	Quiz	IM	ED	$4.99	4+	3.81	I
A7	Flashcards Maker & Mnemonic [51]	2023	Yes	Various	Quiz	NO	ED	$0.00	4+	4.73	I
A8	Get with the Script—Thai [52]	N/A	No	Language	Quiz	IM	ED	$0.00	4+	0	I
A9	Ginkgo Memory & Brain Training [53]	2019	Yes	Various	Card	IM	ED	$0.00	4+	4.52	I, A
A10	Hanzi Trainer [54]	2022	Yes	Language	Cognitive	IM	ED	$0.00	4+	3	I
A11	Kanji Trainer [55]	2023	Yes	Language	Cognitive	IM	ED	$0.00	4+	0	I
A12	Medical Mnemonic & Notes Maker [56]	N/A	No	Medicine	Quiz	NO	ED	$0.00	4+	4.91	I
A13	Medical Terminologies Quiz [57]	2018	Yes	Medicine	Quiz	NO	ED	$1.99	12+	0	I
A14	Memli (Mnemonic Dictionary) [58]	2023	Yes	Various	Quiz	NO	ED	$0.00	4+	3.57	I, A
A15	Memory Rx [59]	2022	Yes	Medicine	Quiz	AC	ED	$0.00	17+	1	I
A16	Mimir Mnemo [60]	2010	Yes	Generic	Cognitive	IM	NE	$1.99	4+	5	I
A17	mNemo—remember as a genius! [61]	2021	Yes	Generic	Cognitive	IM	NE	$0.00	4+	5	I
A18	Mnemo Major System Trainer [62]	2013	Yes	Generic	Cognitive	IM	NE	$12.99	4+	4.17	I
A19	Mnemonic Brain—Training [63]	2019	Yes	Generic	Cognitive	IM	NE	$0.00	4+	0	I
A20	Mnemonic Game [64]	N/A	No	Generic	Cognitive	IM	NE	$0.00	4+	0	I
A21	Mnemonic Words—1000 Words [65]	2021	Yes	Language	Quiz	IM	ED	$0.00	4+	5	I
A22	Mnemonica Major Memory System [66]	2022	Yes	Generic	Cognitive	IM	NE	$3.99	4+	4	I
A23	Mnemonics—easy memorizing [67]	2016	Yes	Generic	Cognitive	IM	NE	$0.00	EV	3.79	A
A24	Mnemonics Pro [68]	2024	Yes	Generic	Cognitive	IM	NE	$0.99	4+	0	I
A25	Mnemonics: memory development [69]	2022	Yes	Various	Quiz	IM	ED	$0.00	EV	0	A
A26	Mnemonics! [70]	2024	Yes	Generic	Cognitive	IM	NE	$0.00	4+	5	I
A27	Pandanese: Learn Chinese Hanzi [71]	2023	Yes	Language	Quiz	KW	ED	$0.00	4+	1.5	I
A28	Picmonic Nursing & NCLEX Study [72]	2022	Yes	Medicine	Quiz	IM	ED	$0.00	17+	4.88	I
A29	Pondernaut [73]	2015	Yes	Bible Verses	Quiz	KW	ED	$0.00	4+	5	I
A30	Qualuff Puzzles [74]	2022	Yes	Various	Puzzle	ACS	NE	$2.99	12+	0	I
A31	Rattlin’ Words [75]	2023	Yes	Generic	Puzzle	AC	NE	$0.00	12+	5	I
A32	Wacronyms [76]	2020	Yes	Generic	Puzzle	AC	NE	$0.00	EV	3.13	I, A

Table 6. Summary of included apps and findings. A: Android, AC: Acronym, ACS: Acrostic, AG: Age Group, ED: Educational, EV: Everyone, SCR: Score (/5), I: iOS, IM: Image, KW: Keyword, NO: Note Organization, NE: Non-Educational, PL: Platform, PU: Purpose, PY: Publication Year, RU: Received Updates SN: Serial Number, TM: Type of Mnemonics, N/A: Not Applicable.

SN	App Name	PY	RU	Subject	Game Type	TM	PU	Price	AG	SCR	PL
A1	ACRACU [45]	N/A	No	Various	Card	AC	ED	$0.00	4+	0	I
A2	Acromania [46]	2023	Yes	Generic	Puzzle	AC	NE	$0.00	12+	0	I
A3	Alpha-Guess [47]	2021	Yes	Various	Puzzle	AC	NE	$0.00	9+	0	I
A4	Anatnomics: Anatomy mnemonics [48]	2020	Yes	Anatomy	Quiz	AC	ED	$8.99	12+	0	I
A5	Dr. Moku’s Hiragana Mnemonics [49]	2010	Yes	Language	Quiz	IM	ED	$4.99	4+	4.11	I
A6	Dr. Moku’s Katakana Mnemonics [50]	2011	Yes	Language	Quiz	IM	ED	$4.99	4+	3.81	I
A7	Flashcards Maker & Mnemonic [51]	2023	Yes	Various	Quiz	NO	ED	$0.00	4+	4.73	I
A8	Get with the Script—Thai [52]	N/A	No	Language	Quiz	IM	ED	$0.00	4+	0	I
A9	Ginkgo Memory & Brain Training [53]	2019	Yes	Various	Card	IM	ED	$0.00	4+	4.52	I, A
A10	Hanzi Trainer [54]	2022	Yes	Language	Cognitive	IM	ED	$0.00	4+	3	I
A11	Kanji Trainer [55]	2023	Yes	Language	Cognitive	IM	ED	$0.00	4+	0	I
A12	Medical Mnemonic & Notes Maker [56]	N/A	No	Medicine	Quiz	NO	ED	$0.00	4+	4.91	I
A13	Medical Terminologies Quiz [57]	2018	Yes	Medicine	Quiz	NO	ED	$1.99	12+	0	I
A14	Memli (Mnemonic Dictionary) [58]	2023	Yes	Various	Quiz	NO	ED	$0.00	4+	3.57	I, A
A15	Memory Rx [59]	2022	Yes	Medicine	Quiz	AC	ED	$0.00	17+	1	I
A16	Mimir Mnemo [60]	2010	Yes	Generic	Cognitive	IM	NE	$1.99	4+	5	I
A17	mNemo—remember as a genius! [61]	2021	Yes	Generic	Cognitive	IM	NE	$0.00	4+	5	I
A18	Mnemo Major System Trainer [62]	2013	Yes	Generic	Cognitive	IM	NE	$12.99	4+	4.17	I
A19	Mnemonic Brain—Training [63]	2019	Yes	Generic	Cognitive	IM	NE	$0.00	4+	0	I
A20	Mnemonic Game [64]	N/A	No	Generic	Cognitive	IM	NE	$0.00	4+	0	I
A21	Mnemonic Words—1000 Words [65]	2021	Yes	Language	Quiz	IM	ED	$0.00	4+	5	I
A22	Mnemonica Major Memory System [66]	2022	Yes	Generic	Cognitive	IM	NE	$3.99	4+	4	I
A23	Mnemonics—easy memorizing [67]	2016	Yes	Generic	Cognitive	IM	NE	$0.00	EV	3.79	A
A24	Mnemonics Pro [68]	2024	Yes	Generic	Cognitive	IM	NE	$0.99	4+	0	I
A25	Mnemonics: memory development [69]	2022	Yes	Various	Quiz	IM	ED	$0.00	EV	0	A
A26	Mnemonics! [70]	2024	Yes	Generic	Cognitive	IM	NE	$0.00	4+	5	I
A27	Pandanese: Learn Chinese Hanzi [71]	2023	Yes	Language	Quiz	KW	ED	$0.00	4+	1.5	I
A28	Picmonic Nursing & NCLEX Study [72]	2022	Yes	Medicine	Quiz	IM	ED	$0.00	17+	4.88	I
A29	Pondernaut [73]	2015	Yes	Bible Verses	Quiz	KW	ED	$0.00	4+	5	I
A30	Qualuff Puzzles [74]	2022	Yes	Various	Puzzle	ACS	NE	$2.99	12+	0	I
A31	Rattlin’ Words [75]	2023	Yes	Generic	Puzzle	AC	NE	$0.00	12+	5	I
A32	Wacronyms [76]	2020	Yes	Generic	Puzzle	AC	NE	$0.00	EV	3.13	I, A

Table 7. Synthesis of Subject, Mnemonic Type, Type of Game, and Purpose.

Criterion	Value	Apps	Count (%)
Subject Area	Anatomy	A4	1 (3.13%)
	Bible Verses	A29	1 (3.13%)
	Language	A5, A6, A8, A10, A11, A21, A27	7 (21.9%)
	Medicine	A12, A13, A15, A28	4 (12.5%)
	Generic	A2, A16, A17, A18, A19, A20, A22, A23, A24, A26, A31, A32	12 (37.5%)
	Various	A1, A3, A7, A9, A14, A25, A30	7 (21.9%)
Mnemonic Type	Acronym	A1, A2, A3, A4, A15, A31, A32	7 (21.9%)
	Acrostic	A30	1 (3.13%)
	Image	A5, A6, A8, A9, A10, A11, A16, A17, A18, A19, A20, A21, A22, A23, A24, A25, A26, A28	18 (56.3%)
	Keyword	A27, A29	2 (6.25%)
	Note Organization	A7, A12, A13, A14	4 (12.5%)
Type of Game	Card	A1, A9	2 (6.25%)
	Cognitive	A10, A11, A16, A17, A18, A19, A20, A22, A23, A24, A26	11 (34.4%)
	Puzzle	A2, A3, A30, A31, A32	5 (15.6%)
	Quiz	A4, A5, A6, A7, A8, A12, A13, A14, A15, A21, A25, A27, A28, A29	14 (43.8%)
Purpose	Educational	A1, A4, A5, A6, A7, A8, A9, A10, A11, A12, A13, A14, A15, A21, A25, A27, A28, A29	18 (56.3%)
	Non-Educational	A2, A3, A16, A17, A18, A19, A20, A22, A23, A24, A26, A30, A31, A32	14 (43.8%)

Table 7. Synthesis of Subject, Mnemonic Type, Type of Game, and Purpose.

Criterion	Value	Apps	Count (%)
Subject Area	Anatomy	A4	1 (3.13%)
	Bible Verses	A29	1 (3.13%)
	Language	A5, A6, A8, A10, A11, A21, A27	7 (21.9%)
	Medicine	A12, A13, A15, A28	4 (12.5%)
	Generic	A2, A16, A17, A18, A19, A20, A22, A23, A24, A26, A31, A32	12 (37.5%)
	Various	A1, A3, A7, A9, A14, A25, A30	7 (21.9%)
Mnemonic Type	Acronym	A1, A2, A3, A4, A15, A31, A32	7 (21.9%)
	Acrostic	A30	1 (3.13%)
	Image	A5, A6, A8, A9, A10, A11, A16, A17, A18, A19, A20, A21, A22, A23, A24, A25, A26, A28	18 (56.3%)
	Keyword	A27, A29	2 (6.25%)
	Note Organization	A7, A12, A13, A14	4 (12.5%)
Type of Game	Card	A1, A9	2 (6.25%)
	Cognitive	A10, A11, A16, A17, A18, A19, A20, A22, A23, A24, A26	11 (34.4%)
	Puzzle	A2, A3, A30, A31, A32	5 (15.6%)
	Quiz	A4, A5, A6, A7, A8, A12, A13, A14, A15, A21, A25, A27, A28, A29	14 (43.8%)
Purpose	Educational	A1, A4, A5, A6, A7, A8, A9, A10, A11, A12, A13, A14, A15, A21, A25, A27, A28, A29	18 (56.3%)
	Non-Educational	A2, A3, A16, A17, A18, A19, A20, A22, A23, A24, A26, A30, A31, A32	14 (43.8%)

3.3.1. Serious Game Characteristics

Table 8 shows occurrences of the IN-3PACIFIC characteristics. In particular, all (32/32, 100%) the mnemonics serious games we found succeed with regards to feedback, interactivity, and challenge. They are slightly more likely to give a sense of agency or control (20/32, 62.5%) than not, and no (0/32, 0%) such game manages to invoke a sense of spatial presence.

Table 8. Summary of IN-3PACIFIC characteristics (Immersion, Narrative, Presence (Self), Presence (Social), Presence (Spatial), Agency or Control, Identity, Feedback, Interactivity, Challenge).

ID	I	N	P	P	P	A/C	I	F	I	C
A1								✓	✓	✓
A2	✓		✓	✓		✓	✓	✓	✓	✓
A3	✓		✓	✓		✓	✓	✓	✓	✓
A4						✓		✓	✓	✓
A5	✓	✓						✓	✓	✓
A6	✓	✓						✓	✓	✓
A7						✓		✓	✓	✓
A8								✓	✓	✓
A9								✓	✓	✓
A10		✓				✓		✓	✓	✓
A11		✓				✓		✓	✓	✓
A12						✓		✓	✓	✓
A13								✓	✓	✓
A14	✓		✓				✓	✓	✓	✓
A15						✓		✓	✓	✓
A16								✓	✓	✓
A17		✓				✓		✓	✓	✓
A18	✓		✓			✓	✓	✓	✓	✓
A19						✓		✓	✓	✓
A20								✓	✓	✓
A21	✓					✓		✓	✓	✓
A22						✓		✓	✓	✓
A23						✓		✓	✓	✓
A24	✓							✓	✓	✓
A25						✓		✓	✓	✓
A26								✓	✓	✓
A27		✓				✓		✓	✓	✓
A28	✓					✓		✓	✓	✓
A29								✓	✓	✓
A30	✓					✓		✓	✓	✓
A31	✓		✓	✓		✓	✓	✓	✓	✓
A32	✓		✓	✓		✓	✓	✓	✓	✓
Percent	37.5	18.8	18.8	12.5	0	62.5	18.8	100	100	100
Agreement (%)	90.7	78.1	75	100	100	78.1	78.1	100	100	100
Cohen’s Kappa	0.803	0	0.426	1	1	0.504	0.471	1	1	1

Table 8. Summary of IN-3PACIFIC characteristics (Immersion, Narrative, Presence (Self), Presence (Social), Presence (Spatial), Agency or Control, Identity, Feedback, Interactivity, Challenge).

ID	I	N	P	P	P	A/C	I	F	I	C
A1								✓	✓	✓
A2	✓		✓	✓		✓	✓	✓	✓	✓
A3	✓		✓	✓		✓	✓	✓	✓	✓
A4						✓		✓	✓	✓
A5	✓	✓						✓	✓	✓
A6	✓	✓						✓	✓	✓
A7						✓		✓	✓	✓
A8								✓	✓	✓
A9								✓	✓	✓
A10		✓				✓		✓	✓	✓
A11		✓				✓		✓	✓	✓
A12						✓		✓	✓	✓
A13								✓	✓	✓
A14	✓		✓				✓	✓	✓	✓
A15						✓		✓	✓	✓
A16								✓	✓	✓
A17		✓				✓		✓	✓	✓
A18	✓		✓			✓	✓	✓	✓	✓
A19						✓		✓	✓	✓
A20								✓	✓	✓
A21	✓					✓		✓	✓	✓
A22						✓		✓	✓	✓
A23						✓		✓	✓	✓
A24	✓							✓	✓	✓
A25						✓		✓	✓	✓
A26								✓	✓	✓
A27		✓				✓		✓	✓	✓
A28	✓					✓		✓	✓	✓
A29								✓	✓	✓
A30	✓					✓		✓	✓	✓
A31	✓		✓	✓		✓	✓	✓	✓	✓
A32	✓		✓	✓		✓	✓	✓	✓	✓
Percent	37.5	18.8	18.8	12.5	0	62.5	18.8	100	100	100
Agreement (%)	90.7	78.1	75	100	100	78.1	78.1	100	100	100
Cohen’s Kappa	0.803	0	0.426	1	1	0.504	0.471	1	1	1

3.3.2. Experiential Learning Characteristics

Table 9 shows the occurrences of the C-SHARP characteristics for each of the 32 assessed games and the inter-rater reliability metrics. Learner empowerment (aka agency) tends to be the most common experience learning feature employed in the games (20/32, 62.5%), followed by novel experience (18/32, 56.3%). However, almost none of the games were set at a particular place (1/32, 3.13%) and time period (0/32, 0%).

Table 9. Summary of C-SHARP characteristics. C: Critical Reflection, INV: Investigation-Like, MSB: Multisolution-Based; S: Situated in Context, PLA: Place, TIM: Time; HA: Hands-On Activity/Participation, LEM: Learner Empowerment, COL: Collaboration With Others; R: Risk/Novel problems, NOV: Novel Experience, CHL: Challenge, SP/UN: Spontaneity/Unpredictability; P: Pragmatic/Purposeful, INQ: Inquiry-Based, CCM: Co-Construction of Meaning.

	C		S		HA		R			P
ID	INV	MSB	PLA	TIM	LEM	COL	NOV	CHL	SP/UN	INQ	CCM
A1								✓
A2		✓			✓	✓	✓	✓	✓	✓	✓
A3	✓				✓	✓	✓	✓	✓		✓
A4					✓			✓
A5							✓	✓
A6							✓	✓
A7					✓			✓		✓
A8							✓	✓
A9							✓	✓
A10					✓		✓	✓
A11					✓		✓	✓
A12					✓			✓		✓
A13								✓		✓
A14								✓		✓
A15					✓			✓
A16
A17					✓		✓	✓
A18					✓		✓	✓
A19					✓		✓	✓	✓
A20								✓
A21					✓		✓	✓
A22					✓			✓
A23					✓			✓
A24								✓
A25					✓			✓	✓
A26								✓
A27					✓		✓	✓
A28					✓		✓	✓
A29			✓				✓	✓
A30	✓				✓		✓	✓		✓
A31		✓			✓	✓	✓	✓		✓	✓
A32		✓			✓	✓	✓	✓	✓	✓	✓
Percent	6.25	9.38	3.13	0	62.5	12.5	56.3	100	15.6	25	12.5
Agreement (%)	100	100	96.9	93.8	78.1	100	81.3	100	93.8	96.9	100
Cohen’s Kappa	1	1	0.652	0	0.504	1	0.631	1	0.796	0.92	1

Table 9. Summary of C-SHARP characteristics. C: Critical Reflection, INV: Investigation-Like, MSB: Multisolution-Based; S: Situated in Context, PLA: Place, TIM: Time; HA: Hands-On Activity/Participation, LEM: Learner Empowerment, COL: Collaboration With Others; R: Risk/Novel problems, NOV: Novel Experience, CHL: Challenge, SP/UN: Spontaneity/Unpredictability; P: Pragmatic/Purposeful, INQ: Inquiry-Based, CCM: Co-Construction of Meaning.

	C		S		HA		R			P
ID	INV	MSB	PLA	TIM	LEM	COL	NOV	CHL	SP/UN	INQ	CCM
A1								✓
A2		✓			✓	✓	✓	✓	✓	✓	✓
A3	✓				✓	✓	✓	✓	✓		✓
A4					✓			✓
A5							✓	✓
A6							✓	✓
A7					✓			✓		✓
A8							✓	✓
A9							✓	✓
A10					✓		✓	✓
A11					✓		✓	✓
A12					✓			✓		✓
A13								✓		✓
A14								✓		✓
A15					✓			✓
A16
A17					✓		✓	✓
A18					✓		✓	✓
A19					✓		✓	✓	✓
A20								✓
A21					✓		✓	✓
A22					✓			✓
A23					✓			✓
A24								✓
A25					✓			✓	✓
A26								✓
A27					✓		✓	✓
A28					✓		✓	✓
A29			✓				✓	✓
A30	✓				✓		✓	✓		✓
A31		✓			✓	✓	✓	✓		✓	✓
A32		✓			✓	✓	✓	✓	✓	✓	✓
Percent	6.25	9.38	3.13	0	62.5	12.5	56.3	100	15.6	25	12.5
Agreement (%)	100	100	96.9	93.8	78.1	100	81.3	100	93.8	96.9	100
Cohen’s Kappa	1	1	0.652	0	0.504	1	0.631	1	0.796	0.92	1

3.3.3. Relationship Between General, Serious Game, and Experiential Characteristics

We conducted a Chi-square test to uncover the relationships between the general characteristics shown in Table 7 and the domain-specific characteristics shown in Table 8 and Table 9. The Chi-square test also examined the relationships between the characteristics in each of the three categories of characteristics. Figure 2 and Figure 3 show the significant relationships between game characteristics in Table 7 and between them and demographic characteristics such as age group. Figure 4 shows the significant relationships between pairs of characteristics in Table 8 and Table 9. Note that we grouped Price values into Paid and Free; Subject values into Language and Others; Publication Year values into 2010–2014, 2015–2019, 2020–2024, and Unknown; Age Group values into Younger (Everyone or 4+) and Older (9+, 12+, or 17+); Score values into High (at least 4), Low (greater than 0 but less than 4), and No Rating (0 since the lowest rating is actually 1); and Mnemonic Type values into Acronym, Image, and Other. Because the three characteristics Collaboration With Others, Co-Construction of Meaning, and Social Presence all share the same Yes/No value for any given app, we aggregated them into one group simply called Collaboration to avoid repetition. For the same reason, we combined Self-Presence and Identity into Self-Presence/Identity. Moreover, we renamed Learner Empowerment as Empowerment for brevity.

Figure 2. Significant relationships beginning with Age Group, Purpose, Publication Year, or Subject.

Figure 2. Significant relationships beginning with Age Group, Purpose, Publication Year, or Subject.

Figure 3. Significant relationships beginning with Game Type or Mnemonic Type.

Figure 3. Significant relationships beginning with Game Type or Mnemonic Type.

Figure 4. Significant relationships beginning with Immersion, Self-Presence/Identity, Narrative, Collaboration, or Multisolution-Based.

Figure 4. Significant relationships beginning with Immersion, Self-Presence/Identity, Narrative, Collaboration, or Multisolution-Based.

4. Discussion

Based on the presented results, we discuss the key findings emanating from our analysis of the 32 included apps considering the four main research questions that underpin the systematic review.

4.1. Mnemonics Serious Games Available on the App Stores

Table 7 shows the key descriptive information about the 32 assessed games. Mnemonics serious games vary in terms of numerous characteristics. Regarding subject, besides the games which have a generic focus, a considerable number of them focus on using mnemonics to teach language exclusively, and the same goes for using them to teach medicine exclusively, albeit to a lesser extent. This is unsurprising, as these two subjects are very memory-intensive [5,12]. The games with generic focus (totalling more than one-third) tend to take a more rounded approach, either by teaching various subjects to players in turns or by not featuring any situations that have a clear subject. That said, anatomy [48] as well as biblical verses [73] were rarely touched on, as only one game is only concerned about each subject. Although only one game focuses on biblical verses in the current review, finding that mnemonics are also applied in learning religious content may not be surprising as mnemonics were used to memorize religious text in times of old when the printing press and/or modern technology such as computers and smartphones were not available and at our fingertips to serve as our extended memories [77,78].

Regarding Game Type, mnemonics serious games tend to be puzzles or quizzes, which are both common when it comes to brain training. Quite a few take a less specific approach to brain training; their methods cannot be easily classified (we listed these simply as “Cognitive”). They seldom appear as card games [45,53]. Regarding Mnemonic Type, the most common types are acronyms and images, which makes sense since they appear in many places. Note organization makes an occasional appearance, and both keywords and acrostics rarely appear. Regarding Purpose, it is almost split halfway between intending to formally educate individuals and serving a more casual, everyday role (which we listed as “Non-Educational”).

Regarding Price, a lot of mnemonics serious games tend to be free, but it is not uncommon for such a game to need to be bought. Non-zero prices we see range from slightly less than CAD 2 [57,60] to nearly CAD 13 [62]. Regarding Age Group, almost every game is intended for ages 4 and up. Sometimes the range is stricter, occasionally being for ages 12 and up, and sometimes even being for ages 17 and up. This could be because of the high difficulty of a subject or at least part of its content, though not necessarily. However, some games are meant for all ages, and one game, Alpha-Guess [47], has the rather unique range of ages 9 and up. Regarding Score, quite a few games do not have any ratings, but among the ones that do, they tend to have decent scores (i.e., at least 4) or mediocre ones (i.e., at least 3 and less than 4). Occasionally some games have a rather poor score [59,71]. Regarding Platforms, most of the mnemonics serious games we found were found on the App Store, while just a few were found on Google Play. However, as stated before, this does not imply that far fewer of these games are on Google Play, but rather that our specific search did not find them there.

4.2. Game Elements Employed in Mnemonics Serious Games on the App Stores

Here, we discuss the IN-3PACIFIC characteristics listed in Table 8. Mentions of subjectivity are based on agreement rates and Cohen’s Kappa.

Regarding Immersion, a fair amount of mnemonics serious games (37.5%) likely succeed in giving players a feeling of being present in the game in some way. However, more of these games likely fail to do so. Even opinions on whether a particular game is immersive can slightly differ. Regarding Narrative, only a few games (18.8%) possibly give off a sense of narrative, and the apparent existence of this is highly subjective. One reason is that games without a scenario involving defined characters can still possibly treat the player as one, specifically, the star of the show; whether the player feels like they are one is open to interpretation. For example, the game “mNemo—remember as a genius!” [61] features the player starting off presumably without knowledge of mnemonics and not doing so well at memorization, then learning mnemonics, using them in various tasks, and doing better at memorization. This could be interpreted as a first-person character on a quest to become stronger, though not everyone would think this way. Most games, however, clearly forgo any intention of a plot and simply throw tasks at the player.

Regarding Self Presence, not many mnemonics serious games (18.8%) seem to make players feel like their persona’s experiences are their own. This is usually trivially true because either players do not feel present in the game at all (i.e., no Immersion) or there seems to be no persona representing them (i.e., no Identity). However, Self Presence, like Narrative, is highly subjective; this time, it is because of the Identity characteristic, which is discussed later. Regarding Social Presence, fewer games (12.5%) involve interaction with any virtual actors, because there are no virtual actors. There is likely no subjectivity here, as it should be easy to discern whether there are virtual actors. They can be non-player characters (NPCs) or characters controlled by other players; either way, they should be recognizable. Regarding Spatial Presence, none of the games (0%) we found made players feel physically embodied in the game environment for two simple reasons: there are no opportunities to create physical avatars, and the only “environments” shown are 2D interfaces that players simply look at (which is not enough to count as physical embodiment). This should not be and is not subjective either.

Regarding Agency or Control, mnemonics serious games appear slightly more likely (62.5%) to give players sufficient ability to customize parts of the game to their liking, as opposed to not giving them this ability. Yet, this characteristic is highly subjective, as different people have different standards for what amounts of freedom are sufficient. For instance, some people might be satisfied with visual settings such as light/dark mode, while others might want gameplay settings such as difficulty adjustments. Even the degree of these settings can determine whether someone thinks they have sufficient freedom. Regarding Identity, not many of the games (18.8%) allow people to create a character or profile representing them. Surprisingly, this turns out to be more subjective than expected. One reason is that some of the games require creating an account; however, whether the account is personalized enough to count as a profile is open to interpretation. The game Memory Rx [59] takes this further. It has a tab called “Profile”, but the only personalization feature there is the choice of mnemonics. While some might consider this enough, others might not.

Regarding Feedback, all the mnemonics games we found (100%) did indeed provide sufficient feedback, which was unsurprising given the importance and indispensability of feedback in games in particular and digital applications in general. Regarding Interactivity, all of the games (100%) allowed players to initiate and receive feedback from them as if they were entities that can be communicated with. Even the simplest of the games still managed to make players feel like they are interacting with someone rather than using a mere tool or machine. This does not seem as subjective as we anticipated; perhaps people naturally see these sufficiently advanced technologies as those to “work with” rather than “use”. Regarding Challenge, all of the games (100%) did provide players different degrees of complexity. This was to be expected of games that were meant to teach skills to players, since such games often introduce skills early on but then require mastery of them later.

4.3. Fostering of Experiential Learning by Mnemonics Serious Games on the App Stores

Here, we discuss the C-SHARP characteristics listed in Table 9. Once again, mentions of subjectivity are based on agreement rates and Cohen’s Kappa.

Regarding the Investigation-Like category, almost none of the mnemonics serious games we found (6.25%) require players to use elements in the game’s context to solve problems. They tend to either guide players formally or leave them to their own devices but with obvious implications of what to do. One of the only two that featured this characteristic are Alpha-Guess [47], which requires players to guess acronyms based on the category, the number of words, and the set of available letters. The other is Qualuff Puzzles [74], which requires players to decode sentences encoded with numbers by using clues to find out which letter is associated with a particular number. Regarding the Multisolution-Based category, the only games (9.38%) that featured more than one possible solution are those where players decide the best solution among several. The three games are Acromania [46], Rattlin’ Words [75], and Wacronyms [76]; these three all involve multiple players coming up with the best phrase based on an ordered set of letters (e.g. Make a phrase with the acronym LOL) and voting which phrase is the best. The other games have a clear sense of correctness with only one correct answer. Regarding Critical Reflection overall, it appears not many mnemonics serious games demonstrate parts of it, and no game demonstrates it completely.

Regarding Place, the only mnemonics serious game we found (3.13%) that definitively featured a particular location is Pondernaut [73], which was set on the Moon. Although Alpha-Guess [47] featured a castle, this feature was not enough to establish that the game was predicated on a medieval setting. All other games only featured rather simple interfaces, which did not represent any specific location or landmark. Regarding Time, none of the games featured a particular time period. The games that came close to featuring a time period by virtue of having certain places or landmarks include Pondernaut [73], which featured an astronaut suit that is not specific enough, and Alpha-Guess [47], which featured a castle. All other games only featured rather simple interfaces that do not represent any time period.

We discussed Learner Empowerment, also known as Agency or Control, in the previous subsection. Regarding Collaboration With Others, not many games (12.5%) presented this opportunity. Alpha-Guess [47], for example, allows players to discuss with one another what a certain mnemonic in question might be, and the games that feature multiple solutions [46,75,76] involve presenting acronym ideas to one another. The remaining games are exclusively single-player. Overall, though the majority of the games succeed at fostering Learner Empowerment, most fail at supporting Hands-On Activity/Participation.

Regarding Novel Experience, a slight majority of games (56.3%) tend to foster it, and even so, this is a subjective matter. What counts as novel differs from person to person, based on their prior experiences or what is sufficient to count as novel in their eyes. We discussed Challenge in the previous subsection. Regarding Spontaneity/Unpredictability, not many games (15.6%) manage to keep players on their toes throughout their playthroughs, and the ones that do tend to rely on giving completely different tasks [63,69] or possibly unfamiliar terms [46,47,76]. Even so, this feeling is subjective for similar reasons to those for Novel Experience. We also felt that Rattlin’ Words [75] was not as unpredictable as Acromania [46] and Wacronyms [76] despite the same premise of coming up with acronyms given an ordered set of letters since we felt its sets were more commonly heard. Overall, though all the games fulfilled Challenge and a lot fulfilled Novel Experience, only a few fulfilled all three aspects of Risk/Novel Problems.

Regarding the Inquiry-Based parameter, only a select few games (25%) appear to demonstrate it. The multiple solutions games [46,75,76] require players to ask themselves what is the most outrageous yet coherent acronym they can form. A few other games [51,57,58] allow creating flashcards where the player must decide what are the best questions and answers to put on them. Qualuff Puzzles [74] requires the player to ask themselves what word best fits a definition and number of letters to use. Subjectivity exists, but it is kept at a minimum. Regarding Co-Construction of Meaning, all the games that involved Collaboration With Others (12.5%) [46,47,75,76] involved this characteristic, too, as the specific collaboration involves sharing ideas and thus learning from each other. No other game has demonstrated this characteristic. Overall, a rather low number of games achieve all aspects of Pragmatic/Purposeful.

4.4. Relationships Between the Characteristics of Mnemonics Serious Games on the App Stores

Here, we discuss possible explanations for the significant relationships we found between various characteristics.

Figure 2 shows the significant relationships where the first characteristic is either Age Group, Purpose, Publication Year, or Subject. Regarding Age Group, games meant for older players are more likely to use acronyms while those for younger players are more likely to use images; this is unsurprising, given that works catering to younger people often focus on images while those catering to older people usually feature more text. Games for older players have a higher chance of being puzzles while those for younger players are more likely to be generalized cognitive games, possibly out of concern that puzzles may be too confusing for them. Games for both age groups do include a decent amount of quizzes, however, as quizzes can be easily made for people of different ages. Games meant for older players are more likely to involve collaboration; perhaps working together may be seen as rather complicated for younger players. A similar case exists regarding whether games have an investigation-like nature, likely for similar reasons.

Regarding Purpose, games that are more casual and non-educational are more likely to feature Self Presence and Identity; perhaps the feeling of yourself being in another environment clashes with how traditional teaching methods or those closer to them mainly involve simply presenting screens of images and text to learners. Non-educational games are the only ones that possibly involve collaboration; perhaps educational games avoid doing so just in case players might mislead each other.

Regarding Publication Year, games in more recent years and games with an unknown initial release date are more likely to be free; the increasing abundance of and dependence on apps in general (partially due to COVID-19) may have led developers to avoid charging for their apps so potential customers would not be stolen away by competitors. Games with a known release date have all received updates while those without one apparently have not. If they actually did, then this was not stated; it is not so surprising given they did not state their release dates either. Only games in the early 2010s and early 2020s have a chance of featuring Immersion; it is unknown why those in the late 2010s and those without a known release date failed to do so. Games in more recent years are more likely to feature Learner Empowerment; perhaps more developers are recognizing the importance of letting players tailor their own experiences to maximize benefits from the game.

Regarding Subject, language games are practically guaranteed to be educational; teaching language likely requires a formal approach, after all. Games for other subjects might take a more casual approach. Language games are far more likely to feature narrative; telling a story does appear to be a good way to teach language, more so than teaching other subjects. Language games all give a novel experience; of course, learning language is easily perceived as novel. Teaching other subjects does not necessarily elicit this same perception.

Figure 3 shows the significant relationships where the first characteristic is either Game Type or Mnemonic Type. Regarding Game Type, quiz and card games are all educational, while puzzles and most cognitive games are non-educational. It is rather ironic how card games, mainly associated with fun, are educational here, while puzzle games, mainly taken seriously, are non-educational here. It turns out that the card games ACRACU [45] and Gingko Memory & Brain Training [53] use cards to ingrain certain mnemonics into a player’s head, similar to how traditional education often repeats concepts to ingrain them into learners’ heads. Meanwhile, the puzzle games here are more relaxed. Puzzle games are all immersive while most of the other games are not; of course puzzles are expected to require greater attention. Puzzle games are also the only ones that are possibly Multisolution-Based, though a possible explanation is that all three puzzle games that fulfill this [46,75,76] are the ones where players debate on who gave the best solution. These three games and Alpha-Guess [47], which are all puzzle games, are the only games to involve collaboration; they involve discussion with other players. Two puzzle games (Alpha-Guess [47] and Qualuff Puzzles [74] are the only games to have an investigation-like nature; their respective gameplay mechanics are discussed in the previous section. Puzzle games are also the most likely to feature Self-Presence/Identity, though this may be because of the Multisolution-Based games outright requiring profile creation and interaction with real people who also have profiles. Only puzzles and quizzes are possibly inquiry-based, which makes sense, given how puzzles and quizzes in general require careful thinking. Puzzles are the most likely to feature spontaneity/unpredictability, though this is likely because of the multisolution-based games again.

Regarding Mnemonic Type, only games with acronyms are possibly multisolution-based, but this is increasingly obvious because of the constantly aforementioned games that involved debating which proposed acronym is the best [46,75,76]. Similar cases exist for Self-Presence/Identity and collaboration, for the same reason along with Alpha-Guess [47] joining in. Only the three acronym debate games [46,75,76] and the note organization (flashcard) games [51,56,57,58] are possibly inquiry-based, for reasons both obvious and stated before. Games with acronyms tend to be puzzles or to a lesser degree quizzes, games with images tend to be cognitive or to a lesser degree quizzes, and games with other mnemonic types tend to be quizzes. The first two quickly make a lot of sense, but what about the third? Perhaps quizzes are so versatile and varied that other mnemonic types are better with them. Also, the note organization games [51,56,57,58] use flashcards to prepare players for quizzes.

Figure 4 shows the significant relationships where the first characteristic is either Immersion, Self-Presence/Identity, Narrative, Collaboration, or Multisolution-Based. Only games with Immersion can possibly feature Self-Presence/Identity, likely because Immersion involves presence in general while Self-Presence is a stricter version (and also relies on Identity). If Immersion is unfulfilled, logically Self-Presence will be as well. Only games with Immersion can possibly involve Collaboration, though once again, the ones doing so are the acronym debate games [46,75,76] and Alpha-Guess [47]. A similar case exists for Multisolution-Based games, but without Alpha-Guess [47]. Games without Immersion have a moderate chance of giving a Novel Experience, but games with Immersion have a higher chance of doing so; perhaps if a game is immersive, players pay more attention as opposed to dozing off and thus have a greater chance to feel like they are experiencing something new. Alternatively, a new experience itself can be immersive.

Regarding Self-Presence/Identity, only games that support this characteristic turn out to feature Collaboration. Particularly, only the acronym-based debate games [46,75,76] and Alpha-Guess [47] supported Self-Presence/Identity since, as stated numerous times before, they involved creating profiles and interacting with other players who have their own profiles. The acronym-based debate games [46,75,76] also demonstrate Multisolution-Based and Inquiry-Based features, but Alpha-Guess [47] demonstrated neither. Games with Self-Presence/Identity are also more likely to feature Spontaneity/Unpredictability, once again due to interactions with other players who might not think the same way. However, Rattlin’ Words [75] does not since, as stated before, its letter sets seem to be more commonly heard. Also, Mnemonic Brain—Training [63] and Mnemonics: memory development [69] do not feature Self-Presence/Identity, but still feature Spontaneity/Unpredictability because of how there were vastly different tasks in the same game.

Regarding Narrative, games without a Narrative are almost equally likely to feature Novel Experience or not, while games with a Narrative are certain to feature Novel Experience. It certainly helps that a new story can easily be perceived as novel, though the lack of one does not make a perception impossible.

Regarding Collaboration, only those that feature it have a chance of being Multisolution-Based; by now, it is extremely clear which ones do and why [46,75,76]. A similar case exists for the Inquiry-Based feature, except that the flashcard games [51,56,57,58] still manage to be Inquiry-Based without Collaboration because, as stated before, coming up with good flashcards requires asking oneself what to put on them. Another similar case exists for Spontaneity/Unpredictability, except instead of the flashcard games, the games with various different tasks [63,69] are the ones that feature it without Collaboration.

Regarding Multisolution-Based games, the ones that feature this (i.e., the acronym debate games [46,75,76]) are guaranteed to be Inquiry-Based, too, alongside the flashcard games [51,56,57,58], which do not. As for Spontaneity/Unpredictability, Acromania [46] and Wacronyms [76] feature it as well, but Rattlin’ Words [75] does not, for reasons stated before. The games with various different tasks [63,69] also feature it despite not being multisolution-based.

4.5. Justifications for Low Cohen’s Kappa Values

It should be noted that several characteristics’ Cohen’s Kappa values are below 0.7. Regarding IN-3PACIFIC, four of its characteristics have a Cohen’s Kappa below 0.7, namely Narrative (0), Self Presence (0.426), Agency or Control (0.504), and Identity (0.471). However, all of these can be attributed to their high subjectivity which we previously described. In addition, Narrative’s Cohen’s Kappa of 0 is a result of all agreements being in favor of rejection but at least one disagreement existing. Regarding C-SHARP, four of its characteristics have a Cohen’s Kappa below 0.7, namely Place (0.652), Time (0), Learner Empowerment (0.504), and Novel Experience (0.631). Learner Empowerment and Novel Experience were already stated to have high subjectivity, while Place and Time had a few disagreements that drastically lowered the Cohen’s Kappa. Place had all but one agreement be in favor of rejection, and it had only one disagreement, which was for Alpha-Guess [47], about whether the display of a castle was enough to count as a setting. All of Time’s agreements were in favor of rejection, and there were only two disagreements. One was for Alpha-Guess, about whether the display of a castle implied a medieval time period, and the other was for Pondernaut [73], about whether the display of astronauts implied a relatively modern time period. Despite these characteristics’ low Cohen’s Kappa values, the characteristics should still not be disregarded, especially with their decent to exceptional agreement percentages; Narrative, Agency or Control (also called Learner Empowerment), and Identity all have 78.1%, Self Presence has 75%, Place has 96.9%, Time has 93.8%, and Novel Experience has 81.3%.

4.6. Limitations and Future Work

There are a number of limitations to this review. First, our search is potentially over-constrained. Our inclusion criterion regarding only considering apps with English as a language option might have led us to miss out on relevant apps that do not have this option. Also, searching only the App Store and Google Play might have missed apps exclusive to other app stores, such as Microsoft Store. The search strings that we used may have missed relevant apps even in the stores that we searched; the fact that we found far more relevant apps in the App Store than in Google Play may suggest imperfections in our search strings. There really might be far fewer relevant apps in Google Play, but we think this is rather unlikely. Second, some of the apps that we screened out may have been relevant after all, but only ignored due to human error or subjective opinion. Third, there may have been new relevant apps introduced in between the time of the original search (15 May 2024) and the time that this paper was completed, or the apps we found may have been updated within this time frame and had their characteristics altered. Fourth, some of the apps’ characteristics that we described were possibly subjective, and even our claims of which ones were subjective and how much could have been subjective, too, given that we made them based on agreement rates and Cohen’s Kappa. The significant relationships we found involving these characteristics are also debatable, since we split some characteristics’ values into groups while not doing the same for some others. Doing this for more or fewer characteristics would likely alter the number and nature of significant relationships found. Even some of the significant relationships had small sample sizes; increasing sample sizes would make them more trustworthy. Future scoping/systematic reviews may seek to account for these four limitations, such as by including more opinions during screening and characteristic evaluation. Fifth, we did not analyze any of the review comments that players made about the games; in future work, we plan to analyze these reviews to gain more insight into the findings presented in this paper, which will enable us to provide useful design guidelines that can be beneficial to the development of mnemonics serious games. Sixth, despite addressing how Ejaz and Oyibo [35] did not discuss combining mnemonics serious games for retrieval practice with mnemonics creation tools for knowledge construction and retention, we did not discuss it either in this paper; future research efforts could focus on this subject matter.

5. Conclusions

We presented a synthesis of mnemonics serious games available in the app stores using Djaouti et al.’s [79] GPS model, IN-3PACIFIC, and C-SHARP as analytical game design frameworks. To the best of our knowledge, this systematic review is the first to analyze several mnemonics serious games and present the findings to foster the development of effective applications aimed to promote long-term learning. The review found that most of the evaluated games were launched between 2010 and 2024. The full 2010s decade recorded 9 applications in total, while 2020–2024 (only half a decade) recorded 19, slightly over double. This drastic increase highlights the increasing awareness and importance of mnemonics serious games, which necessitated this review that uncovered key findings. First, mnemonics serious games are most likely to teach Language or Medicine, take the form of puzzles or quizzes, feature acronyms and images, and have high ratings (at least 4 out of 5) or moderate ratings (at least 3 out of 5). Second, all the games we found supported Feedback, Interactivity, and Challenge, and a slight majority provided a sense of Agency and Control, yet many of them were lacking when it came to the remaining IN-3PACIFIC characteristics listed by Blumberg et al. [41]. Third, these games are almost equally likely to either provide a Novel Experience to players or not, yet they tend to fail to achieve most of the other C-SHARP characteristics laid out by Morris [42], aside from Challenge and Learner Empowerment. Fourth, many of the IN-3PACIFIC and C-SHARP characteristics have a significant relationship with each other. In conclusion, we recommend that efforts be made to create new mnemonics serious games that achieve more of these characteristics, which would likely lead to increased success in terms of realization of game purpose, effectiveness, and adoption. The fact that many of these characteristics are significantly related to each other suggests that fulfilling one characteristic led to fulfilling others. However, characteristics relating to a story in which a player participates (e.g., Immersion, Narrative, the Three Presences, Identity, Place, Time) should be given more attention rather than ignored as many of the evaluated apps did. Research has shown that including a story involving the player has positive effects on both a game’s memorability and users’ learning [80]. Moreover, characteristics relating to cooperation (e.g. Collaboration With Others, Co-Construction of Meaning) should be given attention as cooperation has been shown to improve academic achievement [81]. More importantly, improving existing games such as the ones evaluated in this review is essential as well, especially given that they are mobile apps that require updates from time to time.