Next Article in Journal
Ontology Fixing by Using Software Engineering Technology
Next Article in Special Issue
A ‘Speed—Difficulty—Accuracy’ Model Following a General Trajectory Motor Task with Spatial Constraints: An Information-Based Model
Previous Article in Journal
Special Issue on Fracture and Fatigue Assessments of Structural Components
Previous Article in Special Issue
Human Cognitive and Motor Abilities in the Aging Workforce: An Information-Based Model
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:

Combined Method for Evaluating Accessibility in Serious Games

Luis Salvador-Ullauri
Patricia Acosta-Vargas
Mario Gonzalez
2 and
Sergio Luján-Mora
Department of Software and Computing Systems, University of Alicante, 03690 Alicante, Spain
Intelligent and Interactive Systems Laboratory, Universidad de Las Américas, Quito 170125, Ecuador
Author to whom correspondence should be addressed.
Appl. Sci. 2020, 10(18), 6324;
Submission received: 5 August 2020 / Revised: 28 August 2020 / Accepted: 2 September 2020 / Published: 11 September 2020
(This article belongs to the Collection Human Factors in the Digital Society)


Nowadays, one of the learning resources in the educational area are serious games, also called training games; they are games designed with a different purpose than fun, whose main objective is to reinforce the new concepts more creatively. However, not all existing serious games are accessible in a way that allows access to a more significant number of users. Therefore, this research proposes to apply a combined method to evaluate accessibility in serious games, considering the Web Content Accessibility Guidelines (WCAG) 2.1. As a case study, we evaluated the accessibility of 82 serious games developed by Physical Education Technology Interactive Simulations at the University of Colorado. We propose to replicate this combined method for users with various types of disabilities, considering the various accessibility barriers. As future work, we suggest generating an accessibility heuristic evaluation focused on serious games, based on the accessibility issues identified. Finally, we believe it is essential to strengthen accessibility policies in each country, as well as implement best practices that generate innovation by incorporating diversity in building and designing more inclusive serious games.

1. Introduction

Belitski and Heron [1] argue that it is essential to complement formal education with teaching methods supported by group activities, simulations, and serious games. Serious games, including interactive simulations, are a powerful means of supporting college student learning [2]. Among the main benefits of simulations is to clarify concepts and improve understanding of the different topics taught in higher education institutions. Several authors agree that serious games improve students’ comprehension skills as they learn [3].
Aviation and militia-oriented, high-definition video games have been giving way to other types of games based on computers, consoles, and mobile devices, thanks to the reduction of technology costs. Education in the 21st century [4] requires the acquisition of new skills by teachers and students. Many of these skills are intimately related to new science, technology, engineering, and mathematics (STEM).
The prominent representatives of governments, businesses, and academia focus on how technology can help in the acquisition of these new skills. For example, Microsoft carries out events aimed at disseminating and promoting the use of video games in education. EduGameDay is an example of this type of event where professionals from the education sector and the video game industry demonstrate the advantage of using this resource as a pedagogical tool.
Another example of the initiative of the use of video games in education is GameOn. This video game is a text-based adventure, developed by IBM, built to help students explore microservice architectures and related concepts.
In this way, the acquisition of the skills necessary for education in the 21st century [4] can improve with the incorporation of technological tools in learning processes, including in the medical and health area; serious games could be used to facilitate these learning processes.
Of course, the use of serious games can be an alternative to current didactic tools [5], but these resources must be carefully designed so that their usefulness is valued. Some research affirms the indisputable benefits of including serious games in education, while other research points to the need for a more thorough evaluation of these types of resources before using them in the classroom [6].
Recent studies affirm that the use of video games in education has not been fully adopted because it is necessary to evaluate teachers’ understanding and acceptance and their effectiveness within different educational contexts. Teachers need to be convinced about the usefulness of using video games in class, and they need to understand how they can evaluate the knowledge acquired [7]. While some remain skeptical, most agree that serious games have great potential for learning [8,9].
Although the use of videogames has increased in the United States and several European countries, it is necessary to evaluate their effectiveness, which may be subject, as happens with other media, to an adequate didactic design as well as accessibility. Evaluating the accessibility of video games is a crucial factor in promoting the evaluation of their educational content. Among the main benefits of an accessible video game, we can enumerate the following: (1) allows the inclusion of all types of users; (2) improves access to learning content; (3) helps to achieve better learning outcomes; (4) allows the reuse of content on multiple devices; and (5) allows users with a permanent or temporary disability to receive and understand its educational content, as well as be able to use it correctly.
According to data from the World Health Organization (WHO), around a billion users worldwide suffer some form of physical or mental disability [10]. Therefore, accessibility in serious games is essential to provide a better interaction between users and video games. Without a doubt, the main reason for designing accessible serious games is to provide access to a more significant number of users, including people with some type of disability.
Therefore, this research proposes to apply a combined method to evaluate accessibility in serious games, taking into account the Web Content Accessibility Guidelines (WCAG) 2.1. In this research, we evaluated the accessibility of 82 serious games developed in HTML5 by Physical Education Technology (PhET) Interactive Simulations at the University of Colorado [11]. PhET offers fun, free, interactive simulations in Mathematics, Biology, Chemistry, Earth Science, and Physics [12] that are based on research. It has a total of 83 simulations developed in HTML5, 57 applications in Java via CheerpJ, 63 simulations developed in Java, and 12 applications in Flash. The applications can be executed online or downloaded to a computer. All simulations include the HTML5 source code and PhET’s Javascript, located on PhET’s GitHub page. The PhET project has several sponsors that make these resources free to all teachers and students.
In this study, we consider web accessibility as a starting point, which implies how users perceive, navigate, understand and interact on the web [13]; therefore, it is essential to keep in mind that the level of accessibility is the fundamental basis to facilitate access to serious games, especially for users with disabilities.
The manual method applied in this investigation comprises nine phases: (1) select serious games; (2) define the type of user; (3) define the test scenario; (4) explore each game to evaluate; (5) list the barriers based on Web Content Accessibility Guidelines (WCAG) 2.1; (6) evaluate with automatic tools and manually; (7) record evaluation data; (8) classify and analyze data; and (9) provide suggestions to improve accessibility. Besides, in the evaluation, the authors considered the WCAG 2.1 [14,15] based on five parameters: (1) accessible content; (2) visible focus; (3) accessible with a keyboard; (4) association of labels and controls; (5) controls for animation and audio. This research invites reflection and considers the importance of complying with and applying accessibility standards in the design of serious games considering diversity.
This research can serve as a guide for serious game designers and developers to apply WCAG 2.1 with an acceptable level of accessibility; additionally, this study can serve as a starting point for future work related to accessibility in serious games.
This research is structured as follows: Section 1 presents the introduction; Section 2 describes the background and previous work related to accessibility in serious games; Section 3 presents the method and the case study; Section 4 discusses the evidence and results; Section 5 presents conclusions and future work.

2. Background and Related Work

Currently, there are a large number of websites offering serious simulations and games. Statista [16] estimated that market revenue based on serious games worldwide is expected to grow from USD 3.5 billion in 2018 to 24 billion in 2024. Furthermore, in these times of social distancing [17], serious gaming has become highly supportive of teaching resources in the educational area. However, not all serious games are accessible. Accessibility refers to the condition if someone, regardless of their disability, can use serious games without barriers that prevent regular use and interaction with it.

2.1. Serious Games

The definition of serious games has been around before computing devices and entertainment. According to Schell [18], the definition indicates that games include goals, challenges, and rules to win or lose. Abt [19] indicates that “serious games” have an explicit and carefully thought-out educational purpose and are not intended to be played with fun only in mind. López et al. [20] argue that a serious game retains all the characteristics of a typical game, but point to a higher purpose than mere fun.
Jaramillo-Alcázar et al. [21] formulate that serious games allow the teaching of various types of concepts but that various games are not accessible since they do not focus on groups with disabilities. Salvador-Ullauri et al. [22] explain that it is a great challenge to implement serious games to support learning processes, especially for people with cognitive disabilities. In previous studies [22], the authors evaluated accessibility in ten serious games considering WCAG 2.0 [23]. The results revealed that serious games did not reach an adequate level of accessibility.

2.2. Accessibility

According to Park and Kim [24], accessibility in serious games makes it possible to guarantee that any user in any technical circumstance can access the content; for this reason, in our research, we applied WCAG 2.1 [14], the last official version of June 5, 2018, that provides recommendations to make content more accessible including people with disability. WCAG 2.1 consists of 4 principles, 13 guidelines, and 78 compliance criteria, and includes an undetermined number of sufficient techniques and advice. The four principles are the same as those presented in WCAG 2.0 [23]:
Principle 1: Perceivable—All users must be able to perceive the content in a visual, sound, or tactile way; Principle 2: Operable—Users must be able to use and navigate the interface components; Principle 3: Understandable—Both the content and the controls of the interface for its management must be understandable to the user; Principle 4: Robust—Content must be robust to be interpreted reliably by the most significant number of users with current and future technologies.
WCAG 2.1 [14] proposes success criteria associated with one of the following compliance levels: Level “A”: Minimum level of accessibility, when not reaching it, users cannot access the content of the web; Level “AA”: Intermediate level, implies that it is difficult for users to access the content; Level “AAA”: Maximum level, when the users can access the content without difficulty.
The authors found several accessibility studies in interactive simulations as part of the serious games contributing to this research. All these works were selected considering the interests of the scientific community to integrate people with disabilities into the academic world using new technologies.
Araújo et al. [25] argued that video games are increasingly popular but are not accessible, which represents a significant challenge for accessibility experts and game designers. The authors presented a study on the existing guidelines and recommendations for accessibility in video games, and they proposed 10 design recommendations for people with visual disabilities.
Cairns et al. [26] indicated that video games provide a cultural outlet where more players can be included and interact to do activities in a balanced way between different users. This event is possible if we create design environments that provide inclusive opportunities. The authors suggest including the guidelines with a language of accessibility of the game; they propose to (1) include a structure for the vocabulary of the game, (2) empower to meet the challenges of the game, (3) improve the player experience. Besides, they show how incorporating the guidelines in the development of video games provide accessible experiences to a more significant number of users.
Park and Kim [24] argued that the legislation of the accessibility guidelines could guarantee easy access to web content, considering users with disabilities but not in video games, since the contents of the web or mobile application consist of reasonably simple information in comparison with the contents of the video game. The content of the video game includes a more significant number of (1) characters, (2) players, (3) conflicts between them, and (4) updating a character when completing a mission. Therefore, they explain that it is necessary to analyze and classify the accessibility guidelines to evaluate each video game.
Waki et al. [27] stated that, currently, the lack of accessibility in digital games imposes barriers for people with disabilities. The authors propose a process to evaluate a set of integrated guidelines. The results revealed that the set of integrated guidelines allows determining the accessibility of digital games and refine the set of integrated guidelines.
Westin et al. [28] formulated that accessibility in video games consists of eliminating the possible barriers that prevent people with disabilities from accessing video games. The authors compared WCAG 2.1 with a set of accessibility guidelines for digital games. They compared 107 guidelines for accessible games.
Wilson and Crabb [29] indicated that video games, in particular, games on mobile devices, have evolved rapidly throughout the world. One problem identified in this topic is the accessibility that users face, especially if they have some kind of disability. In conclusion, the authors were able to determine the participants’ knowledge of accessibility guidelines and identify opinions on the importance of applying accessibility guidelines when creating accessible content for mobile games.
Spyridonis and Daylamani-Zad [30] argued the designers’ lack of commitment in designing serious games in the application of WCAG. They proposed to (1) focus on user-centered design, (2) identify the types of users, (3) apply WCAG to serious game mechanics, (4) measure user satisfaction, and (5) apply mixed methods. The results revealed that when applying the WCAG, the serious games presented an innovative and attractive solution.

3. Method and Case Study

In this research, we propose the application of a combined method to evaluate accessibility in serious games, considering the Web Content Accessibility Guidelines (WCAG) 2.1. As a case study, we evaluated the accessibility in 82 serious games developed by the PhET project [12]. The evaluation started on 19 January 2020 and ended on 20 June 2020. The study involved two expert evaluators in accessibility, who had experienced since 2015 and have contributed several articles in the area when there were discrepancies in the evaluation of the collaboration of a third expert was requested. In this study, the combined method includes the application of two automatic tools and manual evaluation. The evaluation method is summarized in the eight steps shown in Figure 1.
Phase 1: Select the serious games. In this phase, we selected the serious games of the PhET project. To select the sample size, we apply equation (1).
n = N Z 2 P Q ( N 1 ) E 2 + Z 2 P Q
where n is the sample size to be calculated, N is the size of the universe, in this case, 87 serious games, with a confidence level of 97%, where Z corresponds to 2.17, with an error margin E = 0.03. Applying the formula, we obtained the value of 82, which is the number of serious games, selected sequentially, and evaluated in this research by the two accessibility experts.
Table 1 contains the evaluated serious games, including the identification of each game and the name of the serious games within each URL (The complete URL is constructed by adding to the beginning of each address For example, the complete address for the first serious game is
Phase 2: Define the user’s type. Two experts in software application accessibility carried out the evaluation, in this case, on the serious games. This study defined the WCAG 2.1 for users with low vision [31]. According to the WHO [32], blindness and vision impairment affect at least 2.2 billion people around the world. Of those, 1 billion have a preventable vision impairment or one that has yet to be addressed. Reduced vision can have long-lasting effects on various aspects of life, and with increasing age, older people tend to decrease their presbyopia-related visual ability [33]. Based on these definitions, accessibility experts evaluated serious games by applying WCAG 2.1 to identify accessibility barriers. The experts have experience in the evaluation of mobile applications, web accessibility, and accessibility of educational resources since 2015 and have published several articles in high impact journals related to the topic.
Phase 3: Define the test scenario. In this phase, the authors define the scenarios to navigate and interact in serious games and reach the goal. The task is to (1) enter serious games, (2) review the functionality of each serious game, and (3) check if there are barriers that prevent accessibility for serious games. In this case study, an accessibility barrier for a person with low vision [34] means that the person cannot effectively move from one point to another within serious games due to visual acuity problems.
Phase 4: Explore each serious game to evaluate. In the fourth phase, the user explores and becomes familiar with the interaction mechanisms of serious games. In this phase, the evaluators identify (1) the functionalities of serious gaming; (2) if the content is adjustable with the zoom and the appearance of the game; and (3) the change of behavior according to the device, the context, and the applied configuration.
Figure 2 presents a screenshot of one of the evaluated games; in this example, it is a serious game used in Physics to explain the topic of the projectile launch.
Phase 5: List the barriers based on WCAG. This phase is essential, in which we list the barriers related to WCAG 2.1 for the evaluation of serious games. The barriers are listed in Table 2, which contains 29 guidelines considered in the manual evaluation of each serious game. Table 2 shows the guideline, the barrier, the WCAG 2.1 principle, the success criteria, and the level.
Phase 6: Evaluate with automatic tools and manually. In this phase, we evaluated the 82 serious games detailed in Table 1. We applied a combined evaluation using two automatic tools: (1) Colour Contrast Analyser (CCA), version 3.0.1 [35], a tool used to analyze some WCAG 2.1 to set the input colors of plain text. It also allows support for alpha transparency in foreground colors, includes a color blindness simulator. This tool allows a foreground and background color swatch to be taken to measure the contrast against which a report is output with WCAG 2.1. (2) Photosensitive Epilepsy Analysis Tool (PEAT), version 1.6 [36], allows evaluation in serious games or animations whether the content presents flickering or rapid transitions between light and dark background colors that can generate photosensitive seizures caused by certain types of flashing in serious gameplay, including mouse-overs that cause large areas of the screen to turn on and off quickly. The tool applies some of the WCAG 2.1 and 2.2, including restrictions related to frequency, luminance, area, and color of any flicker. Before analyzing the serious games with PEAT, the games were transformed into format Audio Video Interleave (.AVI) using a batch processing program to take a sample of the video.
Phase 7: Record evaluation data. In this phase, we present the registration of the data obtained from the evaluation of the accessibility of serious games with CCA and PEAT tools; the evaluation dataset [37] can be found at The codes assigned to each column, which correspond to the WCAG success criteria, where C1 = 1.4.3 Contrast (Minimum) AA, Regular Text, C2 = 1.4.3 Contrast (Minimum) AA, Large Text, C3 = 1.4.6 Contrast (Enhanced) AA, Regular Text, C4 = 1. 4.6 Contrast (Enhanced) AAA, Large Text, C5 = 1.4.11 Contrast without text AA (Button Reset), C6 = 1.4.11 Contrast without text AA (Others), P1 = State, P2 = Luminosity flash failures, P3 = Red flash failures, P4 = Extended flash alerts. The number one (1) indicates that the barrier has been exceeded, and the number zero (0) indicates that it has not been exceeded. Figure 3 presents the data recorded in the manual evaluation of the serious games with WCAG 2.1, contains the identifier assigned to each serious game with the 29 guidelines. We place the value of one (1) if the barrier is exceeded and zero (0) when the barrier was not exceeded. In Figure 3, the darkest color represents that the barrier was exceeded and the lightest color that the barrier was not exceeded.
Phase 8: Classify and analyze data. In this phase, the data of serious games were classified, considering the four principles of web accessibility proposed in WCAG 2.1. We grouped the data obtained with the automatic CCA and PEAT tools with which the contrast and photosensitivity that affects users with epilepsy were analyzed. The data obtained in the manual evaluation were grouped to identify the presence of each of the accessibility principles, the success criteria, and the level of accessibility, which will be detailed in the results analysis section. This process takes a long time and is therefore very demanding. Recorded data and analysis are available at Mendeley ( so that the assessment can be replicated [37].

4. Results and Discussion

In this phase, we analyze the results with automatic and manual evaluation tools. Applying descriptive statistics to the evaluation data with the CCA and PEAT tools, we have that the mean value is 3.9, the standard error is 0.21; the median is 5.0, the mode is 5.0, the standard deviation is 1.9, the variance of the sample is 3.64, the minimum value is 1.0, the maximum is 7.0. Figure 4 shows the evaluation with the CCA and PEAT tools; we found that 24 serious games meet from 1 to 2.5 in the evaluated contrast and photosensitivity parameters, 10 serious games meet from 2.5 to 4, the next 30 from 4 to 5.5, and 18 serious games meet of 5.5 to 7 points. From the data in Table 3, we find that the 82 serious games passed the photosensitivity test.
Table 3 shows a summary of the manual evaluation of 82 serious games; it contains the guideline, barrier, accessibility principle, success criteria, level, total, and percentage of accessibility evaluation in serious games with the manual method. The total column shows the value of the guidelines that exceed the barrier. We found that the 82 serious games compliance the following guidelines: (1) G05 related to easy to read font, (2) G16 related to well-spaced elements, (3) G25 related to language, (4) G26 related to consistent navigation.
Figure 5 shows the percentage of compliance with accessibility principles of the 82 serious games evaluated. We found that the perceptible principle is fulfilled in 54.4% of the total, the comprehensible one registers 27.4% of fulfillment, the operable one registers 18.2%, and the robust one registers 0.0%. We can conclude that in the evaluation of the accessibility of the 82 serious games, the most violated principle is the robust one.
Figure 6 shows the manual evaluation of the accessibility in 82 serious games; we obtained the following success criteria that overcome the barriers in the manual evaluation: (1) the success criteria representing 64% of the total are 1.1.1 for easy to read font, 1.4.12 for well-spaced elements, 3.1.1 for language, 3.2.3 for consistent navigation and 2.3.1 for luminance flash failures; (2) the success criteria representing 9.7% of the total is 1.4.3 for enhanced contrast; (3) the success criteria representing 5% of the total is 1.4.1 for the use of color; (4) the success criteria representing 8.4% of the total are 2.3.3 for animation from interactions, and 2.2.2 for pause, stop, hide; (5) the rest of the success criteria representing 12.8% of the total.
Figure 7 shows the evaluation of accessibility in serious games with the following results: (1) the guidelines G05, G16, G25, G02, G14, G21, G17, G01, G29, G27, G06, G10, G11, and G23 represent 66.6% of total compliance concerning Level A; (2) G26, G18, G04, G15, G19, G07, G08, G12, G13, and G24 guidelines represent 24.3% of total compliance for Level AA; (3) G03, G20, G22, G09, and G28 guidelines represent 9.1% of total compliance for Level AAA. For a serious game to have an acceptable level of accessibility, WCAG 2.1 suggests considering level AA [14].

5. Conclusions and Future Work

During the literature review, we found that there is no solid work focused on accessibility with WCAG 2.1. Therefore, this work proposes the application of a combined method of evaluating accessibility in serious games and establishes 29 guidelines based on WCAG. From the case study, we can conclude that the most neglected accessibility principles are operable and robust; the operable principle refers to how people interact and have control during interaction with serious games. On the other hand, “robust” barriers related to related ones can be addressed by including assistive technology that allows better compatibility with current and future tools.
This combined approach can be replicated for other serious games that include users with different types of disabilities. To improve accessibility in serious gaming, we propose to include (1) automatic transcriptions or a bot to transcribe the audio or video to text without human intervention, so that the user can review the transcriptions while playing; (2) sign language to establish a communication channel with the social environment, this option is useful for users with hearing disabilities; (3) photosensitivity control, to configure the excess of light and brightness useful in users with epilepsy problems; (4) external devices to allow virtual and augmented reality, for example, with the oculus quest the user can achieve greater immersion in the learning process; and (5) a contextual help option so that users can operate without losing sight of what they are doing. The authors can conclude that no serious games among those selected have reached an acceptable level of accessibility. Therefore, serious game developers should make significant efforts to improve accessibility.
This study has its limitations because it is a combined method that includes a manual part where accessibility experts intervene; the evaluation results depend on the experience and ability to evaluate serious games. From the theoretical point of view, we present a new method to evaluate accessibility in serious games based on a combined method. On the other hand, in practical terms, this study tries to wake up and motivate serious game developers to apply WCAG 2.1 guidelines to make serious games more accessible and inclusive that consider diversity. Additionally, this method can serve as a reference for future studies related to accessibility in simulations and serious games.
As future work, we suggest (1) testing this method with users with different types of disabilities and applying the corresponding disability-related guidelines; (2) generating a heuristic accessibility evaluation focused on serious games, based on the accessibility issues identified in WCAG 2.1; (3) expanding the serious games database with some applications other than serious games to deepen the analysis and evaluation, and (4) developing a software application that includes the WCAG 2.1 guidelines to assist in the evaluation of serious games.

Author Contributions

Conceptualization, L.S.-U. and P.A.-V.; methodology, L.S.-U.; investigation, L.S.-U.; M.G. and P.A.-V.; writing—original draft preparation, L.S.-U. and P.A.-V.; writing—review and editing L.S.-U., S.L.-M., M.G. and P.A.-V.; supervision, S.L.-M.; project administration, P.A.-V. and L.S.-U. All authors have read and agreed to the published version of the manuscript.


This research was funded by Universidad de Las Américas-Ecuador, as part of an internal research project FGE.PAV.19.11.

Conflicts of Interest

The authors declare no conflict of interest.


  1. Belitski, M.; Heron, K. Expanding entrepreneurship education ecosystems. J. Manag. Dev. 2017, 36, 163–177. [Google Scholar] [CrossRef]
  2. Martin, F.; Betrus, A.K. Instructional simulations and games. In Digital Media for Learning; Springer: Cham, Germany, 2019; pp. 85–110. [Google Scholar]
  3. Cheng, M.T.; Chen, J.H.; Chu, S.J.; Chen, S.Y. The use of serious games in science education: A review of selected empirical research from 2002 to 2013. J. Comput. Educ. 2015, 2, 353–375. [Google Scholar] [CrossRef] [Green Version]
  4. He, W.; Xu, G.; Kruck, S.E. Online IS education for the 21st century. J. Inf Syst. Educ. 2019, 25, 1. [Google Scholar]
  5. Kazimoglu, C.; Kiernan, M.; Bacon, L.; Mackinnon, L. A serious game for developing computational thinking and learning introductory computer programming. Proc. Soc. Behav. Sci. 2012, 47, 1991–1999. [Google Scholar] [CrossRef] [Green Version]
  6. De Freitas, S.; Liarokapis, F. Serious games: A new paradigm for education? In Serious Games and Edutainment Applications; Springer: London, UK, 2011; pp. 9–23. [Google Scholar]
  7. Assaf, M.; van Hillegersberg, J.; Spil, T.; Arikat, N. Teachers’ perceptions about using serious games in formal education in Jordan: Possibilities and limitations. In Proceedings of the Engineering Education Conference, IEEE, Dubai, United Arab Emirates, 8–11 April 2019; pp. 436–441. [Google Scholar]
  8. Egenfeldt-Nielsen, S.; Heide Smith, J.; Pajares Tosca, S. Serious Games and gamification—When entertainment is not enough. In Understanding Video Games; Routledge: Abingdon, UK, 2015; pp. 247–280. [Google Scholar]
  9. Larson, K. Serious games and gamification in the corporate training environment: A literature review. Tech. Trends 2020, 64, 319–328. [Google Scholar] [CrossRef]
  10. World Health Organization (WHO). World Report on Disability. Available online: (accessed on 10 May 2020).
  11. Meadows, M.L.; Caniglia, J.C. Using PhET simulations in the mathematics classroom. Math Teach 2019, 112, 386–389. [Google Scholar] [CrossRef]
  12. Interactive Simulations. PhET: Free Online Physics, Chemistry, Biology, Earth Science and Math Simulations. Available online: (accessed on 25 April 2020).
  13. Acosta-Vargas, P.; Salvador-Ullauri, L.; Luján-Mora, S. A heuristic method to evaluate web accessibility for users with low vision. IEEE Access 2019, 7, 125634–125648. [Google Scholar] [CrossRef]
  14. World Wide Web Consortium. Web Content Accessibility Guidelines (WCAG) 2.1. Available online: (accessed on 7 July 2020).
  15. World Wide Web Consortium (W3C). Web Content Accessibility Guidelines (WCAG) 2.2. Available online: (accessed on 10 May 2020).
  16. Statista. Game-based Learning Market Revenue Worldwide in 2018 and 2024. Available online: (accessed on 10 May 2020).
  17. Ainslie, K.E.; Walters, C.E.; Fu, H.; Bhatia, S.; Wang, H.; Xi, X.; Cattarino, L. Evidence of initial success for China exiting COVID-19 social distancing policy after achieving containment. Wellcome Open Res. 2020, 5, 81. [Google Scholar] [CrossRef] [PubMed]
  18. Schell, J. The Art of Game Design: A Book of Lenses; CRC Press: Boka Raton, FL, USA, 2019. [Google Scholar]
  19. Abt, C. Serious Games; University Press of America: Millburn, NJ, USA, 1987; p. 196. [Google Scholar]
  20. López, J.M.; Medina, N.M.; de Lope, R.P. Interaction in video games for people with impaired visual function: Improving accessibility. In Proceedings of the International Conference on Human Computer Interaction, ACM, Salamanca, Spain, 13–16 September 2016; pp. 1–2. [Google Scholar]
  21. Jaramillo-Alcázar, A.; Luján-Mora, S.; Salvador-Ullauri, L. Accessibility assessment of mobile serious games for people with cognitive impairments. In Proceedings of the International Conference on Information Systems and Computer Science, IEEE, Quito, Ecuador, 23–25 November 2017; pp. 323–328. [Google Scholar]
  22. Salvador-Ullauri, L.; Acosta-Vargas, P.; Luján-Mora, S. Accessibility Evaluation of Video Games for Users with Cognitive Disabilities; Springer: Cham, Germany, 2020; pp. 853–859. [Google Scholar]
  23. World Wide Web Consortium (W3C). Web Content Accessibility Guidelines (WCAG) 2.0. Available online: (accessed on 3 August 2020).
  24. Park, H.J.; Kim, S.B. Guidelines of serious game accessibility for the disabled. In Proceedings of the International Conference on Information Science and Applications, IEEE, Suwon, South Korea, 24–26 June 2013; pp. 1–3. [Google Scholar]
  25. Araújo, M.C.; Façanha, A.R.; Darin, T.G.; Sánchez, J.; Andrade, R.M.; Viana, W. Mobile Audio Games Accessibility Evaluation for Users Who are Blind; Springer: Cham, Germany, 2017; pp. 242–259. [Google Scholar]
  26. Cairns, P.; Power, C.; Barlet, M.; Haynes, G. Future design of accessibility in games: A design vocabulary. Int. J. Hum. Comput. Stud. 2019, 131, 64–71. [Google Scholar] [CrossRef]
  27. Waki, A.L.; Fujiyoshi, G.S.; Almeida, L.D. Games Accessibility for Deaf People: Evaluating Integrated Guidelines; Springer: Cham, Germany, 2015; pp. 493–504. [Google Scholar]
  28. Westin, T.; Ku, J.J.; Dupire, J.; Hamilton, I. Game Accessibility Guidelines and WCAG 2.0–A Gap Analysis; Springer: Cham, Germany, 2018; pp. 270–279. [Google Scholar]
  29. Wilson, A.; Crabb, M. W3C Accessibility guidelines for mobile games. Comput. Games J. 2018, 7, 49–61. [Google Scholar]
  30. Spyridonis, F.; Daylamani-Zad, D. A serious game to improve engagement with web accessibility guidelines. Behav. Inf. Technol. 2020, 1–19. [Google Scholar] [CrossRef]
  31. Moreno, L.; Valencia, X.; Pérez, J.; Arrue, M. Exploring the Web navigation strategies of people with low vision. In Proceedings of the International Conference on Human Computer Interaction, ACM, Palma, Spain, 12–14 September 2018; p. 13. [Google Scholar]
  32. World Health Organization (WHO). Blindness and Vision Impairment. Available online: (accessed on 3 September 2020).
  33. Bourne, R.R.; Flaxman, S.R.; Braithwaite, T.; Cicinelli, M.V.; das, A.; Jonas, J.B.; Naidoo, K. Magnitude, temporal trends, and projections of the global prevalence of blindness and distance and near vision impairment: A systematic review and meta-analysis. Lancet. Glob. Heal. 2019, 5, e888–e897. Available online: (accessed on 26 January 2019). [CrossRef] [Green Version]
  34. Brajnik, G. Beyond Conformance: The Role of Accessibility Evaluation Methods; Springer: Berlin, Germany, 2008; pp. 63–80. [Google Scholar]
  35. The Paciello Group. Colour Contrast Analyser. Available online: (accessed on 20 June 2020).
  36. University of Maryland. Photosensitive Epilepsy Analysis Tool. Available online: (accessed on 20 June 2020).
  37. Acosta-Vargas, P.; Salvador-Ullauri, L. Dataset Evaluation in Serious Games. Mendeley Data. Available online: (accessed on 3 September 2020).
Figure 1. Diagram of accessibility evaluation of serious games.
Figure 1. Diagram of accessibility evaluation of serious games.
Applsci 10 06324 g001
Figure 2. Screenshot Projectile Motion—PHET.
Figure 2. Screenshot Projectile Motion—PHET.
Applsci 10 06324 g002
Figure 3. Manual evaluation of serious games.
Figure 3. Manual evaluation of serious games.
Applsci 10 06324 g003
Figure 4. Evaluation of serious games with CCA and PEAT.
Figure 4. Evaluation of serious games with CCA and PEAT.
Applsci 10 06324 g004
Figure 5. Successes of the WCAG principles in manual evaluation.
Figure 5. Successes of the WCAG principles in manual evaluation.
Applsci 10 06324 g005
Figure 6. Success criteria in manual evaluation that exceed the barriers.
Figure 6. Success criteria in manual evaluation that exceed the barriers.
Applsci 10 06324 g006
Figure 7. WCAG levels in manual evaluation that exceed the barriers.
Figure 7. WCAG levels in manual evaluation that exceed the barriers.
Applsci 10 06324 g007
Table 1. Serious games were selected for evaluation.
Table 1. Serious games were selected for evaluation.
11beeballoons-and-static-electricity/1.4.14/balloons-and-static-electricity_es.htmlEarth science
14buiblackbody-spectrum/1.0.7/blackbody-spectrum_es.htmlEarth science
25enediffusion/1.0.4/diffusion_es.htmlEarth science
41gasgases-intro/1.0.5/gases-intro_es.htmlEarth science
42gengas-properties/1.0.4/gas-properties_es.htmlEarth science
58molmolecules-and-light/1.4.14/molecules-and-light_es.htmlEarth science
64phph-scale/1.3.4/ph-scale_es.htmlEarth science
75undunder-pressure/1.1.18/under-pressure_es.htmlEarth science
79wavwave-interference/2.0.2/wave-interference_es.htmlEarth science
80wavwaves-intro/1.0.2/waves-intro_es.htmlEarth science
81wavwave-on-a-string/1.1.22/wave-on-a-string_es.htmlEarth science
Table 2. Guidelines for evaluating accessibility in serious games.
Table 2. Guidelines for evaluating accessibility in serious games.
GuidelineBarrierWCAG 2.1Success CriteriaLevel
G01Accessible keyboardOperable2.1.1 A
G02Luminance flash failuresOperable2.3.1 A
G03Animation from InteractionsOperable2.3.3 AAA
G04Content hovering over focusPerceivable1.4.13 AA
G05Easy to read fontPerceivable1.1.1A
G06Text alternativesPerceivable1.1.1A
G08Automatic transcriptsPerceivable1.2.5AA
G09Sign languagePerceivable1.2.6 AAA
G10Information and relationshipsPerceivable1.3.1A
G11Sensory characteristicsPerceivable1.3.1A
G12Adjust display settingsPerceivable1.3.4AA
G13Interface rearrangementPerceivable1.3.5 AA
G14Use of colorPerceivable1.4.1A
G15Contrast without textPerceivable1.4.11 AA
G16Well-spaced elementsPerceivable1.4.12A
G17Good audio techniquesPerceivable1.4.2A
G18Contrast (Minimum)Perceivable1.4.3AA
G19Images as sharp as possiblePerceivable1.4.5AA
G20Visual presentationPerceivable1.4.8 AAA
G21Pause, stop, hidePerceivable2.2.2A
G22Contrast (Enhanced)Perceivable1.4.6AAA
G23Screen reader supportRobust4.1.2A
G24Status messagesRobust4.1.3 AA
G25LanguageUnderstandable3.1.1 A
G26Consistent navigationUnderstandable3.2.3 AA
G27Labels or instructionsUnderstandable3.3.2 A
G29On FocusUnderstandable3.2.1A
Table 3. Summary of Accessibility Evaluation in Serious Games with the Manual Method.
Table 3. Summary of Accessibility Evaluation in Serious Games with the Manual Method.
GuidelineBarrierWCAG 2.1 (Principle)Success CriteriaLevelTotal%
G01Accessible keyboardOperable2.1.1 A81.3
G02Luminance flash failuresOperable2.3.1 A8113
G03Animation from InteractionsOperable2.3.3 AAA274.2
G04Content hovering over focusPerceivable1.4.13 AA60.9
G05Easy to read fontPerceivable1.1.1A8213
G06Text alternativesPerceivable1.1.1A10.2
G08Automatic transcriptsPerceivable1.2.5AA00
G09Sign languagePerceivable1.2.6 AAA00
G10Information and relationshipsPerceivable1.3.1A00
G11Sensory characteristicsPerceivable1.3.1A00
G12Adjust display settingsPerceivable1.3.4AA00
G13Interface rearrangementPerceivable1.3.5 AA00
G14Use of colorPerceivable1.4.1A325
G15Contrast without textPerceivable1.4.11 AA20.3
G16Well-spaced elementsPerceivable1.4.12A8213
G17Good audio techniquesPerceivable1.4.2A203.1
G18Contrast (Minimum)Perceivable1.4.3AA629.7
G19Images as sharp as possiblePerceivable1.4.5AA20.3
G20Visual presentationPerceivable1.4.8 AAA121.9
G21Pause, stop, hidePerceivable2.2.2A274.2
G22Contrast (Enhanced)Perceivable1.4.6AAA203.1
G23Screen reader supportRobust4.1.2A00
G24Status messagesRobust4.1.3 AA00
G25LanguageUnderstandable3.1.1 A8213
G26Consistent navigationUnderstandable3.2.3 AA8213
G27Labels or instructionsUnderstandable3.3.2 A50.8
G29On FocusUnderstandable3.2.1A60.9

Share and Cite

MDPI and ACS Style

Salvador-Ullauri, L.; Acosta-Vargas, P.; Gonzalez, M.; Luján-Mora, S. Combined Method for Evaluating Accessibility in Serious Games. Appl. Sci. 2020, 10, 6324.

AMA Style

Salvador-Ullauri L, Acosta-Vargas P, Gonzalez M, Luján-Mora S. Combined Method for Evaluating Accessibility in Serious Games. Applied Sciences. 2020; 10(18):6324.

Chicago/Turabian Style

Salvador-Ullauri, Luis, Patricia Acosta-Vargas, Mario Gonzalez, and Sergio Luján-Mora. 2020. "Combined Method for Evaluating Accessibility in Serious Games" Applied Sciences 10, no. 18: 6324.

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop