Combined Method for Evaluating Accessibility in Serious Games

Abstract

Nowadays, one of the learning resources in the educational area are serious games, also called training games; they are games designed with a different purpose than fun, whose main objective is to reinforce the new concepts more creatively. However, not all existing serious games are accessible in a way that allows access to a more significant number of users. Therefore, this research proposes to apply a combined method to evaluate accessibility in serious games, considering the Web Content Accessibility Guidelines (WCAG) 2.1. As a case study, we evaluated the accessibility of 82 serious games developed by Physical Education Technology Interactive Simulations at the University of Colorado. We propose to replicate this combined method for users with various types of disabilities, considering the various accessibility barriers. As future work, we suggest generating an accessibility heuristic evaluation focused on serious games, based on the accessibility issues identified. Finally, we believe it is essential to strengthen accessibility policies in each country, as well as implement best practices that generate innovation by incorporating diversity in building and designing more inclusive serious games.

1. Introduction

Belitski and Heron [1] argue that it is essential to complement formal education with teaching methods supported by group activities, simulations, and serious games. Serious games, including interactive simulations, are a powerful means of supporting college student learning [2]. Among the main benefits of simulations is to clarify concepts and improve understanding of the different topics taught in higher education institutions. Several authors agree that serious games improve students’ comprehension skills as they learn [3].

Aviation and militia-oriented, high-definition video games have been giving way to other types of games based on computers, consoles, and mobile devices, thanks to the reduction of technology costs. Education in the 21st century [4] requires the acquisition of new skills by teachers and students. Many of these skills are intimately related to new science, technology, engineering, and mathematics (STEM).

The prominent representatives of governments, businesses, and academia focus on how technology can help in the acquisition of these new skills. For example, Microsoft carries out events aimed at disseminating and promoting the use of video games in education. EduGameDay is an example of this type of event where professionals from the education sector and the video game industry demonstrate the advantage of using this resource as a pedagogical tool.

Another example of the initiative of the use of video games in education is GameOn. This video game is a text-based adventure, developed by IBM, built to help students explore microservice architectures and related concepts.

In this way, the acquisition of the skills necessary for education in the 21st century [4] can improve with the incorporation of technological tools in learning processes, including in the medical and health area; serious games could be used to facilitate these learning processes.

Of course, the use of serious games can be an alternative to current didactic tools [5], but these resources must be carefully designed so that their usefulness is valued. Some research affirms the indisputable benefits of including serious games in education, while other research points to the need for a more thorough evaluation of these types of resources before using them in the classroom [6].

Recent studies affirm that the use of video games in education has not been fully adopted because it is necessary to evaluate teachers’ understanding and acceptance and their effectiveness within different educational contexts. Teachers need to be convinced about the usefulness of using video games in class, and they need to understand how they can evaluate the knowledge acquired [7]. While some remain skeptical, most agree that serious games have great potential for learning [8,9].

Although the use of videogames has increased in the United States and several European countries, it is necessary to evaluate their effectiveness, which may be subject, as happens with other media, to an adequate didactic design as well as accessibility. Evaluating the accessibility of video games is a crucial factor in promoting the evaluation of their educational content. Among the main benefits of an accessible video game, we can enumerate the following: (1) allows the inclusion of all types of users; (2) improves access to learning content; (3) helps to achieve better learning outcomes; (4) allows the reuse of content on multiple devices; and (5) allows users with a permanent or temporary disability to receive and understand its educational content, as well as be able to use it correctly.

According to data from the World Health Organization (WHO), around a billion users worldwide suffer some form of physical or mental disability [10]. Therefore, accessibility in serious games is essential to provide a better interaction between users and video games. Without a doubt, the main reason for designing accessible serious games is to provide access to a more significant number of users, including people with some type of disability.

Therefore, this research proposes to apply a combined method to evaluate accessibility in serious games, taking into account the Web Content Accessibility Guidelines (WCAG) 2.1. In this research, we evaluated the accessibility of 82 serious games developed in HTML5 by Physical Education Technology (PhET) Interactive Simulations at the University of Colorado [11]. PhET offers fun, free, interactive simulations in Mathematics, Biology, Chemistry, Earth Science, and Physics [12] that are based on research. It has a total of 83 simulations developed in HTML5, 57 applications in Java via CheerpJ, 63 simulations developed in Java, and 12 applications in Flash. The applications can be executed online or downloaded to a computer. All simulations include the HTML5 source code and PhET’s Javascript, located on PhET’s GitHub page. The PhET project has several sponsors that make these resources free to all teachers and students.

In this study, we consider web accessibility as a starting point, which implies how users perceive, navigate, understand and interact on the web [13]; therefore, it is essential to keep in mind that the level of accessibility is the fundamental basis to facilitate access to serious games, especially for users with disabilities.

The manual method applied in this investigation comprises nine phases: (1) select serious games; (2) define the type of user; (3) define the test scenario; (4) explore each game to evaluate; (5) list the barriers based on Web Content Accessibility Guidelines (WCAG) 2.1; (6) evaluate with automatic tools and manually; (7) record evaluation data; (8) classify and analyze data; and (9) provide suggestions to improve accessibility. Besides, in the evaluation, the authors considered the WCAG 2.1 [14,15] based on five parameters: (1) accessible content; (2) visible focus; (3) accessible with a keyboard; (4) association of labels and controls; (5) controls for animation and audio. This research invites reflection and considers the importance of complying with and applying accessibility standards in the design of serious games considering diversity.

This research can serve as a guide for serious game designers and developers to apply WCAG 2.1 with an acceptable level of accessibility; additionally, this study can serve as a starting point for future work related to accessibility in serious games.

This research is structured as follows: Section 1 presents the introduction; Section 2 describes the background and previous work related to accessibility in serious games; Section 3 presents the method and the case study; Section 4 discusses the evidence and results; Section 5 presents conclusions and future work.

2. Background and Related Work

Currently, there are a large number of websites offering serious simulations and games. Statista [16] estimated that market revenue based on serious games worldwide is expected to grow from USD 3.5 billion in 2018 to 24 billion in 2024. Furthermore, in these times of social distancing [17], serious gaming has become highly supportive of teaching resources in the educational area. However, not all serious games are accessible. Accessibility refers to the condition if someone, regardless of their disability, can use serious games without barriers that prevent regular use and interaction with it.

2.1. Serious Games

The definition of serious games has been around before computing devices and entertainment. According to Schell [18], the definition indicates that games include goals, challenges, and rules to win or lose. Abt [19] indicates that “serious games” have an explicit and carefully thought-out educational purpose and are not intended to be played with fun only in mind. López et al. [20] argue that a serious game retains all the characteristics of a typical game, but point to a higher purpose than mere fun.

Jaramillo-Alcázar et al. [21] formulate that serious games allow the teaching of various types of concepts but that various games are not accessible since they do not focus on groups with disabilities. Salvador-Ullauri et al. [22] explain that it is a great challenge to implement serious games to support learning processes, especially for people with cognitive disabilities. In previous studies [22], the authors evaluated accessibility in ten serious games considering WCAG 2.0 [23]. The results revealed that serious games did not reach an adequate level of accessibility.

2.2. Accessibility

According to Park and Kim [24], accessibility in serious games makes it possible to guarantee that any user in any technical circumstance can access the content; for this reason, in our research, we applied WCAG 2.1 [14], the last official version of June 5, 2018, that provides recommendations to make content more accessible including people with disability. WCAG 2.1 consists of 4 principles, 13 guidelines, and 78 compliance criteria, and includes an undetermined number of sufficient techniques and advice. The four principles are the same as those presented in WCAG 2.0 [23]:

Principle 1: Perceivable—All users must be able to perceive the content in a visual, sound, or tactile way; Principle 2: Operable—Users must be able to use and navigate the interface components; Principle 3: Understandable—Both the content and the controls of the interface for its management must be understandable to the user; Principle 4: Robust—Content must be robust to be interpreted reliably by the most significant number of users with current and future technologies.

WCAG 2.1 [14] proposes success criteria associated with one of the following compliance levels: Level “A”: Minimum level of accessibility, when not reaching it, users cannot access the content of the web; Level “AA”: Intermediate level, implies that it is difficult for users to access the content; Level “AAA”: Maximum level, when the users can access the content without difficulty.

The authors found several accessibility studies in interactive simulations as part of the serious games contributing to this research. All these works were selected considering the interests of the scientific community to integrate people with disabilities into the academic world using new technologies.

Araújo et al. [25] argued that video games are increasingly popular but are not accessible, which represents a significant challenge for accessibility experts and game designers. The authors presented a study on the existing guidelines and recommendations for accessibility in video games, and they proposed 10 design recommendations for people with visual disabilities.

Cairns et al. [26] indicated that video games provide a cultural outlet where more players can be included and interact to do activities in a balanced way between different users. This event is possible if we create design environments that provide inclusive opportunities. The authors suggest including the guidelines with a language of accessibility of the game; they propose to (1) include a structure for the vocabulary of the game, (2) empower to meet the challenges of the game, (3) improve the player experience. Besides, they show how incorporating the guidelines in the development of video games provide accessible experiences to a more significant number of users.

Park and Kim [24] argued that the legislation of the accessibility guidelines could guarantee easy access to web content, considering users with disabilities but not in video games, since the contents of the web or mobile application consist of reasonably simple information in comparison with the contents of the video game. The content of the video game includes a more significant number of (1) characters, (2) players, (3) conflicts between them, and (4) updating a character when completing a mission. Therefore, they explain that it is necessary to analyze and classify the accessibility guidelines to evaluate each video game.

Waki et al. [27] stated that, currently, the lack of accessibility in digital games imposes barriers for people with disabilities. The authors propose a process to evaluate a set of integrated guidelines. The results revealed that the set of integrated guidelines allows determining the accessibility of digital games and refine the set of integrated guidelines.

Westin et al. [28] formulated that accessibility in video games consists of eliminating the possible barriers that prevent people with disabilities from accessing video games. The authors compared WCAG 2.1 with a set of accessibility guidelines for digital games. They compared 107 guidelines for accessible games.

Wilson and Crabb [29] indicated that video games, in particular, games on mobile devices, have evolved rapidly throughout the world. One problem identified in this topic is the accessibility that users face, especially if they have some kind of disability. In conclusion, the authors were able to determine the participants’ knowledge of accessibility guidelines and identify opinions on the importance of applying accessibility guidelines when creating accessible content for mobile games.

Spyridonis and Daylamani-Zad [30] argued the designers’ lack of commitment in designing serious games in the application of WCAG. They proposed to (1) focus on user-centered design, (2) identify the types of users, (3) apply WCAG to serious game mechanics, (4) measure user satisfaction, and (5) apply mixed methods. The results revealed that when applying the WCAG, the serious games presented an innovative and attractive solution.

3. Method and Case Study

In this research, we propose the application of a combined method to evaluate accessibility in serious games, considering the Web Content Accessibility Guidelines (WCAG) 2.1. As a case study, we evaluated the accessibility in 82 serious games developed by the PhET project [12]. The evaluation started on 19 January 2020 and ended on 20 June 2020. The study involved two expert evaluators in accessibility, who had experienced since 2015 and have contributed several articles in the area when there were discrepancies in the evaluation of the collaboration of a third expert was requested. In this study, the combined method includes the application of two automatic tools and manual evaluation. The evaluation method is summarized in the eight steps shown in Figure 1.

Phase 1: Select the serious games. In this phase, we selected the serious games of the PhET project. To select the sample size, we apply equation (1).

$$n=\frac{N{Z}^{2}PQ}{\left(N-1\right)E^2+Z^{2}PQ}$$

(1)

where n is the sample size to be calculated, N is the size of the universe, in this case, 87 serious games, with a confidence level of 97%, where Z corresponds to 2.17, with an error margin E = 0.03. Applying the formula, we obtained the value of 82, which is the number of serious games, selected sequentially, and evaluated in this research by the two accessibility experts.

Table 1. Serious games were selected for evaluation.

ID	URL	Subject
1aci	acid-base-solutions/1.2.24/acid-base-solutions_es.html	Chemistry
2are	area-builder/1.1.20/area-builder_es.html	Math
3are	area-model-algebra/1.2.1/area-model-algebra_es.html	Math
4are	area-model-decimals/1.2.1/area-model-decimals_es.html	Math
5are	area-model-introduction/1.2.1/area-model-introduction_es.html	Math
6ari	area-model-multiplication/1.2.1/area-model-multiplication_es.html	Math
7ato	arithmetic/1.0.24/arithmetic_es.html	Math
8bal	atomic-interactions/1.1.0/atomic-interactions_es.html	Chemistry
9bal	balancing-act/1.1.24/balancing-act_es.html	Math
10bal	balancing-chemical-equations/1.2.10/balancing-chemical-equations_es.html	Chemistry
11bee	balloons-and-static-electricity/1.4.14/balloons-and-static-electricity_es.html	Earth science
12ben	beers-law-lab/1.4.18/beers-law-lab_es.html	Physics
13bla	bending-light/1.1.20/bending-light_es.html	Physics
14bui	blackbody-spectrum/1.0.7/blackbody-spectrum_es.html	Earth science
15bui	build-a-fraction/1.0.12/build-a-fraction_es.html	Math
16cap	build-an-atom/1.6.14/build-an-atom_es.html	Physics
17cha	capacitor-lab-basics/1.6.19/capacitor-lab-basics_es.html	Physics
18cir	charges-and-fields/1.0.47/charges-and-fields_es.html	Physics
19cir	circuit-construction-kit-dc/1.1.5/circuit-construction-kit-dc_es.html	Physics
20col	circuit-construction-kit-dc-virtual-lab/1.1.5/circuit-construction-kit-dc-virtual-lab_es.html	Physics
21con	color-vision/1.1.23/color-vision_es.html	Biology
22cou	concentration/1.3.20/concentration_es.html	Physics
23cur	curve-fitting/1.0.0/curve-fitting_es.html	Math
24dif	coulombs-law/1.0.9/coulombs-law_es.html	Physics
25ene	diffusion/1.0.4/diffusion_es.html	Earth science
26ene	energy-forms-and-changes/1.0.11/energy-forms-and-changes_es.html	Physics
27equ	energy-skate-park-basics/1.1.19/energy-skate-park-basics_es.html	Physics
28equ	equality-explorer/1.0.12/equality-explorer_es.html	Math
29equ	equality-explorer-basics/1.0.12/equality-explorer-basics_es.html	Math
30exp	equality-explorer-two-variables/1.0.12/equality-explorer-two-variables_es.html	Math
31far	expression-exchange/1.1.14/expression-exchange_es.html	Math
32for	faradays-law/1.1.23/faradays-law_es.html	Physics
33fra	forces-and-motion-basics/2.3.16/forces-and-motion-basics_es.html	Physics
34fra	fractions-equality/1.1.1/fractions-equality_es.html	Math
35fra	fraction-matcher/1.2.1/fraction-matcher_es.html	Math
36fra	fractions-intro/1.0.12/fractions-intro_es.html	Math
37fri	fractions-mixed-numbers/1.0.12/fractions-mixed-numbers_es.html	Math
38fun	friction/1.5.10/friction_es.html	Physics
39fun	function-builder/1.0.23/function-builder_es.html	Math
40gas	function-builder-basics/1.0.14/function-builder-basics_es.html	Math
41gas	gases-intro/1.0.5/gases-intro_es.html	Earth science
42gen	gas-properties/1.0.4/gas-properties_es.html	Earth science
43gra	gene-expression-essentials/1.0.16/gene-expression-essentials_es.html	Biology
44gra	graphing-lines/1.3.10/graphing-lines_es.html	Math
45gra	graphing-quadratics/1.1.5/graphing-quadratics_es.html	Math
46gra	graphing-slope-intercept/1.1.9/graphing-slope-intercept_es.html	Math
47gra	gravity-and-orbits/1.1.15/gravity-and-orbits_es.html	Physics
48gra	gravity-force-lab/2.2.0/gravity-force-lab_es.html	Physics
49hoo	gravity-force-lab-basics/1.0.0/gravity-force-lab-basics_es.html	Physics
50iso	hookes-law/1.0.23/hookes-law_es.html	Chemistry
51joh	isotopes-and-atomic-mass/1.1.9/isotopes-and-atomic-mass_es.html	Chemistry
52lea	john-travoltage/1.5.12/john-travoltage_es.html	Physics
53mak	least-squares-regression/1.1.20/least-squares-regression_es.html	Math
54mas	make-a-ten/1.0.16/make-a-ten_es.html	Math
55mas	masses-and-springs/1.0.10/masses-and-springs_es.html	Math
56mol	masses-and-springs-basics/1.0.9/masses-and-springs-basics_es.html	Physics
57mol	molarity/1.5.1/molarity_es.html	Chemistry
58mol	molecules-and-light/1.4.14/molecules-and-light_es.html	Earth science
59mol	molecule-shapes/1.2.8/molecule-shapes_es.html	Chemistry
60mol	molecule-shapes-basics/1.2.8/molecule-shapes-basics_es.html	Chemistry
61neu	neuron/1.1.18/neuron_es.html	Biology
62ohm	ohms-law/1.4.7/ohms-law_es.html	Math
63pen	pendulum-lab/1.0.15/pendulum-lab_es.html	Math
64ph	ph-scale/1.3.4/ph-scale_es.html	Earth science
65ph	ph-scale-basics/1.3.4/ph-scale-basics_es.html	Biology
66pli	plinko-probability/1.1.18/plinko-probability_es.html	Math
67pro	projectile-motion/1.0.15/projectile-motion_es.html	Math
68pro	proportion-playground/1.0.15/proportion-playground_es.html	Math
69rea	reactants-products-and-leftovers/1.2.11/reactants-products-and-leftovers_es.html	Chemistry
70res	resistance-in-a-wire/1.6.9/resistance-in-a-wire_es.html	Math
71rut	rutherford-scattering/1.1.9/rutherford-scattering_es.html	Chemistry
72sta	states-of-matter-basics/1.1.8/states-of-matter-basics_es.html	Physics
73sta	states-of-matter/1.1.10/states-of-matter_es.html	Physics
74tri	trig-tour/1.0.22/trig-tour_es.html	Math
75und	under-pressure/1.1.18/under-pressure_es.html	Earth science
76uni	unit-rates/1.0.17/unit-rates_es.html	Math
77vec	vector-addition/1.0.0/vector-addition_es.html	Math
78vec	vector-addition-equations/1.0.0/vector-addition-equations_es.html	Math
79wav	wave-interference/2.0.2/wave-interference_es.html	Earth science
80wav	waves-intro/1.0.2/waves-intro_es.html	Earth science
81wav	wave-on-a-string/1.1.22/wave-on-a-string_es.html	Earth science
82bui	build-a-molecule/latest/build-a-molecule_en.html	Chemistry

contains the evaluated serious games, including the identification of each game and the name of the serious games within each URL (The complete URL is constructed by adding to the beginning of each address https://phet.colorado.edu/sims/html/. For example, the complete address for the first serious game is https://phet.colorado.edu/sims/html/acid-base-solutions/latest/acid-base-solutions_es.html).

Phase 2: Define the user’s type. Two experts in software application accessibility carried out the evaluation, in this case, on the serious games. This study defined the WCAG 2.1 for users with low vision [31]. According to the WHO [32], blindness and vision impairment affect at least 2.2 billion people around the world. Of those, 1 billion have a preventable vision impairment or one that has yet to be addressed. Reduced vision can have long-lasting effects on various aspects of life, and with increasing age, older people tend to decrease their presbyopia-related visual ability [33]. Based on these definitions, accessibility experts evaluated serious games by applying WCAG 2.1 to identify accessibility barriers. The experts have experience in the evaluation of mobile applications, web accessibility, and accessibility of educational resources since 2015 and have published several articles in high impact journals related to the topic.

Phase 3: Define the test scenario. In this phase, the authors define the scenarios to navigate and interact in serious games and reach the goal. The task is to (1) enter serious games, (2) review the functionality of each serious game, and (3) check if there are barriers that prevent accessibility for serious games. In this case study, an accessibility barrier for a person with low vision [34] means that the person cannot effectively move from one point to another within serious games due to visual acuity problems.

Phase 4: Explore each serious game to evaluate. In the fourth phase, the user explores and becomes familiar with the interaction mechanisms of serious games. In this phase, the evaluators identify (1) the functionalities of serious gaming; (2) if the content is adjustable with the zoom and the appearance of the game; and (3) the change of behavior according to the device, the context, and the applied configuration.

Figure 2 presents a screenshot of one of the evaluated games; in this example, it is a serious game used in Physics to explain the topic of the projectile launch.

Phase 5: List the barriers based on WCAG. This phase is essential, in which we list the barriers related to WCAG 2.1 for the evaluation of serious games. The barriers are listed in

Table 2. Guidelines for evaluating accessibility in serious games.

Guideline	Barrier	WCAG 2.1	Success Criteria	Level
G01	Accessible keyboard	Operable	2.1.1	A
G02	Luminance flash failures	Operable	2.3.1	A
G03	Animation from Interactions	Operable	2.3.3	AAA
G04	Content hovering over focus	Perceivable	1.4.13	AA
G05	Easy to read font	Perceivable	1.1.1	A
G06	Text alternatives	Perceivable	1.1.1	A
G07	Subtitled	Perceivable	1.2.4	AA
G08	Automatic transcripts	Perceivable	1.2.5	AA
G09	Sign language	Perceivable	1.2.6	AAA
G10	Information and relationships	Perceivable	1.3.1	A
G11	Sensory characteristics	Perceivable	1.3.1	A
G12	Adjust display settings	Perceivable	1.3.4	AA
G13	Interface rearrangement	Perceivable	1.3.5	AA
G14	Use of color	Perceivable	1.4.1	A
G15	Contrast without text	Perceivable	1.4.11	AA
G16	Well-spaced elements	Perceivable	1.4.12	A
G17	Good audio techniques	Perceivable	1.4.2	A
G18	Contrast (Minimum)	Perceivable	1.4.3	AA
G19	Images as sharp as possible	Perceivable	1.4.5	AA
G20	Visual presentation	Perceivable	1.4.8	AAA
G21	Pause, stop, hide	Perceivable	2.2.2	A
G22	Contrast (Enhanced)	Perceivable	1.4.6	AAA
G23	Screen reader support	Robust	4.1.2	A
G24	Status messages	Robust	4.1.3	AA
G25	Language	Understandable	3.1.1	A
G26	Consistent navigation	Understandable	3.2.3	AA
G27	Labels or instructions	Understandable	3.3.2	A
G28	Help	Understandable	3.3.5	AAA
G29	On Focus	Understandable	3.2.1	A

, which contains 29 guidelines considered in the manual evaluation of each serious game. Table 2 shows the guideline, the barrier, the WCAG 2.1 principle, the success criteria, and the level.

Phase 6: Evaluate with automatic tools and manually. In this phase, we evaluated the 82 serious games detailed in Table 1. We applied a combined evaluation using two automatic tools: (1) Colour Contrast Analyser (CCA), version 3.0.1 [35], a tool used to analyze some WCAG 2.1 to set the input colors of plain text. It also allows support for alpha transparency in foreground colors, includes a color blindness simulator. This tool allows a foreground and background color swatch to be taken to measure the contrast against which a report is output with WCAG 2.1. (2) Photosensitive Epilepsy Analysis Tool (PEAT), version 1.6 [36], allows evaluation in serious games or animations whether the content presents flickering or rapid transitions between light and dark background colors that can generate photosensitive seizures caused by certain types of flashing in serious gameplay, including mouse-overs that cause large areas of the screen to turn on and off quickly. The tool applies some of the WCAG 2.1 and 2.2, including restrictions related to frequency, luminance, area, and color of any flicker. Before analyzing the serious games with PEAT, the games were transformed into format Audio Video Interleave (.AVI) using a batch processing program to take a sample of the video.

Phase 7: Record evaluation data. In this phase, we present the registration of the data obtained from the evaluation of the accessibility of serious games with CCA and PEAT tools; the evaluation dataset [37] can be found at https://data.mendeley.com/datasets/t2tr35ww4c/5. The codes assigned to each column, which correspond to the WCAG success criteria, where C1 = 1.4.3 Contrast (Minimum) AA, Regular Text, C2 = 1.4.3 Contrast (Minimum) AA, Large Text, C3 = 1.4.6 Contrast (Enhanced) AA, Regular Text, C4 = 1. 4.6 Contrast (Enhanced) AAA, Large Text, C5 = 1.4.11 Contrast without text AA (Button Reset), C6 = 1.4.11 Contrast without text AA (Others), P1 = State, P2 = Luminosity flash failures, P3 = Red flash failures, P4 = Extended flash alerts. The number one (1) indicates that the barrier has been exceeded, and the number zero (0) indicates that it has not been exceeded. Figure 3 presents the data recorded in the manual evaluation of the serious games with WCAG 2.1, contains the identifier assigned to each serious game with the 29 guidelines. We place the value of one (1) if the barrier is exceeded and zero (0) when the barrier was not exceeded. In Figure 3, the darkest color represents that the barrier was exceeded and the lightest color that the barrier was not exceeded.

Phase 8: Classify and analyze data. In this phase, the data of serious games were classified, considering the four principles of web accessibility proposed in WCAG 2.1. We grouped the data obtained with the automatic CCA and PEAT tools with which the contrast and photosensitivity that affects users with epilepsy were analyzed. The data obtained in the manual evaluation were grouped to identify the presence of each of the accessibility principles, the success criteria, and the level of accessibility, which will be detailed in the results analysis section. This process takes a long time and is therefore very demanding. Recorded data and analysis are available at Mendeley (https://data.mendeley.com/datasets/t2tr35ww4c/5) so that the assessment can be replicated [37].

4. Results and Discussion

In this phase, we analyze the results with automatic and manual evaluation tools. Applying descriptive statistics to the evaluation data with the CCA and PEAT tools, we have that the mean value is 3.9, the standard error is 0.21; the median is 5.0, the mode is 5.0, the standard deviation is 1.9, the variance of the sample is 3.64, the minimum value is 1.0, the maximum is 7.0. Figure 4 shows the evaluation with the CCA and PEAT tools; we found that 24 serious games meet from 1 to 2.5 in the evaluated contrast and photosensitivity parameters, 10 serious games meet from 2.5 to 4, the next 30 from 4 to 5.5, and 18 serious games meet of 5.5 to 7 points. From the data in

Table 3. Summary of Accessibility Evaluation in Serious Games with the Manual Method.

Guideline	Barrier	WCAG 2.1 (Principle)	Success Criteria	Level	Total	%
G01	Accessible keyboard	Operable	2.1.1	A	8	1.3
G02	Luminance flash failures	Operable	2.3.1	A	81	13
G03	Animation from Interactions	Operable	2.3.3	AAA	27	4.2
G04	Content hovering over focus	Perceivable	1.4.13	AA	6	0.9
G05	Easy to read font	Perceivable	1.1.1	A	82	13
G06	Text alternatives	Perceivable	1.1.1	A	1	0.2
G07	Subtitled	Perceivable	1.2.4	AA	0	0
G08	Automatic transcripts	Perceivable	1.2.5	AA	0	0
G09	Sign language	Perceivable	1.2.6	AAA	0	0
G10	Information and relationships	Perceivable	1.3.1	A	0	0
G11	Sensory characteristics	Perceivable	1.3.1	A	0	0
G12	Adjust display settings	Perceivable	1.3.4	AA	0	0
G13	Interface rearrangement	Perceivable	1.3.5	AA	0	0
G14	Use of color	Perceivable	1.4.1	A	32	5
G15	Contrast without text	Perceivable	1.4.11	AA	2	0.3
G16	Well-spaced elements	Perceivable	1.4.12	A	82	13
G17	Good audio techniques	Perceivable	1.4.2	A	20	3.1
G18	Contrast (Minimum)	Perceivable	1.4.3	AA	62	9.7
G19	Images as sharp as possible	Perceivable	1.4.5	AA	2	0.3
G20	Visual presentation	Perceivable	1.4.8	AAA	12	1.9
G21	Pause, stop, hide	Perceivable	2.2.2	A	27	4.2
G22	Contrast (Enhanced)	Perceivable	1.4.6	AAA	20	3.1
G23	Screen reader support	Robust	4.1.2	A	0	0
G24	Status messages	Robust	4.1.3	AA	0	0
G25	Language	Understandable	3.1.1	A	82	13
G26	Consistent navigation	Understandable	3.2.3	AA	82	13
G27	Labels or instructions	Understandable	3.3.2	A	5	0.8
G28	Help	Understandable	3.3.5	AAA	0	0
G29	On Focus	Understandable	3.2.1	A	6	0.9

, we find that the 82 serious games passed the photosensitivity test.

Table 3 shows a summary of the manual evaluation of 82 serious games; it contains the guideline, barrier, accessibility principle, success criteria, level, total, and percentage of accessibility evaluation in serious games with the manual method. The total column shows the value of the guidelines that exceed the barrier. We found that the 82 serious games compliance the following guidelines: (1) G05 related to easy to read font, (2) G16 related to well-spaced elements, (3) G25 related to language, (4) G26 related to consistent navigation.

Figure 5 shows the percentage of compliance with accessibility principles of the 82 serious games evaluated. We found that the perceptible principle is fulfilled in 54.4% of the total, the comprehensible one registers 27.4% of fulfillment, the operable one registers 18.2%, and the robust one registers 0.0%. We can conclude that in the evaluation of the accessibility of the 82 serious games, the most violated principle is the robust one.

Figure 6 shows the manual evaluation of the accessibility in 82 serious games; we obtained the following success criteria that overcome the barriers in the manual evaluation: (1) the success criteria representing 64% of the total are 1.1.1 for easy to read font, 1.4.12 for well-spaced elements, 3.1.1 for language, 3.2.3 for consistent navigation and 2.3.1 for luminance flash failures; (2) the success criteria representing 9.7% of the total is 1.4.3 for enhanced contrast; (3) the success criteria representing 5% of the total is 1.4.1 for the use of color; (4) the success criteria representing 8.4% of the total are 2.3.3 for animation from interactions, and 2.2.2 for pause, stop, hide; (5) the rest of the success criteria representing 12.8% of the total.

Figure 7 shows the evaluation of accessibility in serious games with the following results: (1) the guidelines G05, G16, G25, G02, G14, G21, G17, G01, G29, G27, G06, G10, G11, and G23 represent 66.6% of total compliance concerning Level A; (2) G26, G18, G04, G15, G19, G07, G08, G12, G13, and G24 guidelines represent 24.3% of total compliance for Level AA; (3) G03, G20, G22, G09, and G28 guidelines represent 9.1% of total compliance for Level AAA. For a serious game to have an acceptable level of accessibility, WCAG 2.1 suggests considering level AA [14].

5. Conclusions and Future Work

During the literature review, we found that there is no solid work focused on accessibility with WCAG 2.1. Therefore, this work proposes the application of a combined method of evaluating accessibility in serious games and establishes 29 guidelines based on WCAG. From the case study, we can conclude that the most neglected accessibility principles are operable and robust; the operable principle refers to how people interact and have control during interaction with serious games. On the other hand, “robust” barriers related to related ones can be addressed by including assistive technology that allows better compatibility with current and future tools.

This combined approach can be replicated for other serious games that include users with different types of disabilities. To improve accessibility in serious gaming, we propose to include (1) automatic transcriptions or a bot to transcribe the audio or video to text without human intervention, so that the user can review the transcriptions while playing; (2) sign language to establish a communication channel with the social environment, this option is useful for users with hearing disabilities; (3) photosensitivity control, to configure the excess of light and brightness useful in users with epilepsy problems; (4) external devices to allow virtual and augmented reality, for example, with the oculus quest the user can achieve greater immersion in the learning process; and (5) a contextual help option so that users can operate without losing sight of what they are doing. The authors can conclude that no serious games among those selected have reached an acceptable level of accessibility. Therefore, serious game developers should make significant efforts to improve accessibility.

This study has its limitations because it is a combined method that includes a manual part where accessibility experts intervene; the evaluation results depend on the experience and ability to evaluate serious games. From the theoretical point of view, we present a new method to evaluate accessibility in serious games based on a combined method. On the other hand, in practical terms, this study tries to wake up and motivate serious game developers to apply WCAG 2.1 guidelines to make serious games more accessible and inclusive that consider diversity. Additionally, this method can serve as a reference for future studies related to accessibility in simulations and serious games.

As future work, we suggest (1) testing this method with users with different types of disabilities and applying the corresponding disability-related guidelines; (2) generating a heuristic accessibility evaluation focused on serious games, based on the accessibility issues identified in WCAG 2.1; (3) expanding the serious games database with some applications other than serious games to deepen the analysis and evaluation, and (4) developing a software application that includes the WCAG 2.1 guidelines to assist in the evaluation of serious games.