Accessibility Analysis of Worldwide COVID-19-Related Information Portals

Since the beginning of the COVID-19 pandemic, communication technology has demonstrated its usefulness in sharing and receiving health data and communicating with the public. This study evaluated the accessibility of 199 websites containing official COVID-19 information related to medical schools, governments, ministries, and medical associations, obtained from the Geneva Foundation for Medical Education and Research website. We used the Web Content Accessibility Guidelines 2.1 to evaluate web accessibility, using a six-phase process with an automatic review tool. The study results reveal that the highest number of barriers encountered are concentrated in the perceivable principle with 6388 errors (77.8%), followed by operability with 1457 (17.7%), then robustness with 291 (3.5%), and finally understandability with 78 errors (0.9%). This study concludes that most COVID-19-related websites that provide information on the context of the pandemic do not have an adequate level of accessibility. This study can contribute as a guide for designing inclusive websites; web accessibility should be reviewed periodically due to technological advances and the need to adapt to these changes.


Introduction
World Health Organization (WHO) reports reveal that 15% of the world's population has a disability [1]. The prevalence of people with disabilities is significantly higher in developing countries [2]. As COVID-19 has a global large-scale impact [3], it is essential to recognize that accessibility is fundamental to protecting the most vulnerable people whose disabilities make it difficult to access other health and educational services, including schooling, transportation, and health-related information.
During the COVID-19 pandemic, technology has enabled governments and healthrelated institutions to inform and share content intended to increase health promotion [4]. In this sense, several websites were created to report COVID-19 epidemiological data, prevention strategies, and other measures to try to ameliorate the impact of the pandemic worldwide [5]. These websites have become a powerful tool for the public to obtain information and to use as contrast with misinformation, i.e., information that was not generated by governments, including countries' health departments and non-governmental organizations, as well as universities or medical centers.
However, not all websites are accessible to all citizens, especially those whose eyesight has diminished with advancing age due to presbyopia. Web-based communication is essential, so removing accessibility barriers is required for websites to serve all people who and seek information on social networks or other misinformation channels that attempt to mislead, confuse, and alarm the public.
The applied method comprises an automatic review using the WAVE [11] in six stages detailed in the Methods Section. This research can help future research associated with accessibility for any website. In addition, it can be an input to help application developers to design software with artificial intelligence to help experts evaluate accessibility in websites applying WCAG 2.1.
The rest of the manuscript is structured as follows: Section 2 presents the background and related works; Section 3 shows the methods and materials; Section 4 contains the results and analysis; Section 5 contains the discussion of the evaluated websites; and finally, Section 6 includes the conclusions, limitations, and future works obtained in this research.

Background and Related Works
The authors of [4] noted that, during the COVID-19 pandemic, technology was essential for health agencies to communicate with the public. The study analyzed two COVID-19 government websites using the WCAG standard. The analyzed websites required improvement in the accessibility of COVID-19 information as it plays a crucial role in the health and safety of the public.
The exploratory study of [12] investigated the web accessibility of 54 official COVID-19 vaccine registry websites in the US and their compliance with WCAG 2.0 and 2.1 guidelines. The results revealed compliance with WCAG 2.0 and 2.1 guidelines. Non-compliance creates difficulties for users with disabilities to access information on websites and the article proposed recommendations for governments to improve the accessibility of their websites.
The study of [13] argued that the United States Centers for Disease Control and Prevention allocates vaccines to individual states for distribution. Individuals seeking COVID-19 vaccine eligibility require information and thus need to visit health department websites. However, different levels of technological expertise, reading skills, and language preferences may prevent individuals from obtaining the necessary information regarding COVID-19. The results indicate that 14% of the websites did not have complete smartphone visibility and 18% were in English. The authors suggested that websites should enable mobile access to information in multiple languages, using a language that is easier for the public to understand to address these problems.
The research of [14,15] addressed the issue by considering the situation of the health crisis that required greater attention and reliance on trustworthy sources for citizens, especially those organizations representing competent authorities, such as the World Health Organization (WHO). Therefore, information must be understandable by and accessible to all people, regardless of technology access, language, culture, or disability, according to the W3C [4]. The study of [14] contributed to the development of proposals and suggested ways to improve the accessibility of healthcare content, especially vulnerable in this pandemic. The results revealed that understandable and perceptible principles are essential for communicating with citizens. Additionally, they are directly related to easy-to-understand content in the first case and offer text alternatives for non-textual content in the second case, especially for visually impaired people who use readers and even older people who have vision problems due to aging.
According to WHO [7] figures worldwide, it is estimated that approximately 1.3 billion people live with some form of visual impairment. The leading causes of poor vision are uncorrected refractive errors and cataracts. Most people with poor vision are over 50 years of age; however, vision loss can affect people of all ages. Vision impairment poses a substantial global economic burden, as the annual costs due to lost productivity associated with vision impairment from uncorrected myopia and presbyopia are estimated to be USD 244 billion and USD 25.4 billion, respectively, worldwide. According to several studies, visual impairment is associated with decreased quality of life [1,2,6,7], limitations in the performance of daily activities, and increased risk of accidents, depression, social isolation, and other alterations in health status [16].
In general, Web accessibility means that people with disabilities can use the Web, and it refers to a Web design that allows people to perceive, understand, navigate, and interact with the Web while providing the content [17]. In addition, Web accessibility benefits everyone, including the elderly who have had their abilities reduced because of age. The W3C [18] Web Accessibility Initiative (WAI) is an organization that introduced accessibility guidelines known as Web Content Accessibility Guidelines (WCAG). It provides WCAG 1.0 [19], WCAG 2.0 [20], WCAG 2.1 [8], WCAG 2.2 [21], and WCAG 3.0 [22] for three levels of conformance A, AA, and AAA. WCAG aims to ensure that websites or their content are accessible and inclusive for people with disabilities.
According to World Bank statistics, one billion people worldwide, 15% of the world's population [1], live with some form of disability, such as visual impairment, hearing impairment, physical disability, or cognitive or neurological impairment.
According to World Health Organization (WHO) reports, among the entire population of the world, 3.2% of people have a visual impairment, 6% of people have hearing or hearing difficulties, 2.6% of people have a neurological difficulty, and 1% of people have physical disabilities or require a wheelchair [23].
From the described above, our research addresses and analyzes websites containing information on COVID-19 to share the results found in the evaluation of the total sample of sites obtained from Geneva Foundation for Medical Education and Research [24]; in addition, WHO is included.

Materials and Methods
The evaluation process comprised two phases: (1) selecting the automatic evaluation tool that meets the accessibility standards and (2) evaluating the accessibility of the websites with the tool that achieves the best results in the selection.
In the evaluation, we considered some parameters such as the accessibility four principles (perceivable, operable, understandable, and robust), the application of the guidelines, and the version and the level of accessibility (A, AA, and AAA). In addition, we analyzed features such as loading times, license, report, plugin, support and maintenance, functionality, portability, usability, connectivity, and security. To select the most appropriate tool, we present a summary in Table 1, containing ID, tool, WCAG 2.1 and 2.0, level A, AA, AAA, and total and percentage score.
The results indicate that those that meet 100% of the parameters of the tools evaluated correspond to Tenon, WAVE [11,25], Web Accessibility Checker, and Mauve++ Tool. In the selection analysis of the auto-math evaluation tool, parameters such as loading times, license, report, plugin, support, functionality, portability, usability, connectivity, and security were considered. In the evaluation, 1 was recorded if the tool complied, and 0 if it did not comply. Similarly, the eight tools were then calculated for compliance or non-compliance with the level of accessibility, and the percentage score was calculated. It was observed that the tools Tenon, WAVE, Web Accessibility Checker, and Mauve++ Tool reach 13.9% each.
In addition, we added criteria for analyzing the quality of the tool. Although these tools are an invaluable aid in evaluating the accessibility of websites, readers should understand that they are far from infallible and have limitations that can lead to false positives. The evaluation of the tools depends on the experience of the evaluators as well as the evaluators' judgments. The results are summarized in Table 2, containing the identifier, loading times, license, report, plugin, support, functionality, portability, usability, connectivity, security, total, percentage additional score, and percentage average. Where ID = identifier, Lt = loading times, L = license, R = report, Pn = plugin, St = support, F = functionality, P = portability, U = usability, C = connectivity, S = security. Excellent = 3, Very Good = 2, Good = 1, Regular = 0.
To calculate the percentage average, we used the percentage score from Table 1 with the additional percentage score. As a result, the tool that has the best performance in the evaluation is WAVE with 17.3%, followed by Mauve++ Tool with 16.1% and the third-place Tenon with 13.7%.
Phase 2: Once the tool was defined, the sample of websites to be evaluated was obtained from the Geneva Foundation for Medical Education and Research (GFMER) [24]. The data were available in the Mendeley repository. A total of 198 GFMER websites were evaluated, and the WHO website was also included. The evaluation began on 15 October 2021, and five evaluators participated. An automatic review method was applied using the WAVE tool for the evaluation. The method is described in six steps, summarized in Figure 1.
In step 1, we took the total sample of GFMER [24] websites and included one of the most consulted websites in pandemics, such as the World Health Organization (WHO). We selected 199 websites to evaluate accessibility. Table 3 presents a sample of the first 40 websites evaluated; the total evaluated websites and the analysis are available with open access to the data in the Mendeley Data repository [25]. 2021, and five evaluators participated. An automatic review method was applied using the WAVE tool for the evaluation. The method is described in six steps, summarized in Figure 1. In step 1, we took the total sample of GFMER [24] websites and included one of the most consulted websites in pandemics, such as the World Health Organization (WHO). We selected 199 websites to evaluate accessibility. Table 3 presents a sample of the first 40 websites evaluated; the total evaluated websites and the analysis are available with open access to the data in the Mendeley Data repository [25].    In step 2, we experimented with the navigation and interaction with each website that provides official information about COVID-19. In step 3, we defined the test environment so that the same functions are performed on all websites to be evaluated. The test environment is used for user acceptance testing activities. In this phase, we previously installed the Google Chrome browser version 101.0.4951.64 for 64 bits, added the WAVE Evaluation Tool extension under, and offered by WebAIM, version 3.1.6, with an updated date of 14 October 2021, under the Windows 10 operating system. Then, we performed the following actions: (1) open the browser; (2) type the URL address of the website; and (3) load the main page.
In step 4, once the actions of step 3 are performed, we run the WAVE plugin; we waited for the website analysis to load. We reviewed the summary and detailed information from the WAVE tool.
In step 5, we recorded the data from the Summary section in a spreadsheet; in this study, we used Microsoft Excel 365. We examined the values of Errors, Contrast Errors, Alerts, Features, Structural Elements, and ARIA. In addition, we inspected each of the barriers related to Errors and Contrast Errors from the Details option. The data were classified according to the four accessibility principles and the success criteria.
Finally, in step 6, we analyzed the data and identified the websites with the highest number of errors and contrast errors that should be corrected to improve accessibility. For this study, we considered level AA. The results of this analysis are detailed in the Discussion section.

Results and Analysis
This section presents the evaluation results of the total GFMER sample, including 199 websites; they were classified and tabulated according to the four accessibility principles of WCAG 2.1 [8]. We considered the principle of being perceivable related to users with visual impairment, specifically low vision, whose visual ability decreased due to presbyopia related to aging. The information was summarized using Microsoft Excel with the support of dynamic reports. Table 4 presents the evaluation of the first 40 websites; the total sample is in the open data repository Mendeley [25].
In Table 4, errors [11] indicate a significant accessibility problem; the number of errors is a good indicator of real end-user accessibility problems that require urgent resolution. The contrast errors [11] parameter reveals the contrast and use of color vital for accessibility; visually impaired users must be able to perceive the page content.
Alerts [11] correspond to elements that may cause accessibility problems, where the evaluator is the one who decides the accessibility impact of the website.
Features [11] imply that the elements can improve accessibility when implemented correctly. Structural elements are related to some web page title, indicating that it has been marked as a top-level title or various milestones.
Finally, accessible rich internet applications (ARIA) [11] is a collection of attributes that define how to perform web content and applications; thus, it influences accessibility when misused.  These results show that the websites of the mentioned countries should urgently correct the websites to reduce accessibility barriers, especially for users with a low vision directly related to contrast errors. The results reveal that the leading websites with significant accessibility problems do not necessarily correspond to some developed or developing countries; it is supported by the study of the million made by WAVE [11], where the percentage of low contrast errors corresponds to 86.3%, mainly related to the perceivable principle.  Table 5 presents the four accessibility principles according to WCAG 2.1, the guideline, the success criteria, the level of accessibility, and the number of barriers identified in the analyzed websites, all related to the corresponding principle.  The results reveal that the leading websites with significant accessibility problems do not necessarily correspond to some developed or developing countries; it is supported by the study of the million made by WAVE [11], where the percentage of low contrast errors corresponds to 86.3%, mainly related to the perceivable principle. Table 5 presents the four accessibility principles according to WCAG 2.1, the guideline, the success criteria, the level of accessibility, and the number of barriers identified in the analyzed websites, all related to the corresponding principle.
The perceivable principle [8] is related to users with low vision and corresponds to the success criterion 1.4.3 of minimum contrast; as it can be seen, it is the criterion with the most accessibility barriers with 54.6%, followed by 1.1.1 of missing alternative text with 16.6% and 2.4.4 of link purpose with 9.5%. The rest of the success criteria are under 7.1%. Where the first number refers to the accessibility principle, the second number relates to the guideline, and the third number to the success criteria defined by WCAG 2.1 [8]. Figure 3 presents a summary of the errors identified according to the four principles of accessibility [8]. We can observe that the principle repeated most frequently in the analysis is perceivable with 6388, representing 77.8%. This principle relates to the information transmitted by non-textual content through textual alternatives. Screen readers detect the presence of images and other non-textual content, but they cannot interpret their content. For example, they will not know whether the content is a photograph of an object or a decorative figure. It is the developers' responsibility to communicate to the screen reader, and therefore to the end user, the information or function provided by the non-textual content. When creating a textual alternative, the first thing to consider is how the page will be presented or "heard" when the images or non-textual content are not displayed. Following this is the operable principle with 17.7%, followed by robust with 3.5%, and finally understandable with 0.9%. The perceivable principle [8] is related to users with low vision and corresponds to the success criterion 1.4.3 of minimum contrast; as it can be seen, it is the criterion with the most accessibility barriers with 54.6%, followed by 1.1.1 of missing alternative text with 16.6% and 2.4.4 of link purpose with 9.5%. The rest of the success criteria are under 7.1%. Figure 3 presents a summary of the errors identified according to the four principles of accessibility [8]. We can observe that the principle repeated most frequently in the analysis is perceivable with 6388, representing 77.8%. This principle relates to the information transmitted by non-textual content through textual alternatives. Screen readers detect the presence of images and other non-textual content, but they cannot interpret their content. For example, they will not know whether the content is a photograph of an object or a decorative figure. It is the developers' responsibility to communicate to the screen reader, and therefore to the end user, the information or function provided by the non-textual content. When creating a textual alternative, the first thing to consider is how the page will be presented or "heard" when the images or non-textual content are not displayed. Following this is the operable principle with 17.7%, followed by robust with 3.5%, and finally understandable with 0.9%.  Figure 4 presents the problems related to the accessibility guidelines that are repeated most frequently as 1.4 [8], which corresponds to distinguishable with 54.6%, followed by 2.4, which refers to navigable with 17.1%, followed by 1.1 [8] related to text alternatives with 16.6%. The rest of the problems are less than 5.9%. We can see that the most significant number of problems are related to the perceivable principle.  Figure 4 presents the problems related to the accessibility guidelines that are repeated most frequently as 1.4 [8], which corresponds to distinguishable with 54.6%, followed by 2.4, which refers to navigable with 17.1%, followed by 1.1 [8] related to text alternatives with 16.6%. The rest of the problems are less than 5.9%. We can see that the most significant number of problems are related to the perceivable principle.

Discussion
The results show the most significant number of barriers are concentrated in the perceptible principle with 77.8%, which contrasts with the study of the million applied by WAVE [11], followed by the operable principle with 17.7%, robust with 3.5%, and finally, understandable with 0.9%. The guideline with the highest number of errors corresponds to distinguishable with 54.6%, followed by navigable with 17.1% and text alternatives with 16.6%.
The success criteria with the most barriers correspond to contrast with 54.6%, followed by missing alternative text with 16.6% and link purpose with 9.5%. The level of accessibility most affected corresponds to AA with 61.6%, followed by A with 38.4%. This study suggests that the websites providing information on COVID-19 do not have an adequate level of accessibility, according to WCAG 2.1 [8].
Of the websites evaluated, we found that the countries with zero contrast errors are the Maldives, France, Ethiopia, South Africa, Kazakhstan, El Salvador, Croatia, Samoa,

Discussion
The results show the most significant number of barriers are concentrated in the perceptible principle with 77.8%, which contrasts with the study of the million applied by WAVE [11], followed by the operable principle with 17.7%, robust with 3.5%, and finally, understandable with 0.9%. The guideline with the highest number of errors corresponds to distinguishable with 54.6%, followed by navigable with 17.1% and text alternatives with 16.6%.
The success criteria with the most barriers correspond to contrast with 54.6%, followed by missing alternative text with 16.6% and link purpose with 9.5%. The level of accessibility most affected corresponds to AA with 61.6%, followed by A with 38.4%. This study suggests that the websites providing information on COVID-19 do not have an adequate level of accessibility, according to WCAG 2.1 [8].
Of the websites evaluated, we found that the countries with zero contrast errors are the Maldives, France, Ethiopia, South Africa, Kazakhstan, El Salvador, Croatia, Samoa,

Discussion
The results show the most significant number of barriers are concentrated in the perceptible principle with 77.8%, which contrasts with the study of the million applied by WAVE [11], followed by the operable principle with 17.7%, robust with 3.5%, and finally, understandable with 0.9%. The guideline with the highest number of errors corresponds to distinguishable with 54.6%, followed by navigable with 17.1% and text alternatives with 16.6%.
The success criteria with the most barriers correspond to contrast with 54.6%, followed by missing alternative text with 16.6% and link purpose with 9.5%. The level of accessibility most affected corresponds to AA with 61.6%, followed by A with 38.4%. This study suggests that the websites providing information on COVID-19 do not have an adequate level of accessibility, according to WCAG 2.1 [8].
Of the websites evaluated, we found that the countries with zero contrast errors are the Maldives, France, Ethiopia, South Africa, Kazakhstan, El Salvador, Croatia, Samoa, Ecuador, Norway, Qatar, Botswana, Australia, Poland, Montenegro, Sri Lanka, Iran, Lithuania, and Denmark.
In the analysis of the four accessibility principles in Table 5, we can observe that the most frequently repeated error is the perceivable principle related to visual impairment problems.
On the other hand, using the WAVE tool [11] as an aid does not offer a complete solution; a manual review report by a human expert is needed to complement the study. In future studies, we will apply combined and heuristic methods for manual review [6,17]. However, this tool can be used for comparative purposes to understand the behavior of a website regarding accessibility issues.
From the results obtained in this research, it can be concluded that the evaluated websites must be improved to reach an adequate level of accessibility, for which they must apply the WCAG 2.1 [8]. to achieve the minimum AA level required by the WCAG 2.1, we suggest: Build minimalist and lightweight designs to make the website load faster; this is highly valued by users, even representing a reduction in the use of data from their devices [8].
Reduce the number of clicks to reach the content efficiently; exploring a website cannot become a journey through a maze with no way out [6,8]. Use typography that represents the essence of the brand but is also universal. Avoid justifying texts with simultaneous right and left alignment; preferably, use left alignment and write short paragraphs to avoid visual fatigue [8].
To have a quality design on the website, taking care of the graphic line of the page [18][19][20], using an appropriate color palette and contrast, an attractive structuring of the contents, and a correct distribution of white spaces.
Include alternative mechanisms [8,18] to present the same information to people with some visual disability, such as color blindness or allowing the configuration of monochromatic screens.
Place good quality images with the Alt tag or alternative text and title [6, 8,17]; the tags function as pop-up descriptions and ensure accessibility to the content from any browser.
Design a simple navigation menu that can be easily located [6,8], and use short, precise words that do not generate confusion because they are similar or very technical. It is essential not to exceed the number of sections, a maximum of eight, and levels or sub-sections, a maximum of two.
Maintain a balance between quality and quantity of content for each element [6, 8,17], such as images, videos, tables, and audio.
Include a space for contact information that is easy to find and always up to date; avoid including banners or moving content, such as gifs, as they can be challenging to read, and users may confuse them with unwanted advertising [6, 8,11].
Apply auto-magnetic review tools, such as WAVE [11], during the website design cycle as a quick and low-cost option. Finally, it is significant to perform development evaluations early, using accessibility validators to find early web development and accessibility issues.

Conclusions, Limitations, and Future Works
The errors found in the evaluated websites were mostly related to low contrast representations, images without alternative text, empty links, entries without labels in forms, empty buttons, and documents without language attributes. Many accessibility barriers extend beyond automated reviews, requiring qualitative analysis, and user testing. Nevertheless, we can begin to reduce these types of errors on websites by applying automatic review tools that do not require much technical effort, time, or budget.
The evaluation process includes two types of problems: (1) errors that must be corrected urgently as the "errors" and (2) errors that can wait to be corrected and will depend on the criteria of an evaluator who can determine if errors should be corrected regarding specific contrast errors, alerts, structural errors, ARIA for the website to meet the minimum required level of accessibility such as AA [8].
One of the limitations is that the accessibility study of the COVID-19 websites was conducted with experts in web accessibility, not with users with any disability. Future work will expand the study with tests applied to patients with visual impairment related to low vision. This study can contribute as a guide to the design of accessible and inclusive websites. On the other hand, it is essential to disseminate and review knowledge about current legislation and the bodies that regulate web accessibility. Web accessibility has to be reviewed periodically due to the technological advances that occur and the need to adapt to these changes.
Accessibility should be an essential consideration at all stages of the software development cycle, as well as quality and usability, and can be evaluated at different stages of the application development cycle to achieve this goal. The essential task should be to disseminate knowledge on how to implement web accessibility effectively and focus on innovative implementation techniques that will attract the attention of developers.
Applying WCAG 2.1 is a challenge for web and mobile application development professionals to raise awareness of this discipline, and they must be updated on accessibility guidelines to adapt web portals to the new WCAG versions.
This research presented recommendations for good practices that stimulate different actors and stressed the importance of joint efforts in the search of solutions to eliminate access barriers to COVID-19 information websites, allow advances in human development, and promote accessibility policies to improve the quality of life [1] of all people.
Finally, one of the challenges [26] of medicine is implementing health technology and the Internet of Medical Things (IoMT) to connect various medical devices that enable usable and accessible medical care. On the other hand, technology such as blockchain will help to reduce security risks for people by including systems such as hybrid technologies, combining knowledge from big data analytics, data mining, and artificial intelligence to achieve more inclusive, secure, and practical websites.