Web Accessibility in an Academic Management System in Brazil: Problems and Challenges for Attending People with Visual Impairments

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

The purpose of the study was to evaluate the SIGAA system in terms of accessibility and usability. While the study presents interesting results, there are some issues that should be addressed during the revision of the current version of the manuscript.

1/ Correct the valid definition of usability and accessibility (see for example ISO 9241-11).

2/ Correct the order of references in the content (see for example reference [28]).

3/ avoid using statements with must (for example, "the user experience must", line 138)

4/ avoid multiple references (for example, see the beginning of the Related Works section); in my opinion, you should remove such content, since it provides no information other than numbers in parentheses.

5/ Provide a summary of related work at the end of the section; this will show how your study contributes to the current body of knowledge.

6/ The reference [44] is unnecessary and not in English. I suggest deleting this reference.

7/ Equation No. 1 - please provide a valid reference. On the other hand, it seems useless to count the cardinality of the set, instead please refer to the similar studies, since you used a convenience sampling method; besides, 45 participants are enough to perform such a study as yours.

8/ But first and foremost, which attributes did you evaluate? Please specify the usability attributes and link them to a specific set of measures. Same with accessibility. In addition, please define research questions. In this view you can apply the GQM approach.

To summarize. The current version of your manuscript needs a major revision. 

Comments on the Quality of English Language

I think the language could be improved.

Author Response

For research article

Response to Reviewer X Comments
1. Summary
Thank you very much for taking the time to review this manuscript. Please find the detailed responses below and the corresponding revisions/corrections highlighted/in track changes in the re-submitted files. 2. Questions for General Evaluation Reviewer’s Evaluation Response and Revisions   Does the introduction provide sufficient background and include all relevant references? Must be improved   Dear reviewer, thank you very much for taking the time to provide your feedback. Below are the revisions/corrections corresponding to the issues listed here.   Are all the cited references relevant to the research? Must be improved     Is the research design appropriate? Must be improved     Are the methods adequately described? Must be improved     Are the results clearly presented? Must be improved     Are the conclusions supported by the results? Must be improved

3. Point-by-point response to Comments and Suggestions for Authors

The purpose of the study was to evaluate the SIGAA system in terms of accessibility and usability. While the study presents interesting results, there are some issues that should be addressed during the revision of the current version of the manuscript.

Comments 1: Correct the valid definition of usability and accessibility (see for example ISO 9241-11).

Dear Reviewer, we appreciate the revisions provided. We understand the significance of the accessibility definitions presented in ISO 9241-11. However, for this research, we believe that the definitions provided by ISO 25010 are more appropriate to our context, as it pertains to software product quality, specifically the SIGAA system.

In the revised version, we have included the definitions of usability and accessibility between lines 25 and 32, marked in blue.

Comments 2/ Correct the order of references in the content (see for example reference [28]).

The order of citations in the text follows the template provided by the journal. Thus, each citation is assigned a unique number, so if the citation is repeated elsewhere in the text, it will retain the same number.

For example, citation number 13 was first cited in line 56. It was subsequently cited again in lines 179 and 181. Since it is the same citation, the template assigns it the same number.

Regarding citation number 28, after the revision, it has been removed from the text.

Comments 3/ avoid using statements with must (for example, "the user experience must", line 138)

The suggestions to avoid the use of statements with "must" have been accepted. Therefore, the following lines have been revised (lines 14 to 19 and 520). It is important to note that, due to the revision, some lines have been moved for better coherence and cohesion of the text.

Comments 4/ avoid multiple references (for example, see the beginning of the Related Works section); in my opinion, you should remove such content, since it provides no information other than numbers in parentheses.

In response to the recommendation, the first paragraph of the Related Works section (line 101) has been removed. After revising the text, it was decided to eliminate the Background section and incorporate, in a concise manner, the key definitions into the Introduction and Methodology. These changes are highlighted in blue in the revised version of the text.

Comments 5/ Provide a summary of related work at the end of the section; this will show how your study contributes to the current body of knowledge.

Lines 464 to 486 have been added at the end of the Methodology section. Due to this addition, it was necessary to adjust the Conclusions section of the paper (lines 488 to 520), which are highlighted in blue in the revised version of the text.

Comments 6/ The reference [44] is unnecessary and not in English. I suggest deleting this reference.

The citation and corresponding reference [44] have been removed.

Comments 7/ Equation No. 1 - please provide a valid reference. On the other hand, it seems useless to count the cardinality of the set, instead please refer to the similar studies, since you used a convenience sampling method; besides, 45 participants are enough to perform such a study as yours.

In response to the insertion of the reference for Equation 1, reference and corresponding citation [31] have been added in line 234.

The need to count the cardinality of the set is justified by the choice of the study method. The method used was the confidence level of the sample, calculated by Equation 1 (lines 236 to 237). It is important to note that 45 is the total population of the study (seven students with total disability and 38 with partial disability), and to achieve a 90% confidence level, seven volunteers were required, as calculated by Equation 1 and shown in Table 1 (lines 248 to 249).

Comments 8/ But first and foremost, which attributes did you evaluate? Please specify the usability attributes and link them to a specific set of measures. Same with accessibility. In addition, please define research questions. In this view you can apply the GQM approach.

Attributes described by the W3C were evaluated (lines 281 to 288). To this end, the living lab methodology was used, with feedback from participants and observation by researchers (Table 2).

The tasks are described starting from line 267. At the end of each task, students were invited to express their emotional state (line 292) regarding the accessibility issues encountered (Table 3). This was necessary to assess the level of satisfaction and the potential for users to return to the system to complete such tasks, which are fundamental in academic life.

Regarding the online survey, it consisted of four sections described in lines 342 to 368, highlighted in blue in the revised text.

 To summarize. The current version of your manuscript needs a major revision. 

4. Response to Comments on the Quality of English Language

Point 1: In this version we paid more attention to the English language, I hope it was enough.

5. Additional clarifications

We appreciate the feedback and have revised the manuscript accordingly. We hope to have addressed all the recommendations. We are submitting a new revised document, and for better clarity, the modifications are highlighted in blue.

Author Response File: Author Response.pdf

Reviewer 2 Report

Comments and Suggestions for Authors

please see document attached

Comments for author File: Comments.pdf

Author Response

For research article

Response to Reviewer X Comments
1. Summary

Thank you very much for taking the time to review this manuscript. Please find the detailed responses below and the corresponding revisions/corrections highlighted/in track changes in the re-submitted files.

2. Questions for General Evaluation Reviewer’s Evaluation Response and Revisions   Does the introduction provide sufficient background and include all relevant references? Must be improved   Dear reviewer, thank you very much for taking the time to provide your feedback. Below are the revisions/corrections corresponding to the issues listed here.   Are all the cited references relevant to the research? Must be improved     Is the research design appropriate? Must be improved     Are the methods adequately described? Must be improved     Are the results clearly presented? Must be improved     Are the conclusions supported by the results? Must be improved

3. Point-by-point response to Comments and Suggestions for Authors

The paper investigates the accessibility challenges of Brazil's Integrated Academic Activities Management System (SIGAA) for visually impaired users. Using the Web Content Accessibility Guidelines (WCAG) and Brazil's eMAG, the study evaluates SIGAA through a three-phased methodology: user testing with visually impaired students (Living Lab), a survey of IT professionals' knowledge and practices, and an analysis of IT curricula from 20 Brazilian universities. The findings reveal non-compliance with accessibility standards in SIGAA, gaps in professional training, limited adoption of accessibility tools, and insufficient integration of accessibility concepts in academic curricula. The study emphasizes the need for legislative enforcement, professional training, and curriculum updates to enhance accessibility.

 

  Regarding the methodology section.

 

  - Sample Size and Representation

Comments 1:  The Living Lab study involved only seven participants, which, although justified using Nielsen's heuristic, limits generalizability. Expanding the sample would strengthen the validity of the findings. The authors note that "According to Nielsen and Landauer 334 (1993) [45], 85% of usability problems can be detected with five users." Nielsen refers to heuristic evaluations, which HCI specialists perform. However, end users perform this example.

Dear reviewer, we appreciate your comments regarding the text. In order to address your suggestions, the reference and subsequent citation of Nielson Landauer were removed.

Indeed, this reference was incorrect, as the study was based on a confidence level of 90% (Equation 1, Table 2); therefore, to meet this requirement, seven users were necessary. We have added an explanation in lines 246 to 248 and in line 257.

Thus, although the sample size limits the generalization of the results, with a confidence level of 90%, it can be inferred that the system's accessibility evaluation is valid.

Comments 2: The survey of IT professionals needs details on demographic diversity, such as regional or institutional representation, which could add depth to the analysis.

Lines 343 to 368 have been added to provide greater clarity regarding the questions asked in the online survey. Lines 347 to 351 present the questions related to the demographic profile. For this study, it was deemed unnecessary to inquire about data related to the region or institution of the respondents, as the accessibility legislation is of a national scope.

 - Task Design for Usability Testing

Comments 3: Tasks performed by visually impaired users lack detailed explanations about measuring success or failure (e.g., time to complete tasks, error rates, qualitative feedback). Incorporating specific metrics would improve replicability and clarity.

In accordance with the suggestions, lines 277 to 288, highlighted in blue, have been revised to provide a clearer explanation of the measurement of task success or failure. The qualitative feedback is described in lines 264 to 266. The users' emotional state, regarding the success or failure of each task, is outlined in lines 292 to 299.

- Quantitative Analysis

Comments 4: The study uses percentages in surveys but lacks inferential statistical análisis (e.g., Chi-square tests) to determine the significance of observed trends.

The online survey did not aim to correlate categorical data; therefore, there was no need to conduct chi-square tests. The objective of the survey was to understand the profile of IT professionals working with web-based systems, without the intention of correlating the data for any type of inference. The use of percentages was solely employed to identify the predominant profile of the respondents.

Regarding results section.

- Validation of Findings

Comments 5: The study concludes gaps in training and system design but does not compare SIGAA's compliance with other systems, which could contextualize its shortcomings.

The study presents an analysis of a Management System widely used in Brazil across 39 public institutions. This system, like all systems used in public entities, must comply with legal regulations. Therefore, if the analyzed system reveals gaps that hinder the academic experience of individuals with visual impairments, it inherently highlights the lack of accessibility in the country.

Comments 6: Emotional responses during the Living Lab are cataloged but must be deeply analyzed regarding system features. A thematic analysis of participants' qualitative feedback could provide actionable insights.

The analysis of feedback could provide additional insights; however, for this research, the goal was to broadly understand the accessibility challenges in an academic management system in Brazil. It is important to emphasize that, as this work does not aim to exhaust the issue but rather contribute to the understanding of the current accessibility landscape, future studies will require a more in-depth analysis to address more specific issues.

- Accessibility Tools:

Comments 7 : The study does not specify which accessibility evaluation tools were used or their limitations. Including this would enhance technical rigor.

Lines 266 to 268 present the tools used during the Living Lab. It is important to note that, during the tests, the evaluators aimed to recreate a more authentic experience of the participants' daily lives. Thus, they allowed the participants to choose the accessibility mechanisms they typically use in their academic activities. In this way, the goal was to minimize errors arising from the learning curve associated with new accessibility technologies.

 2 - Curriculum Analysis

 Comments 8: The analysis of university curricula is limited to keyword searches. A broader review of course content and pedagogical approaches to accessibility is recommended.

The analysis of curricula was limited to keywords in order to obtain an overview of whether or not the topic of accessibility is addressed in the courses for training IT professionals. It should be highlighted that, for a change in pedagogical projects, the participation of specialists would be necessary to discuss and incorporate these topics into the course content. It is emphasized that this work does not aim to exhaust the topic in its entirety but rather to contribute to research related to these issues.

. Observation about structure and order

- Clarity in Section Transitions

Comments 9: The methodology and results sections could be better aligned by mapping each methodological phase to its corresponding results.

To address this item, lines 464 to 486 have been added. For better flow of the text, the Background section was removed, and the most relevant themes were incorporated into the Introduction and Methodology (highlighted in blue). The Conclusion section has been restructured.

- Presentation of Data

Comments 10: Tables (e.g., task success rates and emotional responses) are included but lack descriptive legends and alignment with discussion points. Each table should be directly referenced and interpreted within the text.

The textual descriptions of Tables 1, 2, and 3, as well as their captions, have been added. Table 1 is described in lines 246 to 248, Table 2 in lines 289 to 291, and Table 3 in lines 301 to 303. Table 4 already had a description in the text.

- Background and Related Works

Comments 11: The literature review overlaps with the introduction and background. Streamlining these sections could reduce redundancy and improve focus.

To address the comments, the Background section has been removed, and the relevant concepts have been integrated throughout the text (Introduction and Methodology).

4. Response to Comments on the Quality of English Language

Point 1: In this version we paid more attention to the English language, I hope it was enough.

5. Additional clarifications

We appreciate the feedback and have revised the manuscript accordingly. We hope to have addressed all the recommendations. We are submitting a new revised document, and for better clarity, the modifications are highlighted in blue.

 

 

 

Author Response File: Author Response.pdf

Round 2

Reviewer 1 Report

Comments and Suggestions for Authors

Dear Authors.

Thank you for addressing my concerns.

 

Reviewer 2 Report

Comments and Suggestions for Authors

The authors provided detailed clarifications addressing each of the observations outlined above. Their responses demonstrate engagement with the feedback, resulting in relevant improvements to the manuscript. Considering these revisions, the article is more coherent and methodologically robust, with enhanced consistency across its key arguments and supporting evidence.