How to Improve Usability in Open-Source Software Projects? An Analysis of Evaluation Techniques Through a Multiple Case Study
Round 1
Reviewer 1 Report
Comments and Suggestions for AuthorsI have reviewed the paper “How to Improve Usability in Open-Source Software Projects? An Analysis of Evaluation Techniques through a Multiple Case Study” and there are some issues and areas for improvement.
Lack of clarity. The abstract section indicates ”we developed a framework for integrating usability evaluation into OSS projects”. In Section 7 Framework – The authors propose a framework based on their volunteer experience in the usability evaluation of different OSS projects [4, 11]. References 4 and 11 are previous work (self-citations) – Table 5 summarizes the phases of the proposed framework for evaluating the usability of OSS projects, but there are not enough details or validation to ensure the framework is reusable for different techniques or projects.
The authors determine how to incorporate three usability techniques (Cognitive walkthrough - CW, formal usability testing - FUT, and thinking aloud - TA) into the development process of three Open Source Software (OSS) projects. It would have been interesting to apply the same technique to all the projects, which would have provided better insight on how to use the framework.
Repetitive text. There are quite repetitive phases, providing multiple explanations of the three usability techniques (CW, FUT, TA) For example lines 50 to 52, 88 to 90, 108, 120 and 126, and more lines as 373-374 and 381 -382, Line 397 “Our research addresses adapting usability techniques, namely CW, FUT, and TA, for integration into the OSS development process.”
Lines 799 – 801.
Lines 701 - 703 and 394, 766 “We use web artifacts, virtual meetings, online surveys, forums, and questionnaires to adjust each.”
The paper briefly mentions “the need arises to evaluate OSS projects by applying different techniques, among which stand out: Cognitive Walkthrough, Formal Usability Testing, and Thinking Aloud. “ but it doesn´t provide a related work about a detailed problem. Section 2.4 could help readers understand the research objectives if moved to the introduction section.
Move the text in line 463 “we adapted the CW to facilitate its incorporation in the selected OSS project.” before mentioning Table 1.
Move the text in line 557 “In this research work, we adapted the FUT to allow its incorporation into the selected OSS project.“ before mentioning Table 2.
In section 4.2, list the 14 steps of the FUT method to be consistent with how steps were listed in sections 4.1 and 4.3
Only two problems were identified into https://goo.su/HBFS9l and one of them is not a problem
https://docs.google.com/document/d/1-kpl07iR-i7DVUAqcLL4FGWtn9-yGOH5/edit
https://goo.su/ZEPHyG and https://n9.cl/b99kf have the same information
5.3 Data analysis
“we proposed several tables and forms to record and present the in formation obtained in each case study.”
The case study was carried out with few users/participants so the tables in doc format were sufficient. However, if it were applied to a larger number of users (geographical distributed worldwide), the data collection and analysis techniques would not be adequate.
The case study deviated from the objectives by not being applied to developer users in the open source communities, but to students who acted as non-expert users in the use of the three projects. It is suggested that the objectives be reconsidered for the viability of the proposed framework.
Author Response
Responses to comments are in the attached file.
Author Response File: 
Author Response.pdf
Reviewer 2 Report
Comments and Suggestions for AuthorsThis article focuses on the application of usability evaluation techniques on open-source software and the challenges presented in this type of development paradigm. Through the incorporation of specific usability testing techniques and their adaptation to OSS peculiarities, the study proposes a usability evaluation framework suitable for the OSS development process.
The article is well written and puts a lot of effort into describing the methodological approach and the author’s intent and considerations during its various steps. Making usability evaluation easier for OSS projects is important since they are both essential aspects of modern software development.
That being said, the article has an issue with verbosity and length which may hinder its readability. In order to add value to the article the authors should consider greatly reducing its length. Special care should be given to avoid repeating information and details that aren’t directly relevant to the study’s innovative aspects should be omitted or described in fewer details.
More specifically, the introduction section already discusses the research background for the article to a satisfactory extent, so section 2 which details the contents of various different works, although interesting, could have been shorter. The purpose of the article is to showcase the new work and not to also act as a detailed literature review. Reducing the length of section 2 might be a good idea.
Moreover, in section 4, where the adaptations of the various evaluation methods are presented, the details are very verbose, and the changes made are very similar in every case. You should consider approaching the adaptations as a whole instead of describing them repeatedly for every evaluation method, thus greatly reducing the length of section 4 and merging it with section 3.
Additionally, the three evaluation methods should only be very briefly described in the context of this research and not given full detailed descriptions since this is not an essential part of this research’s innovative aspect.
In a similar vein, Section 5 repeats a lot of information already established in the introduction or other earlier sections.
Finally, Section 6 should focus more on the actual findings and less on the procedure details.
You might also want to consider providing the implementation materials (forms, invitations, results etc) that you have linked online as supplementary materials in the article’s data availability statement or in an appendix, instead of in the article’s main text. Please read the “Supplementary Materials, Data Deposit and Software Source Code” on MDPI’s instructions for authors.
Overall, the work has value, but it needs to be condensed to its important and innovative aspects.
The level of English in this study is very high and only minor proofreading might be required.
Author Response
Responses to comments are in the attached file.
Author Response File: Author Response.pdf
Round 2
Reviewer 1 Report
Comments and Suggestions for AuthorsThe article and the responses to the 11 comments have been reviewed and addressed, but the authors uploaded the editable file with the tracking of the changes to the platform. Crossed-out texts can be seen and are sometimes difficult to read. For this reason, the page numbers do not correspond to those specified in the responses.
Reviewer 2 Report
Comments and Suggestions for AuthorsThe majority of my concerns has been addressed and due to your work on reducing and focusing the text the overall value of the paper has been made clearer. The paper still remains lengthy, but I feel the matters discussed would be of interest to the community and will allow HCI to become more integral in OSS development and as such I recommend the publication of this work.
