The Value of Open Data in HCI: A Case Report from Mobile Text Entry Research
Round 1
Reviewer 1 Report
This paper provides an interesting and valuable account of replication that relates directly to research in the HCI community. The author is very thorough in the presentation of the motivation, method, and findings. I believe that the paper and its discussion can spark valuable conversations and initiatives in the HCI community. However, I also perceive a few issues that should be addressed before accepting the paper:
- Related Work: Section 2.2 could profit from a number of references that relate to the core of HCI and OS/replication, including references to
research on incentives (https://dl.acm.org/doi/abs/10.1145/3415212),
research on sharing differences that result from different data practices (https://dl.acm.org/doi/10.1145/1958824.1958906),
discussions around tailored science reward mechanisms discussed also in the context of HCI conferences (https://dl.acm.org/doi/10.1145/3461778.3462067)
the role of replication in HCI (https://dl.acm.org/doi/10.1145/3290607.3312905)
- Section 4: This section is very long and because of this difficult to follow at times. I suggest that the author separates this into something like "engagement with open data" and "replication"
- I have issues with the statement "treat datasets as guility until proven innocent" as this might rather increase researchers' concerns for sharing data, rather than motivating them to share and even share their possible (and honest) mistakes. Spinning this into a more positive direction like "encouraging replications that impact trust" would make more sense for me
- I suggest that the author clearly states in the abstract and Introduction that no additional new data were collected for this replication attempt, but rather the original data from a study were solely used to inform the findings. I originally assumed that the findings would be contrasted with findings from new data.
- Line 204: missing reference
Author Response
Thank you for you encouraging overall impression of the paper. Responses to individual comments are below:
---------
- Related Work: Section 2.2 could profit from a number of references that relate to the core of HCI and OS/replication, including references to
A> Thank you for these valuable references, they have been all added in Section 2.2.
- Section 4: This section is very long and because of this difficult to follow at times. I suggest that the author separates this into something like "engagement with open data" and "replication"
A> As suggested, the section has been split into two distinct sections.
- I have issues with the statement "treat datasets as guility until proven innocent" as this might rather increase researchers' concerns for sharing data, rather than motivating them to share and even share their possible (and honest) mistakes. Spinning this into a more positive direction like "encouraging replications that impact trust" would make more sense for me
A> I agree with this insightful recommendation. However, this is the wording used in the recommendations by Hutchinson et al., hence I feel it would be best to retain it in its original form. I have made this clearer in the preceding paragraph, which has now been restructured to reflect that section's contribution more clearly. I have also enclosed the recommendations in quotes, to make it clearer that they are the original wording.
- I suggest that the author clearly states in the abstract and Introduction that no additional new data were collected for this replication attempt, but rather the original data from a study were solely used to inform the findings. I originally assumed that the findings would be contrasted with findings from new data.
A> Thank you for this observation, I have modified the abstract and also the introduction (beginning of last paragraph) accordingly. I found the wording in the review comment to be rather eloquent so I based the changers on it, thank you and I hope this is ok!
- Line 204: missing reference
A> Thank you, this is now fixed.
Reviewer 2 Report
This paper presents a case study which describes an in-depth annalysis of a previous paper and dataset recently published by other authors as part of the proceedings of a conference. Adopting a positive attitude, the author revises all the aspects related to the quality of data, results and conclusions obtained by those authors and reports similar conclusions, although with a higher care about methodological and presentation aspects.
From this non fully satisfactory experience, the author constructs on the topic of open data applied to HCI research and presents a set of guidelines and recommendations which are very valuable for the HCI research community.
In order to be consistent with some of the proposals made in the article, I would suggest the author to include a copy of the sanitized data in the respository where the code is made publicly available under the 'Data Availability Statement'. Obviously, he has preferred to keep the reference to the original data and add his sanitizing and processing code, but in this particular case study, it could be reasonable to clearly specify the subset of sanitized data in order to allow readers to check this if needed.
Author Response
Thank you for you encouraging overall impression of the paper. Responses to individual comments are below:
---------
In order to be consistent with some of the proposals made in the article, I would suggest the author to include a copy of the sanitized data in the respository where the code is made publicly available under the 'Data Availability Statement'
A> Thank you for this recommendation, I can see why this might be valuable to some researchers, so I have added the sanitised data in the repository.
Reviewer 3 Report
The paper analyses the problem of open data for research in the Human-Computer Interaction field. Although many data sources are available, they often remain without any support from the authors once the results are published. The author of the paper addresses causes, effects and analyzes solutions to the problem. The author describes an open collection developed by other researchers in the field of mobile text input as the basis for his research. Based on it, he conducts a series of analyses, where he critically demonstrates errors or omissions in the availbable dataset.
The paper is interesting and in my opinion can be published in its present form after making the necessary editorial corrections, such as: "section ??sect:disc" (line 204), "Samsung Galaxy S2 with a screen of 1440x2560px" (lines 243, 244) - some ambiguity about mapping and resolution, this smartphone has the resolution of 480x800px, etc.
Author Response
Thank you for you encouraging overall impression of the paper. Responses to individual comments are below:
---------
The paper is interesting and in my opinion can be published in its present form after making the necessary editorial corrections, such as: "section ??sect:disc" (line 204), "Samsung Galaxy S2 with a screen of 1440x2560px" (lines 243, 244) - some ambiguity about mapping and resolution, this smartphone has the resolution of 480x800px, etc.
A> Thank you for observing these oversights, the section reference has been fixed, the device was actually an S6 (as per Jiang et al.'s paper) and this has now been corrected in my manuscript.
Round 2
Reviewer 1 Report
I thank the author for responding to the comments and presenting corresponding changes. I propose to accept the manuscript in its current form.
