ColorPoetry: Multi-Sensory Experience of Color with Poetry in Visual Arts Appreciation of Persons with Visual Impairment
Round 1
Reviewer 1 Report
The paper presents an interesting idea that consists in coding color attributes into poems and sound.
Nevertheless, my main concern is that I have found the paper really difficult to follow and read.
Some comments:
- The tutorial content is good, but it distracts if it is not close to your proposal. I think you should write a shorter introduction to justify your proposal. You should not repeat content that has been previously published, as that in your paper [17] unless it is needed to understand your proposal.
- The paper is too verbose and the style is not good. The order in which the information is presented is not clear.
- Your proposal consists in choosing a set of poems and analyze how close they are felt to a given artwork by the participants in the experiments. You say that the participants are asked to experience multiple sensations like vision, sound, haptic and scent. Moreover, table 7 and 8 show labels such as “haptic” or “vision”, but your experiments seem to be related only to poems that are read with different voices (pitch, speed). This is really confusing.
- In the lines 464 to 473 you describe the procedure to compare the perception of the artwork and the poems. Are the poems those in Table 4 and 5?
- I do not understand how you obtain the numbers in the table 9 and 10. I do not understand either the representation in colored bars in the same figure. Please explain how you aggregate the answers from the participants. Please explain how you build the vectors to compare (provide clear examples). On the contrary, I do not see necessary you provide the equation of the cosine similarity. What is the meaning and usefulness of the last column “Sum” in tables 9 and 10?
- I guess the information in Table 10 is the same as that in Table 9 but given in a different way, I do not see it very useful.
- Regarding to the question explained in the lines 475 to 479, I do not see it clearly, but the results of the question seem to be in the lines 516 to 529. Please explain it in different paragraphs or even sections.
- You present a methodology and you have performed some tests with participants, I do not see why you present results from a “system usability test”. What system the participants are talking about? (“the music system was new and interesting…” ?)
- In the “Conclusion” section you highlight a secondary result of your study which is that from the usability test and also mention a result from [17]. You do not provide a clear conclusion about the core of your proposal and main tests described in section 4.2
Author Response
We would like to take this opportunity to express our thanks to reviewer for insightful comments. The manuscript has been revised to address all the reviewer comments. As with the parts painted in blue, the contents and organization in this revision were rewritten and reworked as a whole.
Please see the attached file for the responses.
Author Response File: 
Author Response.pdf
Reviewer 2 Report
I appreciate the efforts put into this manuscript. The main problem that I find with this research is the scale of the experiments. The number of users 15, consisting of 5 male and 10 female is too limited to make any concrete conclusions. Thus, I am a bit skeptic if the numbers provided as significance analysis really makes sense.
Author Response
We would like to take this opportunity to express our thanks to reviewer for insightful comments. The manuscript has been revised to address all the reviewer comments. As with the parts painted in blue, the contents and organization in this revision were rewritten and reworked as a whole.
Please see the attached file for the responses.
Author Response File: Author Response.pdf
Reviewer 3 Report
Este trabalho tem um elevado interesse para a comunidade científica. É assinalável o tratamento dado a um tema que se preocupa com a resolução de problemas para uma comunidade específica, com limitações de natureza visual, partindo de uma abordagem multisensorial que envolve música, pintura, poesia e cor que definem a matriz de uma terapia polinucleada mas relacional e interactiva. O estudo está muito bem desenvolvido ainda que apresente alguns aspectos a corrigir, principalmente, de natureza, formal e de formatação. Os conetúdos podem ser pontualmente enriquecidos com algumas sugestões que estão aporesentadas ao longo da revisão. Recomendo, com grande convicção, que este trabalho seja publicado. Foi um grande prazer tomar contacto com este artigo na qualidade de revisor. Parabéns aos autores e felicidades. Mas, também, a MDPI está de parabéns, em particular a revista Electronics.
Comments for author File: Comments.pdf
Author Response
We would like to take this opportunity to express our thanks to reviewer for insightful comments. The manuscript has been revised to address all the reviewer comments. As with the parts painted in blue, the contents and organization in this revision were rewritten and reworked as a whole.
Please see the attached file for the responses.
Author Response File: Author Response.pdf
Round 2
Reviewer 2 Report
The number of users has been a subject of debate and the golden rule of 5 users is quite old. Nevertheless, I am afraid many factors may influence the optimum number of users! I am not yet convinced that 5 users is the answer in this case, e.g., in some eye-tracking based tasks at least 25 users are suggested! https://people.csail.mit.edu/tjudd/TJuddPhDThesis.pdf
And also, https://link.springer.com/chapter/10.1007/978-3-030-25629-6_22 shows depending on a task 9 subjects may be needed.
Author Response
Reviewer 2
The number of users has been a subject of debate and the golden rule of 5 users is quite old. Nevertheless, I am afraid many factors may influence the optimum number of users! I am not yet convinced that 5 users is the answer in this case, e.g., in some eye-tracking based tasks at least 25 users are suggested! https://people.csail.mit.edu/tjudd/TJuddPhDThesis.pdf
And also, https://link.springer.com/chapter/10.1007/978-3-030-25629-6_22 shows depending on a task 9 subjects may be needed.
We would like to take this opportunity to express our thanks to reviewer for insightful comments.
We further investigated on the number of users required for the usability testing and added the following to the revision.
Line 702:
Even though the magic number 5 rule (Nielsen & Landauer, 1993 [52]) is vastly known and used for usability testing, the sample size is a long-running debate. For example, Molich, Ede, Kaasgaard and Karyukin (2004) [53], on the other hand, find no relation at all between the number of test users and the number of detected usability problems. The meta-analysis by Hwang and Salvendy (2010) shows that 9 users are often required to detect 80% of the problems. Lamontagne et. al (2019) [54] investigated how many users are needed in usability testing to identify negative phenomena caused by a combination of the user interface and the usage context. They focused on identifying psychophysiological pain points (i.e., emotionally irritant experienced by the users) during a human-computer interaction. Fifteen subjects were tested in a new user training context and results show that out of the total psychophysiological pain points experienced by 15 participants, 82% of them were experienced with nine participants.
Therefore, as a future work, we will further perform scaled experiments on people with visual impairment as a future work, along with experiments to find significant differences in perception of the various levels of the visually impaired for the proposed solution.
- Nielsen, J., & Landauer, T. K. (1993, May). A mathematical model of the finding of usability problems. In Proceedings of the INTERACT'93 and CHI'93 conference on Human factors in computing systems(pp. 206-213).
- Molich, R., Ede, M. R., Kaasgaard, K., & Karyukin, B. (2004). Comparative usability evaluation. Behaviour & Information Technology, 23(1), 65-74.
- Hwang, W., & Salvendy, G. (2010). Number of people required for usability evaluation: the 10±2 rule. Communications of the ACM, 53(5), 130-133.
- Lamontagne, C., Sénécal, S., Fredette, M., Chen, S. L., Pourchon, R., Gaumont, Y., ... & Léger, P. M. (2019). User Test: How Many Users Are Needed to Find the Psychophysiological Pain Points in a Journey Map? In International Conference on Human Interaction and Emerging Technologies(pp. 136-142). Springer, Cham.
In addition, we reorganized the sections as follows.
Line 473:
3.7. Implicit association test to find a solution to the problem of ColorPoetry
The purpose of this test is to find a solution to the problem defined in section 3.2 for the instance of Figure 6. To do so, an implicit association test was performed to identify the intimacy of the color stimulus in the poems (Table 6) and the one in the artwork (“The starry night” by Van Gogh) through the intervention of a semantic differential adjective antonym in Table 4 and Table 5.
Line 605:
- Usability Test and Result
In the usability evaluation experiment, SUS (System Usability Score) test was executed. During the SUS test, our proposed method “ColorPoetry” in this paper was compared with “ColorSound [22]”. Fifteen participants who also attended the implicit association test described in Section 3 were asked the following question.
Author Response File: Author Response.pdf
Reviewer 3 Report
The work is now much more coherent and complete. The authors' welcome to some of the observations I made in the first review gave more consistency to a study that was already very interesting and very good. It got even better.
Comments for author File: Comments.pdf
Author Response
We would like to take this opportunity to express our thanks to reviewer for insightful comments and observation the reviewer made in the first review. The paper was improved a lot with the help of the friendly review of the reviewer.
Author Response File: Author Response.pdf
