Multi-Sensory Color Expression with Sound and Temperature in Visual Arts Appreciation for People with Visual Impairment
2. Background and Related Works
2.1. Multi-Sensory Systems
2.2. Multi-Sensory Art
2.3. Multi-Sensory Assistive Devices
2.4. Sound-Color Cross-Modality
2.5. Temperature-Color Cross-Modality
3.1. Previous Method
3.2. Multi-Sensory Improved Method
- Finding out whether it was better to represent color hue with sounds and the color dimensions with temperature, or vice versa.
- Knowing which one was those warmest and coldest colors (that appear in Table 5) from the sound set and the temperature set. In other words, setting the dark dimension as an example, with many sounds representing the quality of “dark” such as “dark green” or “dark red”, it was important to figure out which one of all those dark color-sounds expressed the dark quality of color the best.
- Finding out which sound set from the two options (VIVALDI and CLASSIC) is in general terms more suitable to express the different dimensions of color.
- Introducing the context of the research, explaining about visually impaired people and art exploration, color, and multi-sensory systems.
- Introduction of the Petlier thermal display prototype (only during the third test).
- Lastly, the tester interacted with the different audio or temperature cues and was given the survey questions. Each audio cue was heard as many times as the user needed (the user was given the playback control for repeating it until he/she was ready to answer the survey question related to that cue). Similarly, the user was able to feel the different temperatures as many times as desired before answering to the survey question related to that temperature cue.
4.1. First Test: Classifying Pair of Adjectives
4.2. Second Test: Calculating Weights of Each Pair of Adjectives for Each Dimension
4.3. Final Test: Linking Adjectives to Each Modality Cue
5. Analysis and Results
- The CLASSIC sound set’s cue which best expressed the bright and warm dimensions of color was the same sound. This was not convenient since users would not be able to discern whether the sound was expressing brightness or warmness. The cues for representing each dimension had be all different from each other so the user could relate each particular cue to a particular dimension.
- Similarly, the highest temperature of 38 °C is the best one for representing both the warmness and brightness of the color. Additionally, the coldest temperature of 14 °C was the best temperature cue for conveying darkness and coldness. Like with the CLASSIC sound set, this reiteration of cues made temperature modality not suitable for representing the color dimensions.
- On the contrary, none of the VIVALDI sounds is repeated. Each dimension is conveyed best by a different musical sound cue.
- Except for conveying brightness, in which CLASSIC sound reached a higher score, VIVALDI sound reached higher scores for all the other three dimensions when compared to both the CLASSIC and temperature sets.
5.3. Final Temperature–Sound Coding Multimodal Test and Results
- Introducing the context of the research, the system and the method (5 min).
- Explanation of the 10 different cues and training time for both the six temperature cues and the four sound cues (10 min)
- After that the users were tested on 10 colors. The users were given the multimodal feedback related to each color and were asked to find out which color it represented. In other words, after feeling both the temperature cue and hearing sound at the same time, they were asked about the color. The users were given a printed page similar to Table 16 so they did not need to memorize what each temperature represented. All the users were given the same questions in the same order.
7. Future Work
- The problem for differentiating the colors yellow and orange need to be addressed, since it is the main bottleneck for reaching high accuracy on the system.
- Installing the multi-sensory system and using its method for performing complete on-field user tests with different artworks and on different exhibitions for the visually impaired people. This could be the beginning of research about the effect of multi-sensory systems on art engagement and artwork memorization by the visually impaired people.
- Finding a way to increase the number of colors that can be expressed by the system, without increasing its complexity or making it less engaging.
- Finding other applications based on the same sound-temperature concept or expressing other features of the artworks.
Institutional Review Board Statement
Informed Consent Statement
Conflicts of Interest
- Shams, L.; Seitz, A.R. Benefits of multisensory learning. Trends Cogn. Sci. 2008, 12, 411–417. [Google Scholar] [CrossRef] [PubMed]
- The Guggenheim Museums and Foundation. Mind’s Eye Programs. Available online: https://www.guggenheim.org/event/event_series/minds-eye (accessed on 28 February 2021).
- The Metropolitan Museum of Art. For Visitors Who Are Blind or Partially Sighted. Available online: https://www.metmuseum.org/events/programs/access/visitors-who-are-blind-or-partially-sighted (accessed on 28 February 2021).
- Museum of Moden Art. Accessibility. Available online: https://www.moma.org/visit/accessibility/index#individuals-who-are-blind-or-have-low-vision (accessed on 28 February 2021).
- Bartolome, J.D.I.; Quero, L.C.; Cho, J.; Jo, S. Exploring Thermal Interaction for Visual Art Color Appreciation for the Visually Impaired People. In Proceedings of the 2020 International Conference on Electronics, Information, and Communication (ICEIC), Barcelona, Spain, 19–22 January 2020; pp. 1–5. [Google Scholar]
- Cho, J.D.; Jeong, J.; Kim, J.H.; Lee, H. Sound Coding Color to Improve Artwork Appreciation by People with Visual Impairments. Electronics 2020, 9, 1981. [Google Scholar] [CrossRef]
- Bartolomé, J.I.; Cho, J.D.; Quero, L.C.; Jo, S.; Cho, G. Thermal Interaction for Improving Tactile Artwork Depth and Color-Depth Appreciation for Visually Impaired People. Electronics 2020, 9, 1939. [Google Scholar] [CrossRef]
- Lloyd-Esenkaya, T.; Lloyd-Esenkaya, V.; O’Neill, E.; Proulx, M.J. Multisensory inclusive design with sensory substitution. Cogn. Res. Princ. Implic. 2020, 5, 1–15. [Google Scholar] [CrossRef]
- Faustino, D.B.; Gabriele, S.; Ibrahim, R.; Theus, A.-L.; Girouard, A. SensArt Demo: A Multi-sensory Prototype for Engaging with Visual Art. In Proceedings of the 2017 ACM International Conference on Interactive Surfaces and Spaces, New York, NY, USA, 17 October 2017; pp. 462–465. [Google Scholar] [CrossRef]
- Pursey, T.; Lomas, D. Tate Sensorium: An experiment in multisensory immersive design. Senses Soc. 2018, 13, 354–366. [Google Scholar] [CrossRef]
- Baumgartner, T.; Lutz, K.; Schmidt, C.F.; Jäncke, L. The emotional power of music: How music enhances the feeling of affective pictures. Brain Res. 2006, 1075, 151–164. [Google Scholar] [CrossRef] [PubMed]
- Edirisinghe, C.; Podari, N.; Cheok, A.D. A multi-sensory interactive reading experience for visually impaired children; a user evaluation. Pers. Ubiquitous Comput. 2018, 1–13. [Google Scholar] [CrossRef]
- Brule, E.; Bailly, G.; Brock, A.; Valentin, F.; Denis, G.; Jouffrais, C. MapSense: Multi-Sensory Interactive Maps for Children Living with Visual Impairments. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, San Jose, CA, USA, 7–12 May 2016; pp. 445–457. [Google Scholar]
- Feeling Van Gogh, Van Gogh Museum, Amsterdam. Feel, Smell and Listen to the Sunflowers. Available online: https://www.vangoghmuseum.nl/en/visit/whats-on/feeling-van-gogh (accessed on 28 February 2021).
- Ibraheem, N.A.; Hasan, M.A.; Khan, R.Z.; Mishra, P.K. Understanding color models: A review. ARPN J. Sci. Technol. 2012, 2, 265–275. [Google Scholar]
- Bailey, R.J.; Grimm, C.M.; Davoli, C. The Real Effect of Warm-Cool Colors. Report Number: WUCSE-2006-17. All Computer Science and Engineering Research. 2006. Available online: https://openscholarship.wustl.edu/cse_research/166 (accessed on 31 May 2021).
- Wang, Q.; Spence, C. The Role of Pitch and Tempo in Sound-Temperature Crossmodal Correspondences. Multisens. Res. 2017, 30, 307–320. [Google Scholar] [CrossRef][Green Version]
- Levitan, C.A.; Charney, S.; Schloss, K.B.; Palmer, S.E. The Smell of Jazz: Crossmodal Correspondences Between Music, Odor, and Emotion. In Proceedings of the CogSci, Pasadena, CA, USA, 22–25 July 2015; pp. 1326–1331. [Google Scholar]
- Hamilton-Fletcher, G.; Obrist, M.; Watten, P.; Mengucci, M.; Ward, J. I Always Wanted to See the Night Sky. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems; Association for Computing Machinery (ACM), San Jose, CA, USA, 7–12 May 2016; pp. 2162–2174. [Google Scholar]
- Jia, Z.; Li, J.; Chen, C. Design and Evaluation of a Thermal Tactile Display for Colour Rendering. Int. J. Adv. Robot. Syst. 2015, 12, 162. [Google Scholar] [CrossRef][Green Version]
- Palmer, S.E.; Schloss, K.B.; Xu, Z.; Prado-Leon, L.R. Music-color associations are mediated by emotion. Proc. Natl. Acad. Sci. USA 2013, 110, 8836–8841. [Google Scholar] [CrossRef] [PubMed][Green Version]
- Osgood, C.E.; Suci, G.; Tannenbaum, P.H. The Measurement of Meaning; University of Illinois Press: Urbana, IL, USA, 1957. [Google Scholar]
|Color Hues||Saturated Colors||Bright Colors||Dark Colors|
|Color Hue||Temperature (°C)|
|Color Dimension||Temperature (°C)|
|Color Hues||Saturated Color Sound||Bright/Dark|
|Color Hues||Saturated Color Temp (°C)||Bright/Dark|
VIVALDI or CLASSIC
VIVALDI or CLASSIC
|Calm~Tense||Pointed (Kiki)~Round (Bouba)|
|Warm~Cold||High~Low (e.g., high-pitch~low-pitch)||Active~Inactive|
|Noisy – Quiet||−0.750|
|Cloudy – Clear||1.083|
|Joy – Depressed||−1.333|
|Round – Pointed||0.667|
|High – Low||−1.167|
|Slow – Agile||0.917|
|Rough – Smooth||−0.083|
|Hard – Soft||0.417|
|Strong – Weak||−0.917|
|Active – Inactive||−1.333|
|Calm – Tense||0.167|
|Near – Far||−0.750|
|Introverted – Extroverted||0.833|
|Passionate – Apathetic||−1.167|
|Centritorial – Centrifugal||0.333|
|Adjective Pairs||L/W/C/D Weights|
|VIVALDI||47.361 (RED-L)||49.479 (Orange-D)||28.247 (Green-S)||15.139 (Purple-D)|
|CLASSIC||54.323 (PURPLE-L)||39.063 (Red-D)||26.701 (Purple-L)||9.097 (Orange-D)|
|TEMP||43.935 (38 °C)||34.190 (14 °C)||28.056 (38 °C)||11.698 (14 °C)|
|Color Hues||Saturated Color Temp (°C)||Bright||Dark||Warm||Cold|
|Percentage of Correct Answers|
|Total||Only Hue||Only Dimension|
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.
© 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Bartolome, J.I.; Cho, G.; Cho, J.-D. Multi-Sensory Color Expression with Sound and Temperature in Visual Arts Appreciation for People with Visual Impairment. Electronics 2021, 10, 1336. https://doi.org/10.3390/electronics10111336
Bartolome JI, Cho G, Cho J-D. Multi-Sensory Color Expression with Sound and Temperature in Visual Arts Appreciation for People with Visual Impairment. Electronics. 2021; 10(11):1336. https://doi.org/10.3390/electronics10111336Chicago/Turabian Style
Bartolome, Jorge Iranzo, Gilsang Cho, and Jun-Dong Cho. 2021. "Multi-Sensory Color Expression with Sound and Temperature in Visual Arts Appreciation for People with Visual Impairment" Electronics 10, no. 11: 1336. https://doi.org/10.3390/electronics10111336