Colorophone 2.0: A Wearable Color Sonification Device Generating Live Stereo-Soundscapes—Design, Implementation, and Usability Audit
Abstract
:1. Introduction
Colorophone
2. Design Considerations of SSDs Development
3. Why Color?
4. Existing Color-to-Sound SSDs
5. Color Sonification
5.1. Design Considerations Regarding Auditory Color Space
5.1.1. Psychophysics
5.1.2. Cross-Modal Correspondences
5.1.3. Number of Color Components
5.2. Color Spaces
5.2.1. RYGBW Auditory Color Space
5.2.2. Sounds Associated with Color Components
- ●
- The sounds should:
- ○
- Be pleasant for the user [25];
- ○
- Be calibrated in amplitude corresponding to the maximal color intensity to provide equal loudness for every sound component;
- ○
- Have higher difference in frequency than the critical bands to avoid masking effects [53];
- ○
- >Be preferably perceptually equally spaced in pitch [54]; and,
- ○
- Be associated with colors on the basis of chosen cross-modal correspondences (i.e., blue—low pitch, green—middle-low pitch, yellow—middle-high pitch, red—high pitch).
- ●
- White should be coded by a sound with no characteristic primary frequency such as white noise or rainfall.
5.3. Spatial Color Sonification Algorithm
5.3.1. Sound Localization
5.3.2. Zone Size Determination
6. The Colorophone 2.0 SSD
6.1. Wearable Prototypes
6.2. Processing Unit
6.3. Software
6.3.1. Image Acquisition
6.3.2. Data Processing
6.3.3. Audio Generation
6.3.4. Interface for Researchers
6.3.5. Interface for Blind Users
6.4. Applied System Settings
7. Evaluation of the System
8. Conclusions and Further Work
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Acknowledgments
Conflicts of Interest
References
- Kristjánsson, Á.; Moldoveanu, A.; Jóhannesson, Ó.I.; Balan, O.; Spagnol, S.; Valgeirsdóttir, V.V.; Unnthorsson, R. Designing Sensory-Substitution Devices: Principles, Pitfalls and Potential. RNN 2016, 34, 769–787. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Starkiewicz, W.; Kuliszewski, T. The 80-Channel Elektroftalm. In Proceedings of the International Congress on Technology and Blindness, New York, NY, USA, 1 January 1963; Volume 1, p. 157. [Google Scholar]
- Capp, M.; Picton, P. The Optophone: An Electronic Blind Aid. Eng. Sci. Educ. J. 2000, 9, 137–143. [Google Scholar] [CrossRef]
- Maidenbaum, S.; Abboud, S.; Amedi, A. Sensory Substitution: Closing the Gap between Basic Research and Widespread Practical Visual Rehabilitation. Neurosci. Biobehav. Rev. 2014, 41, 3–15. [Google Scholar] [CrossRef] [Green Version]
- Elli, G.V.; Benetti, S.; Collignon, O. Is There a Future for Sensory Substitution Outside Academic Laboratories? Multisens. Res. 2014, 27, 271–291. [Google Scholar] [CrossRef]
- Proulx, M.J.; Ptito, M.; Amedi, A. Multisensory Integration, Sensory Substitution and Visual Rehabilitation. Neurosci. Biobehav. Rev. 2014, 41, 1–2. [Google Scholar] [CrossRef]
- Reich, L.; Maidenbaum, S.; Amedi, A. The Brain as a Flexible Task Machine: Implications for Visual Rehabilitation Using Noninvasive vs. Invasive Approaches. Curr. Opin. Neurol. 2012, 25, 86–95. [Google Scholar] [CrossRef]
- Ricciardi, E.; Bonino, D.; Pellegrini, S.; Pietrini, P. Mind the Blind Brain to Understand the Sighted One! Is There a Supramodal Cortical Functional Architecture? Neurosci. Biobehav. Rev. 2014, 41, 64–77. [Google Scholar] [CrossRef] [PubMed]
- Rosenblum, L.D.; Dias, J.W.; Dorsi, J. The Supramodal Brain: Implications for Auditory Perception. J. Cogn. Psychol. 2017, 29, 65–87. [Google Scholar] [CrossRef]
- Chebat, D.-R.; Schneider, F.C.; Ptito, M. Spatial Competence and Brain Plasticity in Congenital Blindness via Sensory Substitution Devices. Front. Neurosci. 2020, 14, 1–18. [Google Scholar] [CrossRef]
- Jóhannesson, Ó.; Balan, O.; Unnthorsson, R.; Moldoveanu, A.; Kristjánsson, Á. The Sound of Vision Project: On the Feasibility of an Audio-Haptic Representation of the Environment, for the Visually Impaired. Brain Sci. 2016, 6, 20. [Google Scholar] [CrossRef] [Green Version]
- Csapó, Á.; Wersényi, G.; Nagy, H.; Stockman, T. A Survey of Assistive Technologies and Applications for Blind Users on Mobile Platforms: A Review and Foundation for Research. J. Multimodal. User Interfaces 2015, 9, 275–286. [Google Scholar] [CrossRef] [Green Version]
- Bhowmick, A.; Hazarika, S.M. An Insight into Assistive Technology for the Visually Impaired and Blind People: State-of-the-Art and Future Trends. J. Multimodal. User Interfaces 2017, 11, 149–172. [Google Scholar] [CrossRef]
- Tapu, R.; Mocanu, B.; Zaharia, T. Wearable Assistive Devices for Visually Impaired: A State of the Art Survey. Pattern Recognit. Lett. 2020, 137, 37–52. [Google Scholar] [CrossRef]
- Ancuti, C.; Ancuti, C.; Bekaert, P. ColEnViSon: Color Enhanced Visual Sonifier-A Polyphonic Audio Texture and Salient Scene Analysis; SciTePress-Science and and Technology Publications: Lisboa, Portugal, 2009; pp. 566–572. [Google Scholar]
- Payling, D.; Mills, S.; Howle, T. Hue Music-Creating Timbral Soundscapes from Coloured Pictures. In Proceedings of the 13th International Conference on Auditory Display, Montréal, QC, Canada, 26–29 June 2007; pp. 91–97. [Google Scholar]
- Polo, A.; Sevillano, X. Musical Vision: An Interactive Bio-Inspired Sonification Tool to Convert Images into Music. J. Multimodal User Interfaces 2019, 13, 231–243. [Google Scholar] [CrossRef]
- Van den Doel, K. Soundview: Sensing Color Images by Kinesthetic Audio. In Proceedings of the 2003 International Conference on Auditory Display, Boston, MA, USA, 6–9 July 2003; pp. 303–306. [Google Scholar]
- Hamilton-Fletcher, G.; Wright, T.D.; Ward, J. Cross-Modal Correspondences Enhance Performance on a Colour-to-Sound Sensory Substitution Device. Multisens. Res. 2016, 29, 337–363. [Google Scholar] [CrossRef] [PubMed]
- Abboud, S.; Hanassy, S.; Levy-Tzedek, S.; Maidenbaum, S.; Amedi, A. EyeMusic: Introducing a “Visual” Colorful Experience for the Blind Using Auditory Sensory Substitution. Restor. Neurol. Neurosci. 2014, 32, 247–257. [Google Scholar] [CrossRef] [Green Version]
- Cavaco, S.; Henriques, J.T.; Mengucci, M.; Correia, N.; Medeiros, F. Color Sonification for the Visually Impaired. Procedia Technol. 2013, 9, 1048–1057. [Google Scholar] [CrossRef] [Green Version]
- Eyeborg. 2021. Available online: https://en.wikipedia.org/wiki/Eyeborg (accessed on 9 March 2021).
- Capalbo, Z.; Glenney, B. Hearing Color: Radical Pluralistic Realism and SSDs. In Proceedings of the AP-CAP, Tokyo, Japan, 1 October 2009; pp. 135–140. [Google Scholar]
- Bologna, G.; Deville, B.; Pun, T. On the Use of the Auditory Pathway to Represent Image Scenes in Real-Time. Neurocomputing 2009, 72, 839–849. [Google Scholar] [CrossRef] [Green Version]
- Osinski, D.; Hjelme, D.R. A Sensory Substitution Device Inspired by the Human Visual System. In Proceedings of the 2018 11th International Conference on Human System Interaction (HSI), Gdansk, Poland, 4–6 July 2018; pp. 186–192. [Google Scholar]
- Colorophone. Available online: https://www.colorophone.com (accessed on 29 March 2021).
- Dakopoulos, D.; Bourbakis, N.G. Wearable Obstacle Avoidance Electronic Travel Aids for Blind: A Survey. IEEE Trans. Syst. Man Cybern. C 2010, 40, 25–35. [Google Scholar] [CrossRef]
- Jonas, J.B.; Müller-Bergh, J.A.; Schlötzer-Schrehardt, U.M.; Naumann, G.O. Histomorphometry of the Human Optic Nerve. Investig. Ophthalmol. Vis. Sci. 1990, 31, 736–744. [Google Scholar]
- Spoendlin, H.; Schrott, A. Analysis of the Human Auditory Nerve. Hear. Res. 1989, 43, 25–38. [Google Scholar] [CrossRef]
- Cohen, M.A.; Dennett, D.C.; Kanwisher, N. What Is the Bandwidth of Perceptual Experience? Trends Cogn. Sci. 2016, 20, 324–335. [Google Scholar] [CrossRef] [Green Version]
- Fairchild, M.D. Color Appearance Models. In Wiley-IS&T Series in Imaging Science and Technology, 2nd ed.; Wiley: Chichester, West Sussex, UK; Hoboken, NJ, USA, 2005; ISBN 978-0-470-01216-1. [Google Scholar]
- Gerl, E.J.; Morris, M.R. The Causes and Consequences of Color Vision. Evo. Edu. Outreach 2008, 1, 476–486. [Google Scholar] [CrossRef] [Green Version]
- Hamilton-Fletcher, G.; Ward, J. Representing Colour Through Hearing and Touch in Sensory Substitution Devices. Multisens. Res. 2013, 26, 503–532. [Google Scholar] [CrossRef]
- Kremers, J.; Baraas, R.C.; Marshall, N.J. (Eds.) Human Color Vision; Springer International Publishing: Cham, Switzerland, 2016; ISBN 978-3-319-44976-0. [Google Scholar]
- Torralba, A. How Many Pixels Make an Image? Vis. Neurosci. 2009, 26, 123–131. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Meijer, P.B.L. An Experimental System for Auditory Image Representations. IEEE Trans. Biomed. Eng. 1992, 39, 112–121. [Google Scholar] [CrossRef] [PubMed]
- Strumillo, P. Electronic Interfaces Aiding the Visually Impaired in Environmental Access, Mobility and Navigation. In Proceedings of the 3rd International Conference on Human System Interaction, Rzeszow, Poland, 13–15 May 2010; pp. 17–24. [Google Scholar]
- Bujacz, M. Representing 3D Scenes through Spatial Audio in an Electronic Travel Aid for the Blind. Ph.D. Thesis, Lodz University of Technology (Politechnika Łódzka), Łódź, Poland, 2010; p. 978. [Google Scholar]
- Rodríguez, A.; Yebes, J.J.; Alcantarilla, P.F.; Bergasa, L.M.; Almazán, J.; Cela, A. Assisting the Visually Impaired: Obstacle Detection and Warning System by Acoustic Feedback. Sensors 2012, 12, 17476–17496. [Google Scholar] [CrossRef] [Green Version]
- Ribeiro, F.; Florêncio, D.; Chou, P.A.; Zhang, Z. Auditory Augmented Reality: Object Sonification for the Visually Impaired. In Proceedings of the 2012 IEEE 14th International Workshop on Multimedia Signal Processing (MMSP), Banff, AB, Canada, 17–19 September 2012; pp. 319–324. [Google Scholar]
- Balakrishnan, G.; Sainarayanan, G.; Nagarajan, R.; Yaacob, S. A Stereo Image Processing System for Visually Impaired. Int. J. Comput. Inf. Eng. 2008, 2, 2794–2803. [Google Scholar]
- Saez, J.M.; Escolano, F.; Lozano, M.A. Aerial Obstacle Detection with 3-D Mobile Devices. IEEE J. Biomed. Health Inform. 2015, 19, 74–80. [Google Scholar] [CrossRef]
- Caraiman, S.; Zvoristeanu, O.; Burlacu, A.; Herghelegiu, P. Stereo Vision Based Sensory Substitution for the Visually Impaired. Sensors 2019, 19, 2771. [Google Scholar] [CrossRef] [Green Version]
- Bujacz, M.; Strumiłło, P. Sonification: Review of Auditory Display Solutions in Electronic Travel Aids for the Blind. Arch. Acoust. 2016, 41, 401–414. [Google Scholar] [CrossRef]
- Osinski, D.; Bizon, P.; Midtfjord, H.; Wierzchon, M.; Hjelme, D.R. Designing Auditory Color Space for Color Sonification Systems. In Proceedings of the Proceedings of the 25th International Conference on Auditory Display (ICAD 2019), Newcastle upon Tyne, UK, 23–27 June 2019; pp. 331–334. [Google Scholar]
- Bologna, G.; Deville, B.; Pun, T. Sonification of Color and Depth in a Mobility Aid for Blind People. In Proceedings of the Proceedings of the 16th International Conference on Auditory Display (ICAD2010), Washington, DC, USA, 9–15 June 2010; p. 6. [Google Scholar]
- Stevens, S.S. On the Psychological Law. Psychol. Rev. 1957, 64, 153–181. [Google Scholar] [CrossRef]
- Spence, C. Crossmodal Correspondences: A Tutorial Review. Atten. Percept. Psychophys 2011, 73, 971–995. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Witzel, C.; Gegenfurtner, K.R. Are Red, Yellow, Green, and Blue Perceptual Categories? Vis. Res. 2018, 151, 152–163. [Google Scholar] [CrossRef] [PubMed]
- Moroney, N. The Opposite of Green Is Purple? SPIE: San Jose, CA, USA, 2009; p. 72410N. [Google Scholar]
- NCS Colour-Universal Language for Colour Commmunication. Available online: https://ncscolour.com/ (accessed on 6 March 2021).
- Kuehni, R.G. Focal Colors and Unique Hues. Color Res. Appl. 2001, 26, 171–172. [Google Scholar] [CrossRef]
- Fastl, H.; Zwicker, E. Psychoacoustics: Facts and Models; Springer: New York, NY, USA, 2007; ISBN 978-3-540-68888-4. [Google Scholar]
- Stevens, S.S.; Volkmann, J.; Newman, E.B. A Scale for the Measurement of the Psychological Magnitude Pitch. J. Acoust. Soc. Am. 1937, 8, 185–190. [Google Scholar] [CrossRef]
- Brown, D.J.; Simpson, A.J.R.; Proulx, M.J. Auditory Scene Analysis and Sonified Visual Images. Does Consonance Negatively Impact on Object Formation When Using Complex Sonified Stimuli? Front. Psychol. 2015, 6, 1–9. [Google Scholar] [CrossRef] [Green Version]
- Mills, A.W. On the Minimum Audible Angle. J. Acoust. Soc. Am. 1958, 30, 237–246. [Google Scholar] [CrossRef]
- Perrott, D.R.; Tucker, J. Minimum Audible Movement Angle as a Function of Signal Frequency and the Velocity of the Source. J. Acoust. Soc. Am. 1988, 83, 1522–1527. [Google Scholar] [CrossRef]
- Kumar, S.; Forster, H.M.; Bailey, P.; Griffiths, T.D. Mapping Unpleasantness of Sounds to Their Auditory Representation. J. Acoust. Soc. Am. 2008, 124, 3810–3817. [Google Scholar] [CrossRef]
- Letowski, T.R.; Letowski, S.T. Auditory Spatial Perception: Auditory Localization; Defense Technical Information Center: Fort Belvoir, VA, USA, 2012. [Google Scholar]
- Macpherson, E.A.; Middlebrooks, J.C. Listener Weighting of Cues for Lateral Angle: The Duplex Theory of Sound Localization Revisited. J. Acoust. Soc. Am. 2002, 111, 2219. [Google Scholar] [CrossRef] [Green Version]
- Perrott, D.R.; Saberi, K. Minimum Audible Angle Thresholds for Sources Varying in Both Elevation and Azimuth. J. Acoust. Soc. Am. 1990, 87, 1728–1731. [Google Scholar] [CrossRef]
- Chandler, D.W.; Grantham, D.W. Minimum Audible Movement Angle in the Horizontal Plane as a Function of Stimulus Frequency and Bandwidth, Source Azimuth, and Velocity. J. Acoust. Soc. Am. 1992, 91, 1624–1636. [Google Scholar] [CrossRef]
- Strybel, T.Z.; Manllgas, C.L.; Perrott, D.R. Minimum Audible Movement Angle as a Function of the Azimuth and Elevation of the Source. Hum. Factors 1992, 34, 267–275. [Google Scholar] [CrossRef]
- Algazi, R.; Avendano, C.; Duda, R.O. Estimation of a Spherical-Head Model from Anthropometry. J. Aud. Eng. Soc. 2001, 49, 472–479. [Google Scholar]
- MacDonald, J.A.; Henry, P.P.; Letowski, T.R. Spatial Audio through a Bone Conduction Interface. Int. J. Audiol. 2006, 45, 595–599. [Google Scholar] [CrossRef] [PubMed]
- Cabot, R.C. Preferred Sampling Frequencies for Applications Employing Pulse-Code Modulation; Audio Engineering Society: New York, NY, USA, 2018. [Google Scholar]
- Bluetooth Audio Sunglasses|Bose. Available online: https://www.bose.com/en_us/products/frames.html (accessed on 9 March 2021).
- Aeropex. Available online: https://us.aftershokz.com/products/aeropex (accessed on 16 March 2021).
- National Instruments. Available online: https://www.ni.com/ (accessed on 18 March 2021).
- Degenaar, J.; O’Regan, J.K. Sensorimotor Theory of Consciousness. Scholarpedia 2015, 10, 4952. [Google Scholar] [CrossRef]
- Karnath, H.-O.; Dieterich, M. Spatial Neglect—A Vestibular Disorder? Brain 2006, 129, 293–305. [Google Scholar] [CrossRef] [PubMed]
- Jacquin-Courtois, S.; O’Shea, J.; Luauté, J.; Pisella, L.; Revol, P.; Mizuno, K.; Rode, G.; Rossetti, Y. Rehabilitation of Spatial Neglect by Prism Adaptation. Neurosci. Biobehav. Rev. 2013, 37, 594–609. [Google Scholar] [CrossRef]
- Welch, R.B. Adaptation to Prism-Displaced Vision: The Importance of Target-Pointing. Percept. Psychophys. 1969, 5, 305–309. [Google Scholar] [CrossRef] [Green Version]
- Iachini, T. Does Blindness Affect Egocentric and Allocentric Frames of Reference in Small and Large Scale Spaces? Behav. Brain Res. 2014, 9, 73–81. [Google Scholar] [CrossRef]
- Aggius-Vella, E.; Kolarik, A.J.; Gori, M.; Cirstea, S.; Campus, C.; Moore, B.C.; Pardhan, S. Comparison of Auditory Spatial Bisection and Minimum Audible Angle in Front, Lateral, and Back Space. Sci. Rep. 2020, 10, 1–9. [Google Scholar] [CrossRef] [Green Version]
- Gori, M.; Sandini, G.; Martinoli, C.; Burr, D.C. Impairment of Auditory Spatial Localization in Congenitally Blind Human Subjects. Brain 2014, 137, 288–293. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Vercillo, T.; Burr, D.; Gori, M. Early Visual Deprivation Severely Compromises the Auditory Sense of Space in Congenitally Blind Children. Dev. Psychol. 2016, 52, 847–853. [Google Scholar] [CrossRef] [Green Version]
- Voss, P.; Lassonde, M.; Gougoux, F.; Fortin, M.; Guillemot, J.-P.; Lepore, F. Early- and Late-Onset Blind Individuals Show Supra-Normal Auditory Abilities in Far-Space. Curr. Biol. 2004, 14, 1734–1738. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Foster, D.H. Color Constancy. Vis. Res. 2011, 51, 674–700. [Google Scholar] [CrossRef] [PubMed]
- Alfaro, A.; Bernabeu, Á.; Agulló, C.; Parra, J.; Fernández, E. Hearing Colors: An Example of Brain Plasticity. Front. Syst. Neurosci. 2015, 9. [Google Scholar] [CrossRef] [Green Version]
- Bizoń-Angov, P.; Osiński, D.; Wierzchoń, M.; Konieczny, J. Visual Echolocation Concept for the Colorophone Sensory Substitution Device Using Virtual Reality. Sensors 2021, 21, 237. [Google Scholar] [CrossRef]
- Bregman, A.S. Auditory Scene Analysis: The Perceptual Organization of Sound; MIT Press: Cambridge, MA, USA, 1994; ISBN 978-0-262-52195-6. [Google Scholar]
- Moore, B.C.J.; Gockel, H.E. Properties of Auditory Stream Formation. Philos Trans. R. Soc. Lond. B Biol. Sci. 2012, 367, 919–931. [Google Scholar] [CrossRef] [Green Version]
System/Author | Camera Integration | Real Time | Spatial Sound |
---|---|---|---|
ColEnViSon | No | No | No |
HueMusic | No | No | Yes |
Musical Vision | No | No | Yes |
SoundView | No | Yes | No |
Creole | No | Yes | No |
EyeMusic | Yes | No | Yes |
Sofia Cavaco et al. | Yes | No | Yes |
Eyeborg | Yes | Yes | No |
KromoPhone | Yes | Yes | No |
Colorophone 1.0 | Yes | Yes | No |
See ColOr | Yes | Yes | Yes |
Color Component | Sound Frequency (Hz) | Note | Sound Type |
---|---|---|---|
Red | 1027 | C6 | Musical instruments |
Yellow | 647 | E5 | |
Green | 408 | G#4 | |
Blue | 256 | C4 | |
White | - | - | Rainfall |
Headphones | Bose | Aftershokz |
---|---|---|
Weight (g) | 50 | 26 |
Battery life (h) | Up to 5.5 | 8 |
Charging time 0–100% (min) | 60 | 90 |
Waterproof (IP) | No (IPX2) | Yes (IP67) |
Retail price ($) | 249.95 | 159.95 |
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Osiński, D.; Łukowska, M.; Hjelme, D.R.; Wierzchoń, M. Colorophone 2.0: A Wearable Color Sonification Device Generating Live Stereo-Soundscapes—Design, Implementation, and Usability Audit. Sensors 2021, 21, 7351. https://doi.org/10.3390/s21217351
Osiński D, Łukowska M, Hjelme DR, Wierzchoń M. Colorophone 2.0: A Wearable Color Sonification Device Generating Live Stereo-Soundscapes—Design, Implementation, and Usability Audit. Sensors. 2021; 21(21):7351. https://doi.org/10.3390/s21217351
Chicago/Turabian StyleOsiński, Dominik, Marta Łukowska, Dag Roar Hjelme, and Michał Wierzchoń. 2021. "Colorophone 2.0: A Wearable Color Sonification Device Generating Live Stereo-Soundscapes—Design, Implementation, and Usability Audit" Sensors 21, no. 21: 7351. https://doi.org/10.3390/s21217351