The Senses of Music: Towards a Theoretical Model of Multisensory Musical Experience
Abstract
1. Introduction
2. Philosophical and Theoretical Foundations of Multisensory Perception
2.1. Aristotelian Tradition
2.2. Three Philosophical Trends for Understanding Multisensory Perception
3. Multisensory Phenomena and Conceptual Delimitation
4. Crossmodal Correspondences: Variables and Nature
5. The Senses of Music or the Music of the Senses: A Proposed Model
5.1. Structural Principles (Universality, Congruence, Hierarchical Tendency)
5.2. Interaction Between Musical Organisation, Cognitive Structure, Mobilised Sensory Systems and Affective Aspects
6. Prospects for Future Research
- A.
- Theoretical, (1) to investigate the relationship between statistical learning and patterns of correspondence involving music in specific sensory domains; (2) to compare CCs in response to simple and complex stimuli, verifying the functioning of the congruence effect and hierarchical tendency; (3) to propose a predictive model of CCs in musical experience connecting with expectation theory.
- B.
- Empirical, (1) to investigate how CCs work in real contexts, such as restaurants, concert halls, and learning contexts, in dialogue with more ecological research trends; (2) to explore the role of attention in the perception of multisensory experiences and the emotional factors involved in this process; (3) to analyse the effects of musical training, age, and cultural context on CCs; (4) to investigate the structural basis of multisensory perception using neuroimaging techniques, identifying the areas of the brain involved in multisensory experiences.
- C.
- Teaching, (1) to examine the effects of multisensory approaches on music teaching and learning processes; (2) to provide empirical evidence on multisensory processing from childhood onwards; (3) to investigate the relationships between multisensory experiences and the formation of concepts and memorisation of musical patterns.
7. Considerations
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Acknowledgments
Conflicts of Interest
Abbreviations
| CC | Crossmodal correspondence |
| CCs | Crossmodal correspondences |
References
- Tan, S.; Pfordresher, P.; Harré, R. Psychology of Music: From Sound to Significance; Psychology Press: New York, NY, USA, 2010. [Google Scholar]
- Deroy, O.; Spence, C. Why we are not all synesthetes (not even weakly so). Psychon. Bull. Rev. 2013, 20, 643–664. [Google Scholar] [CrossRef]
- Gallace, A.; Spence, C. Multisensory synesthetic interactions in the speeded classification of visual size. Percept. Psychophys. 2006, 68, 1191–1203. [Google Scholar] [CrossRef] [PubMed]
- Parise, C.; Spence, C. Synesthetic congruency modulates the temporal ventriloquism effect. Neurosci. Lett. 2008, 442, 257–261. [Google Scholar] [CrossRef]
- Parise, C.; Spence, C. “When birds of a feather flock together”: Synesthetic correspondences modulate audiovisual integration in non-synesthetes. PLoS ONE 2009, 4, e5664. [Google Scholar] [CrossRef]
- Parise, C.; Spence, C. Audiovisual crossmodal correspondences and sound symbolism: A study using the implicit association test. Exp. Brain Res. 2012, 220, 319–333. [Google Scholar] [CrossRef]
- Walker, P.; Smith, S. Stroop interference based on the multimodal correlates of haptic size and auditory pitch. Perception 1985, 14, 729–736. [Google Scholar] [CrossRef]
- Evans, K.; Treisman, A. Natural cross-modal mappings between visual and auditory features. J. Vis. 2010, 10, 6. [Google Scholar] [CrossRef]
- Marks, L. On Cross-Modal Similarity: Auditory-Visual Interactions in Speeded Discrimination. J. Exp. Psychol. 1987, 16, 384–394. [Google Scholar] [CrossRef]
- Adeli, M.; Rouat, J.; Molotchnikoff, S. Audiovisual correspondence between musical timbre and visual shapes. Front. Hum. Neurosci. 2014, 8, 352. [Google Scholar] [CrossRef] [PubMed]
- Palmer, S.; Schloss, K.; Xu, Z.; Prado-León, L. Music–color associations are mediated by emotion. Proc. Natl. Acad. Sci. USA 2013, 110, 8836–8841. [Google Scholar] [CrossRef] [PubMed]
- Lindborg, P.; Friberg, A. Colour Association with Music Is Mediated by Emotion: Evidence from an Experiment Using a CIE Lab Interface and Interviews. PLoS ONE 2015, 10, e0144013. [Google Scholar] [CrossRef]
- Whiteford, K.L.; Schloss, K.B.; Helwig, N.E.; Palmer, S.E. Color, music, and emotion: Bach to the blues. i-Perception 2018, 9, 2041669518808535. [Google Scholar] [CrossRef] [PubMed]
- Murari, M.; Schubert, E.; Rodà, A.; Da Pos, O.; De Poli, G. How >:(is Bizet? Icon ratings of music. Psychol. Music 2017, 46, 749–760. [Google Scholar] [CrossRef]
- Marin, M.; Gingras, B.; Bhattacharya, J. Crossmodal transfer of arousal, but not pleasantness, from the musical to the visual domain. Emotion 2012, 12, 618–631. [Google Scholar] [CrossRef]
- Simurra, I.; Vanzella, P.; Sato, J. Timbre and Visual Forms: A crossmodal study relating acoustic features and the Bouba- Kiki Effect. In Proceedings of the 2nd International Conference on Timbre (Timbre 2020), Thessaloniki, Greece (online), 3–4 September 2020; Available online: http://timbre2020.mus.auth.gr/assets/papers/Timbre2020_proceedings.pdf (accessed on 3 April 2026).
- Albertazzi, L.; Canal, L.; Micciolo, R. Cross-modal associations between materic painting and classical Spanish music. Front. Psychol. 2015, 6, 424. [Google Scholar] [CrossRef] [PubMed]
- Albertazzi, L.; Canal, L.; Micciolo, R.; Hachen, I. Cross-Modal Perceptual Organization in Works of Art. i-Perception 2020, 11, 2041669520950750. [Google Scholar] [CrossRef]
- Braun Janzen, T.; de Oliveira, B.; Ventorim Ferreira, G.; Sato, J.R.; Feitosa-Santana, C.; Vanzella, P. The effect of background music on the aesthetic experience of a visual artwork in a naturalistic environment. Psychol. Music 2022, 51, 16–32. [Google Scholar] [CrossRef]
- Guetta, R.; Loui, P. When music is salty: The crossmodal associations between sound and taste. PLoS ONE 2017, 12, e0173366. [Google Scholar] [CrossRef]
- Reinoso-Carvalho, F.; Wang, Q.; van Ee, R.; Persoone, D.; Spence, C. “Smooth operator”: Music modulates the perceived creaminess, sweetness, and bitterness of chocolate. Appetite 2017, 108, 383–390. [Google Scholar] [CrossRef]
- Kantono, K.; Hamid, N.; Shepherd, D.; Yoo, M.J.Y.; Carr, B.T.; Grazioli, G. The effect of background music on food pleasantness ratings. Psychol. Music 2016, 44, 1111–1125. [Google Scholar] [CrossRef]
- Lin, Y.; Hamid, N.; Shepherd, D.; Kantono, K.; Spence, C. Musical and Non-Musical Sounds Influence the Flavour Perception of Chocolate Ice Cream and Emotional Responses. Foods 2022, 11, 1784. [Google Scholar] [CrossRef]
- Galmarini, M.; Silva Paz, R.; Enciso Shoquehuanca, D.; Zamora, D.; Mesz, B. Impact of music on the dynamic perception of coffee and evoked emotions evaluated by temporal dominance of sensations (TDS) and emotions (TDE). Food Res. Int. 2021, 150, 110795. [Google Scholar] [CrossRef]
- Wang, Q.; Spence, C. Assessing the Effect of Musical Congruency on Wine Tasting in a Live Performance Setting. i-Perception 2015, 6, 2041669515593027. [Google Scholar] [CrossRef]
- Velasco, C.; Balboa, D.; Marmolejo-Ramos, F.; Spence, C. Crossmodal effect of music and odor pleasantness on olfactory quality perception. Front. Psychol. 2014, 5, 1352. [Google Scholar] [CrossRef] [PubMed]
- Sathian, K.; Ramachandran, V. Multisensory Perception: From Laboratory to Clinic; Academic Press: Cambridge, MA, USA, 2020. [Google Scholar] [CrossRef]
- Ramachandran, V.; Marcus, Z.; Chunharas, C. Bouba-Kiki: Cross-domain resonance and the origins of synesthesia, metaphor, and words in the human mind. In Multisensory Perception: From Laboratory to Clinic; Sathian, K., Ramachandran, V., Eds.; Academic Press: Cambridge, MA, USA, 2020; pp. 3–40. [Google Scholar] [CrossRef]
- Marks, L. The Unity of Senses: Interrelations Among the Modalities; Academic Press: New York, NY, USA, 1978; Available online: https://play.google.com/books/reader?id=qjq0BQAAQBAJ&pg=GBS.PA5 (accessed on 3 April 2026).
- Bruno, N.; Pavani, F. Perception: A Multisensory Perspective; Oxford University Press: New York, NY, USA, 2018. [Google Scholar]
- Auvray, M.; Spence, C. The multisensory perception of flavor. Conscious. Cogn. 2008, 17, 1016–1031. [Google Scholar] [CrossRef]
- Caznok, Y.B. Música: Entre o Audível e o Visível; Editora Unesp: São Paulo, Brazil, 2008. [Google Scholar]
- Reinoso-Carvalho, F.; Dakduk, S.; Wagemans, J.; Spences, C. Not Just Another Pint! The Role of Emotion Induced by Music on the Consumer’s Tasting Experience. Multisens. Res. 2019, 32, 367–400. [Google Scholar] [CrossRef] [PubMed]
- Spence, C.; Richards, L.; Kjellin, E.; Huhnt, A.-M.; Daskal, V.; Scheybeler, A.; Velasco, C.; Deroy, O. Looking for crossmodal correspondences between classical music and fine wine. Flavour 2013, 2, 29. [Google Scholar] [CrossRef]
- Parise, C. Crossmodal Correspondences: Standing Issues and Experimental Guidelines. Multisens. Res. 2016, 29, 7–28. [Google Scholar] [CrossRef] [PubMed]
- Bergantini, L. Multissensorialidade: Contribuições da Arte-Tecnologia a Partir do Caso do Festival Ars Electronica 2019. Ph.D. Thesis, Escola de Comunicações e Artes da Universidade de São Paulo, São Paulo, Brazil, 2021. Available online: https://www.teses.usp.br/teses/disponiveis/27/27159/tde-15022022-120744/en.php (accessed on 3 April 2026).
- Leote, R. (Ed.) Processos perceptivos e multissensorialidade: Entendendo a arte multimodal sob conceitos neurocientíficos. In ArteCiênciaArte; Editora UNESP: São Paulo, Brazil, 2015; pp. 23–44. Available online: https://books.scielo.org/id/mqfvk (accessed on 3 April 2026).
- Stein, B.; Burry, D.; Constantinidis, C.; Laurienti, P.; Meredith, M.; Perraut, T., Jr.; Ramachandran, R.; Röder, B.; Rowland, B.; Sathian, K.; et al. Semantic confusion regarding the development of multisensory integration: A practical solution. Eur. J. Neurosci. 2010, 31, 1713–1720. [Google Scholar] [CrossRef]
- Parise, C.; Spence, C. Audiovisual cross-modal correspondences in the general population. In The Oxford Handbook of Synesthesia; Simner, J., Hubbard, E.M., Eds.; Oxford University Press: Oxford, UK, 2013; pp. 790–815. [Google Scholar] [CrossRef]
- Spence, C. Crossmodal correspondences: A tutorial review. Atten. Percept. Psychophys. 2011, 73, 971–995. [Google Scholar] [CrossRef]
- Brang, D.; Ramachandran, V.S. How do crossmodal correspondences and multisensory processes relate to synesthesia? In Multisensory Perception: From Laboratory to Clinic; Sathian, K., Ramachandran, V.S., Eds.; Academic Press: Cambridge, MA, USA, 2020; pp. 259–281. [Google Scholar] [CrossRef]
- Bragança, G. Relações entre Sensações Sinestésicas, Estados Emocionais e Estruturas Musicais. Ph.D. Thesis, Instituto de Ciências Biológicas da Universidade Federal de Minas Gerais, Belo Horizonte, Brazil, 2014. Available online: https://hdl.handle.net/1843/BUBD-9WMUJF (accessed on 3 April 2026).
- Ramachandran, V.S.; Hubbard, E.M. Hearing colors, tasting shapes. Sci. Am. Mind 2003, 16, 16–23. [Google Scholar] [CrossRef] [PubMed]
- Eitan, Z. How pitch and loudness shape musical space and motion. In The Psychology of Music in Multimedia; Tan, S., Cohen, A., Lipscomb, S., Kendall, R., Eds.; Oxford University Press: Oxford, UK, 2013; pp. 165–191. [Google Scholar] [CrossRef]
- Martino, G.; Marks, L. Synesthesia: Strong and weak. Curr. Dir. Psychol. Sci. 2001, 10, 61–65. [Google Scholar] [CrossRef]
- Spence, C. Assessing the Role of Emotional Mediation in Explaining Crossmodal Correspondences Involving Musical Stimuli. Multisens. Res. 2020, 33, 1–29. [Google Scholar] [CrossRef]
- Spence, C. Exploring Group Differences in the Crossmodal Correspondences. Multisens. Res. 2022, 35, 495–536. [Google Scholar] [CrossRef]
- Cytowic, R.E.; Eagleman, D.M. Wednesday is Indigo Blue: Discovering the Brain of Synesthesia; Boston Review: Cambridge, MA, USA, 2009. [Google Scholar]
- Hauck, P.; Hecht, H. Having a Drink with Tchaikovsky: The Crossmodal Influence of Background Music on the Taste of Beverages. Multisens. Res. 2019, 32, 1–24. [Google Scholar] [CrossRef]
- North, A. The effect of background music on the taste of wine. Br. J. Psychol. 2012, 103, 293–301. [Google Scholar] [CrossRef]
- Wang, Q.; Mesz, B.; Riera, P.; Trevisan, M.; Sigman, M.; Guha, A.; Spence, C. Analysing the Impact of Music on the Perception of Red Wine via Temporal Dominance of Sensations. Multisens. Res. 2019, 32, 455–472. [Google Scholar] [CrossRef] [PubMed]
- Blazhenkova, O.; Kumar, M.M. Angular versus curved shapes: Correspondences and emotional processing. Perception 2018, 47, 67–89. [Google Scholar] [CrossRef]
- Liew, K.; Lindborg, P.; Rodrigues, R.; Styles, S. Cross-Modal Perception of Noise-in-Music: Audiences Generate Spiky Shapes in Response to Auditory Roughness in a Novel Electroacoustic Concert Setting. Front. Psychol. 2018, 9, 178. [Google Scholar] [CrossRef]
- Spence, C.; Sathian, K. Audiovisual crossmodal correspondences: Behavioural consequences and neural underpinnings. In Multisensory Perception: From Laboratory to Clinic; Sathian, K., Ramachandran, V.S., Eds.; Academic Press: Cambridge, MA, USA, 2020; pp. 239–258. [Google Scholar] [CrossRef]
- Ramachandran, V.S.; Hubbard, E.H. Synaesthesia: A window into perception, thought and language. J. Conscious. Stud. 2001, 8, 3–34. [Google Scholar]
- Athanasopoulos, G.; Antović, M. Conceptual integration of sound and image: A model of perceptual modalities. Music Sci. 2018, 22, 72–87. [Google Scholar] [CrossRef]
- Köhler, W. Gestalt Psychology; Liveright: New York, NY, USA, 1929. [Google Scholar]
- Bremner, A.J.; Caparos, S.; Davidoff, J.; de Fockert, J.; Linnell, K.J.; Spence, C. “Bouba” and “ Kiki” in Namibia? A remote culture make similar shape-sound matches, but different shape-taste matches to Westerners. Cognition 2013, 126, 165–172. [Google Scholar] [CrossRef]
- Davis, R. The fitness of names to drawings: A cross-cultural study in Tanganyika. Br. J. Psychol. 1961, 52, 259–268. [Google Scholar] [CrossRef]
- Padulo, C.; Tommasi, L.; Brancucci, A. Implicit Association Effects Between Sound and Food Images. Multisens. Res. 2018, 31, 779–791. [Google Scholar] [CrossRef] [PubMed]
- Padulo, C.; Mangone, M.; Brancucci, A.; Balsamo, M.; Fairfield, B. Crossmodal congruency effects between sound and food pictures in a forced-choice task. Psychol. Res. 2021, 85, 2340–2345. [Google Scholar] [CrossRef] [PubMed]
- London, J.; Burger, B.; Thompson, M.; Toiviainen, P. Speed on the dance floor: Auditory and visual cues for musical tempo. Acta Psychol. 2016, 164, 70–80. [Google Scholar] [CrossRef] [PubMed]
- Kohn, D.; Eitan, Z. moving music: Correspondences of musical parameters and movement dimensions in children’s motion and verbal responses. Music Percept. 2016, 34, 40–55. [Google Scholar] [CrossRef]
| Concept | Definition (Short) | Population Scope | Example |
|---|---|---|---|
| Sensory integration | Combination of information from different senses into one percept or unified event [2,38] | All individuals | Seeing and hearing a performer |
| Crossmodal correspondences | Non-arbitrary feature matchings across modalities [35,36,37,38,39,40] | All individuals (statistical) | High pitch—small visual size |
| Synesthesia | Involuntary, idiosyncratic concurrents triggered by inducers [2,41,42,43] | Minority (~4%) | Pitch—colour in synesthetes |
| Type | Basis | Example |
|---|---|---|
| Structural [39,40,55] | Neural architecture | Shared coding of intensity across senses |
| Statistical [35,39,40] | Environmental regularities | Large instruments—low sounds |
| Semantic [39,40] | Shared linguistic descriptors | “Bright” timbre and “bright” colour |
| Emotional [46,54] | Shared affective meaning | Sad music—dark desaturated colours |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2026 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license.
Share and Cite
Nogueira, C.; Pereira, A.I.; Rodrigues, H. The Senses of Music: Towards a Theoretical Model of Multisensory Musical Experience. Encyclopedia 2026, 6, 94. https://doi.org/10.3390/encyclopedia6050094
Nogueira C, Pereira AI, Rodrigues H. The Senses of Music: Towards a Theoretical Model of Multisensory Musical Experience. Encyclopedia. 2026; 6(5):94. https://doi.org/10.3390/encyclopedia6050094
Chicago/Turabian StyleNogueira, Cristiane, Ana Isabel Pereira, and Helena Rodrigues. 2026. "The Senses of Music: Towards a Theoretical Model of Multisensory Musical Experience" Encyclopedia 6, no. 5: 94. https://doi.org/10.3390/encyclopedia6050094
APA StyleNogueira, C., Pereira, A. I., & Rodrigues, H. (2026). The Senses of Music: Towards a Theoretical Model of Multisensory Musical Experience. Encyclopedia, 6(5), 94. https://doi.org/10.3390/encyclopedia6050094

