Sound Coding Color to Improve Artwork Appreciation by People with Visual Impairments

Cho, Jun Dong; Jeong, Jaeho; Kim, Ji Hye; Lee, Hoonsuk

doi:10.3390/electronics9111981

Open AccessFeature PaperArticle

Sound Coding Color to Improve Artwork Appreciation by People with Visual Impairments

¹

Department of Human Information and Cognition Technology Convergence, SungkyunKwan University, Suwon, Gyeonggi-do 16419, Korea

²

Center for Human Information and Cognition Technology Convergence, SungkyunKwan University, Suwon, Gyeonggi-do 16419, Korea

³

Sound Plan, Seoul 06974, Korea

⁴

Gangbuk Braille Library, Seoul 06974, Korea

^*

Author to whom correspondence should be addressed.

Electronics 2020, 9(11), 1981; https://doi.org/10.3390/electronics9111981

Submission received: 8 October 2020 / Revised: 17 November 2020 / Accepted: 19 November 2020 / Published: 23 November 2020

(This article belongs to the Special Issue Multi-Sensory Interaction for Blind and Visually Impaired People)

Download

Browse Figures

Versions Notes

Abstract

:

The recent development of color coding in tactile pictograms helps people with visual impairments (PVI) appreciate the visual arts. The auditory sense, in conjunction with (or possibly as an alternative to) the tactile sense, would allow PVI to perceive colors in a way that would be difficult to achieve with just a tactile stimulus. Sound coding colors (SCCs) can replicate three characteristics of colors, i.e., hue, chroma, and value, by matching them with three characteristics of sound, i.e., timbre, intensity, and pitch. This paper examines relationships between sound (melody) and color mediated by tactile pattern color coding and provides sound coding for hue, chroma, and value to help PVI deepen their relationship with visual art. Our two proposed SCC sets use melody to improve upon most SCC sets currently in use by adding more colors (18 colors in 6 hues). User experience and identification tests were conducted with 12 visually impaired and 8 sighted adults, and the results suggest that the SCC sets were helpful for the participants.

Keywords:

user experience; visually impaired; color sound coding; accessibility; art appreciation

1. Introduction

Artists believe that art is central to human life. Unfortunately, many people with visual impairments (PVI) do not have access to the world’s visual culture or the opportunity to experience the life-enhancing power of visual art. PVI must have access to the world’s visual culture if they are to participate fully in their communities and the world at large. Such access will improve the quality of their lives and help them gain skills crucial to their education and employment opportunities [1]. Findings from participant observations in touch tours for blind and visually impaired people at the Metropolitan Museum of Art were discussed by Hayhoe [2]. Touch Graphics Inc. exhibits a reproduced painting called “Talking Tactile” at the San Diego Museum of Art [3]. Recently, “BlindTouch” [4,5] has been introduced to enhance PVI’s artwork experience by reproducing painting masterpieces as 3D-printed models that contain touch recognition sensors (based on conductive painting) and provide relevant audio descriptions and sound effects (such as the sound of the wind or a flying butterfly) while the user explores the different features of the artwork with their fingers. The touch interface was evaluated by PVI and found to be effective. The audio feedback can be activated by tapping the fingers. However, PVI still find it difficult to effectively experience color when appreciating a museum’s artwork. Tactile color pictograms (TCPs), which are embossed on a surface that PVI can touch to perceive color information, are used to allow easy accessibility to color information. TCPs have two advantages as an assistive tool when used in conjunction with audio descriptions. First, they allow immediate access to color information through color patterns, just as a sighted person sees colors immediately. Second, embossing a TCP directly on a piece of artwork reproduction surface allows PVI to grasp color and color-related information (e.g., shape, size, brightness, and position) through tactile interaction [6].

However, for color recognition, tactile patterns coding colors (TPCs) alone might not provide a good user experience for PVI (especially those with congenital blindness or complex disabilities) because tactile interactions tend to be slow. Congenitally blind people understand colors through physical and abstract associations. Color audition means the reaction of feeling color in one sound. Gauguin tried to pursue a symbol of innerness as a color. He said, “Color which, like music, is a matter of vibrations, reaches what is most general and therefore most indefinable in nature: its inner power” [7].

Therefore, it might be desirable to provide color images for PVI that convert the color being touched into a corresponding musical sound, such as orchestral or classical music, that codes the color. In other words, to provide better color perception for PVI, a multisensory user experience that combines tactile and musical stimuli might be effective. A combination of TPCs and sound coding colors (SCCs) could also enable PVI to interpret the overall color composition of a piece of artwork. Therefore, we here explore the association of sound tones (timbre, pitch, and intensity) with the color properties (hue, value, and chroma) of the Munsell color system. Based on the observations and series of user tests performed in this research, we here introduce two SCC sets that represent 18 colors in 6 hues.

1.1. Review of the Color System

The Munsell color system is a color space that specifies colors based on the three properties of hue (basic color), value (lightness), and chroma (color intensity) [8]. In this system, the higher the lightness value, the closer the color is to white, and the lower the value, the closer it is to black. Chroma is the vividness (clearness) of a color. Colors with the highest chroma are pure colors (saturated colors), meaning that they are not mixed with other colors, and colors with the lowest chroma are achromatic (white, gray, black). Palmer et al. described 37 colors that reflect the color preferences of US college students [9]. Munsell’s hue/value/chroma data from those 37 colors can be found in [9]. The SCC sets proposed in this paper can express six unique hues (red (R), orange (O), yellow (Y), green (G), blue (B), and purple (P)) using three color dimensions—saturated (S), light (L), and dark (D)—for each hue, as shown in Figure 1.

For example, in Figure 2, the color marked “Light” has a value of seven and chroma of eight. The color marked “Dark” has a value of three and a chroma of eight. The color marked “Saturated” has a value of five and a chroma of 15. The colors with the lowest chroma (0) are achromatic.

1.2. Review of the Sound Representations of Colors

The perception of a sound has several aspects:

Pitch: the frequency of a sound (high or low).
Tone: the timbre of a pitch.
Key: is it a major or minor scale?
Timbre: the characteristic sound of an instrument (e.g., a flute and an oboe playing the same note sound differently).
Chords: two or more notes played at the same time.
Melody: sequentially arranged notes.
Volume: loudness.
Velocity (the musical term): the force with which a note is played, and it is vitally important in making MIDI performances sound human.

Synesthesia, a mixing of the senses, occurs in some individuals. People with strong synesthesia, synesthetes, experience a perception in one sense when a stimulus for another sense is presented (e.g., seeing a color when hearing a sound). This is what Martino and Marks [10] call “strong synesthesia” as opposed to “weak synesthesia,” which describes the simple cross-sensory correspondences that most people experience, e.g., being able to match a tone of a certain pitch to a light of a certain brightness [11]. One of the most common types of synesthesia is seeing color while listening to music or musical notes (e.g., [12,13,14,15]). Gaboski and Odebert [12] showed some tendency to associate color with short musical selections. Slow music is associated with blue, high tones are light, and low tones are dark. Fast and cheerful notes are reminiscent of bright and intense warm colors.

As a synesthete, Kandinsky saw colors corresponding to the sounds of different instruments and used those correspondences in his paintings [13]. He attributed musicality to his paintings and said that painting could generate energy like music, “drawing is rhyming the form with color and showing the moving power through color.” He believed that just as musicians express their feelings in musical forms such as rhythm, timbre, and melody, artists can express their inner experiences of fear, sadness, and joy using various arrangements of color and form. He associated colors and forms with specific emotions, connected them with elements of music, and then sought to write those subjective and emotional experiences into logical and universal laws. “Yellow is stimulus, red is energy, blue is infinite sensibility, and green is calm,” said Kandinsky.

People without synesthesia also report correspondences between music and color. For example, [14,16] found some evidence that music–color correspondences result from an emotional link. Brighter colors such as yellow, red, green, and blue were usually assigned to happy songs, and gray was usually assigned to sad songs [17]. A study by Palmer [9] also suggested that fast notes in a major key are yellow or orange and slow notes in a major key are blue and gray. He found cross-modal correlations based on the associative sensibility between color and music with the help of music that involves emotion.

Newton’s Opticks [18] showed that the colors of the spectrum and the pitches of musical scales are similar (for example, “red” and “C”; “green” and “Ab”). Maryon [19] also explored the similarity between the ratio of each tone to the wavelength of each color to connect them. This method of associating the pitch frequency of the scale with color can be a way of substituting colors and notes for one another [20]. However, the various sensibilities that can be obtained through color are limited by simply substituting colors into the musical scale. Alber Lavigna [21] found that the technique of a composer in organizing an orchestra seems very similar to the technique of a painter applying colors. In other words, a musician’s palette is a list of orchestral instruments.

A comprehensive survey of associations between color and sound can be found in [22], including how different color properties such as value and hue are mapped onto acoustic properties such as pitch and loudness. Using an implicit associations test, those researchers [22] confirmed the following cross-modal correspondences between visual and acoustic features. Pitch was associated with color lightness, whereas loudness mapped onto greater visual saliency. The associations between vowels and colors are mediated by differences in the overall balance of low- and high-frequency energy in the spectrum rather than by vowel identity as such. The hue of colors with the same luminance and saturation was not associated with any of the tested acoustic features, except for a weak preference to match higher pitch with blue (vs. yellow). In other research, high loudness was associated with orange/yellow rather than blue, and high pitch was associated with yellow rather than blue [23].

Chroma has a relationship with sound intensity [23,24]. When the intensity of a sound is strong and loud, its color is close, intense, and deep. However, when the sound intensity is weak, the color feels pale, faint, and far away. Higher value is associated with higher pitch [17,25]. Children of all ages and adults matched pitch to value and loudness to chroma. The value (i.e., lightness) is high and heavy dependent on the light and dark levels of the color. Using the same concept in music, sound is divided into light and heavy feelings according to the high and low octaves of a scale. When the intensity of the sound is strong, the color sensed is close and sharp, whereas when the intensity of the sound is weak, the color becomes distant and muted [26].

Another way to match color and sound is to associate an instrument’s tone with color, as in Kandinsky [14]. A low-pitched cello has a low-brightness dark blue color; a violin or trumpet-like instrument with a sharp tone feels red or yellow; and a high-pitched flute feels like a bright and saturated sky blue. As shown in Table 1, it is possible to compare the sensibility felt in each instrument tone with the sensibility felt in color.

In “SeeColor” [27], when you touch a relief-shaped, embossed outline area, the color associated with that area is transmitted to an instrument’s sound.

In color sonification [28], the color hue, chroma, and value attributes of pixels are mapped onto sound parameters as the f0 (which is related to a sound’s perceived pitch), the spectral envelope of the sound (which influences the perception of timbre), and the intensity (which is related to the sound’s perceived loudness), respectively. The sound output of a hue value is thus a sinusoid whose frequency depends on that value. The lower frequency colors, such as red and orange, give sensations of strength and power, so they relate well with higher pitched sounds.

The future work from [28] is to deal with more complex sounds and rhythmic patterns. However, there was no assessment provided from these methods to confirm the sound-code literacy of PVI.

In this paper, based on the aforementioned observations, we introduce two SCC sets, VIVALDI and CLASSIC, produced with rhythmic instrumental sounds of classical melody. Table 1 shows the previously proposed color–instrument matchings and the two SCC sets proposed by us.

Table 1. Existing SCCs with instruments and ours.

Colors	Kandinsky, 1912 [14]	Lavignac,1895 [29]	Deville, 2009 [27]	Ours: VIVLADI	Ours: CLASSIC
R	Tuba + Trumpet (warm light red) Violin (cool light red) Cello-bass (cool red) Tuba-bass (warm red)	Cornet (Red) Clarinet (Red brown:) Trumpet (Purple red)	Oboe	Violin + Cello	Violin
O	Viola	Trumpet+Trombone (Crimson with orange)	Violin	Clarinet + Bassoon	Viola
Y	Trumpet (light yellow)	French horn	Violin pizzicato	Trumpet + Trombone	Trumpet
G	Violin-bass	Oboe	Flute	Guitar	Oboe
B	Flute (Light blue) Cello (Dark blue) Contrabass (Thick blue) Bass of pipe organ (Deep blue)	Flute (Azure blue) Violin (Blue)	Piano	Piano	Cello
P	English Horn	English Horn	Saxophone	Organ	Organ

2. Materials, Methods

2.1. Chord Coding Colors (CCC)

The purpose of this study was to create sounds by which the colors in an image are expressed. Our first SCC set is a so-called chord coding colors (CCC) set as shown in Table 2. Each color’s chroma and value are represented by a unique musical chord (e.g., the G chord is made up of the notes G, B, and D. The Gm chord is comprised of G, Bb, and D) that sounds as a single note. In the CCC set, the colors red, orange, yellow, blue, green, indigo, and purple are represented by the instruments violin, viola, trumpet, oboe, cello, horn, and saxophone, respectively. The color–instrument assignment mostly stems from Kandinsky in Table 1. High chroma (saturated) maps to a sound intensity denoted by “^” (velocity = 127), and the medium chroma light and dark map to sound intensity (velocity = 60). A high value (light) maps to a high pitch, denoted by “H” (C4 to C6), and a low value (dark) maps to a low pitch, denoted by “L” (C2 to C4). The codes between colors are all the same, so there is no difficulty to understanding saturated, light, and dark.

2.2. VIVALDI SCC

The user feedback on the initial pilot test motivated us to develop our two melody-based SCC sets (Table 3, Table 4 and Table 5). Our VIVALDI SCC was inspired by Vivaldi’s The Four Seasons, which musically represents elements of the four seasons of the year. We extracted some of Vivaldi’s melodies from spring, autumn, and summer that matched the characteristics of saturated, light, and dark colors, respectively. The most important part in matching instruments and colors is avoiding similar tones so that the sounds (colors) are clearly distinguishable. Therefore, to express each color, we used two strings, two wind instruments, and two percussion instruments for easy identification. In addition, the composition of the excerpted part was changed from original sound, and the velocity and pitch were adjusted to clarify the distinction between value and chroma.

2.2.1. Hue

It seems to be a cross-cultural, innate phenomenon that humans do not arbitrarily attach sounds to shapes. The classic example of this is the kiki/bouba effect: when subjects are asked to pick a word that most corresponds to a particular shape, the word kiki is invariably attached to jagged shapes, and bouba to smooth, rounded shapes [30]. One study [31] found that smooth musical timbres were associated with bouba, as were the colors blue, green, and light gray, whereas harsh timbres and the colors red, yellow, and dark gray were associated with kiki. A rounded shape has been associated with sounds from a piano, along with the colors blue and green [31]. In other words, the piano has a rounded tone that corresponds with bouba, so it goes well with green and blue rather than red or yellow. That correspondence is also consistent with Parise’s findings that sine waves (soft sounds) are associated with a rounded shape and square waves are associated with sharp angular shapes [32].

Therefore, to represent our six hues, we chose: red, strings (violin + cello); orange, guitar; yellow, brass (trumpet + trombone); green, woodwinds (clarinet + bassoon); blue, piano; and purple, organ.

Red, a warm color that gives the feeling of a hot temperature, is represented by stringed instruments with a passionate tone (violin + cello). Brass instruments convey energy, as if a bright light were expanding, and thus represent yellow (trumpet + trombone). Orange, which is a mixture of yellow and red, is represented by the warm and energetic acoustic guitar. Green, which gives the eyes a comfortable and psychologically stable feeling, is represented by woodwinds with a soft and non-irritating tone (clarinet + bassoon). Blue, a cold color, is represented by the piano, which produces a solid and dense tone with a refreshing feel. Purple, a combination of warm red and cold blue, is represented by the pipe organ, which combines a keyboard with brass tubes.

To express the low, medium, and high values of each color using three pitch levels, a wide range of pitches is required to distinguish between saturated, light, and dark colors. As some instruments such as the violin, trumpet, and clarinet have limited pitch ranges, instruments with similar timbres and different pitch ranges need to be mixed.

We rearranged the overlapping notes from the original score from Vivaldi’s The Four Seasons in Figure 3 with the one shown in Figure 4 such that the harmony can be maintained sufficiently with using the least number of overlapping notes. The reason to use the least number of overlapping notes while maintaining harmony is to highlight the melody. In the two scores shown in Figure 4 and Figure 5, the number of notes differs, and the second score is divided into two parts: high-end and low-end. The high-end instruments, such as the violin, trumpet, and clarinet, use two to three tracks, and the low-end instruments, such as the cello, trombone, and bassoon, use two tracks. The melody is placed on the first track and is played only by the high-end instruments. The low-end instruments play only chords.

In the case of dark colors, playing too many notes at a low volume could make it difficult to hear the notes clearly, so one of the high-frequency chords was excluded for simplicity. Instruments such as the guitar, piano, and organ have wide ranges to express all the required melodies and chords, and thus can represent both high and low values on their own.

Here, the melodies and chords of VIVALDI SCC set (for red, orange, yellow, green) were recorded in a soundproof studio with the collaboration of a sound designer, a composer, and performers. The blue (piano) and purple (organ) sound codes were produced by MIDI.

2.2.2. Value and Chroma

The expressions of value and chroma are based on each season’s theme sound. Table 2 shows VIVALDI SCC containing with wav sound source for each color. “Saturated” is from spring, “Light” is from autumn, and “Dark” is from summer. In Vivaldi, each season has entry words. Those for spring are: “Spring has come. Little birds say hello to spring”; those for autumn are: “Villagers rejoice and celebrate the joy of harvesting by dancing and singing”; and those for summer are: “In the season when the sun is strong, men and flocks languish, and the trees and grass are exhausted from the heat.” We chose rhythmic melodies to represent saturated, light, and dark while considering those entry words.

High pitched stimuli are generally matched with white and light, highly saturated colors, whereas stimuli from the lower octaves tend to be paired with dark colors [33]. Chroma is expressed through the melody’s velocity [16,23,24,25]; at “saturated,” it is expressed as a velocity value of 120 or similar. “Saturated” uses the melody excerpted from spring, and it was judged to be suitable for high chroma by using an A major chord at A4 (middle pitch) to express high chroma. “Light” uses the main theme of autumn, with a high-pitched A5 sound that is played in quick succession and a light atmosphere using F major at A5 (high pitch) to highlight the high-brightness feature. “Dark” uses a slow melody excerpted from summer made dull by progression through E minor at G4 (low pitch) to express a low value. All instrument groups were transposed (compared to the original sound source) to enable actual performance without losing the pitch characteristics of saturated, light, and dark.

2.3. CLASSIC SCC V1

The user feedback on the initial pilot test also motivated us to develop the second melody-based SCC set called “CLASSIC SCC V1” in Table 4 CLASSIC SCC V1 has extracted melodies that match the characteristics of each color from classical music sources which also express the characteristics of the instrument well. The tone, pitch, and intensity of the first note of each melody extracted from a classical music source was chosen to match the characteristics of hue, value, and saturation. The numbers in the table refer to the start time and end time of the extracted sound source. We chose classical music for two reasons. (1) In order for the PVI to image the relative position and harmony of the various colors in a piece of artwork, as it will be explained in detail later in Section 2.3, it is necessary to combine the SSCs that express the colors of the picture to produce a well-finished song. Therefore, it is necessary to use sound sources of similar genres as SCCs. (2) The word viridi is meaningless to anyone who does not know Latin, but those who have studied Latin immediately think of green. In a similar way, VIVALDI and CLASSIC SCC are color codes expressed by sound, and visually impaired people learn these codes so that they can understand the color composition of the whole picture in a few minutes (e.g., it takes 3 min and 29 s in Table 4). Otherwise, it takes a lot of effort and time to read the tactile color pattern, as in [6], even with all 10 fingers, just like reading Braille.

2.3.1. Hue

The sound sources for the CLASSIC SCC set are solo performances by each chosen instrument that well express the characteristics and performant aspects of the instruments. Musical instruments expressing each color are thus classified and designated to make the colors easily distinguishable from one another. We chose the following instruments to represent the colors (hues) in the CLASSIC SCC.

Red, a warm color that gives a feeling of hot temperature, is a violin that plays a passionate and strong melody. A trumpet plays a melody in the high-frequency range with energy, as if bright light were expanding, to represent yellow. Orange is a viola playing a warm yet energetic melody. Green, which makes the eyes feel comfortable and psychologically stable, is a fresh tonal oboe that plays a soft melody. Blue, a cold color, is a cello that plays a low yet calm melody. Purple, a combination of warm red and cold blue, is a pipe organ that plays a splendid yet solemn melody.

2.3.2. Value and Chroma

In the tactile color pattern CELESTIAL [6], the light red pattern consists of three dots, the medium bright one consists of two dots, and the dark one consists of one dot. The light is symbolized as dots, and the more dots there are, the brighter it is. In a similar way, beat-heavy rhythms in SCC have brighter (i.e., higher value) colors. In addition, “saturated” conveys visual glare and a feeling of being close to me with an intense and clear melody in mid-tone. “Light” uses relatively high and fast notes to convey a light particle feel. “Dark” conveys the feeling of separation. Table 4 shows CLASSIC SCC V1 containing with wav sound source for each color.

2.4. Creating Artwork Music Using CLASSIC SCC V2

CLASSIC SCC V2 (Table 5), a modified version of CLASSIC SCC V1 (Table 4), was used to create music that expresses the composition of the overall color of the work. Thus, each color in the artwork is converted into sound, and then the converted sounds are combined into one piece of music.

Since the classic music used is different for each sound coding color, the composition will be different. Thus, to combine each sound coding color into one piece of music, it is necessary to unify the composition. The composition of all colors is unified with the key of F. F is suitable for the ranges of the instruments used. The cello’s two bars are in the same time as the oboe’s and the trumpet’s single bar. Thus, the tempo was adjusted as follows. Trumpet (yellow) and oboe (green) were allegro with tempos of 120 to 140 bpm, and adjusted to 78 bpm. The cello (blue) had a tempo between 150 and 170 bpm, and was adjusted to 156 bpm, which is twice as much as 78 bpm.

Rather than reproducing each piece of music in a row, it takes a characteristic melody from the sound code and makes them harmonize to form the music, so that color information and artistry can be saved at the same time. The sound strength (velocity) was adjusted to 120 for “saturated” and 50 for both “light” and “dark” so that they could be clearly distinguished.

For example, for Gogh’s starry night, a blue cello plays the bass and continues to play, and a yellow trumpet and a green oboe play the upper notes for the main melody.

Artwork music provides information about the approximate color placement and arrangement throughout a work of art before it separately detects the color of each local image within the piece. To provide sounds that correspond to the positions of colors on an image, each piece of artwork is divided into three or four tracks, and the color flow within each track is expressed as a continuous sound code. It is very important to identify the position of color on an image to form a standardized shape with a track of the same height. As shown in Table 6, we used the CLASSIC SCC to produce sheet music for Vincent van Gogh’s The Starry Night. We decomposed the artwork space into four tracks (rows) of equal height, analyzed the prominent color characteristics of each part, converted them using the SCC set, and recombined them into music with a total length of 3 min and 29 s.

The first track in Table 6 has a star in the blue sky and the moon on the far right. Thus, the cello (blue) plays a bass line, and the trumpet (yellow) and oboe (green) play the main melody. To combine each SCC into a single piece of music, consistency of composition is necessary. The composition of all colors was consistent in the key of F, and the tempo was set to 80/40 bpm. Yellow and green were adjusted to allegro, with a tempo of 120 to 140 bpm. The cello (blue) was set to vivace, between 150 and 170 bpm. All the sounds are consistent with a 4/4 beat.

As shown in Table 6, we used five colors: saturated and dark blue, saturated and bright yellow, and dark green. Thus, the blue night sky on the top track is played by the cello; the bright yellow stars and the moon are played on trumpets; and the dark green cypress tree visible vertically in the second to fourth tracks is played on oboes. As blue, green, and yellow are brought together, a feast of colors unfolds in the form of low and high notes combined to create beautiful music. The wind wriggling in the night sky is a cello with the shining stars in between. The soft glow that follows the wind is a trumpet, and the tranquil village in the bottom track uses a cello and an oboe to create the melodies given to each instrument. Accordingly, the blue color that dominates the overall hue of the picture appears as a cello that plays the bass of the entire song, and while moving the gaze to the right, as if reading a sheet of music from the top left; the visual feeling conveyed by each element that catches the eye is conveyed by the trumpet and oboe. You can feel it aurally through the melody shared between two instruments. A cymbal sounds at the end of every track to indicate which track is playing. This method not only allows the listener to analyze the individual elements that make up a piece of artwork, but also exhibits the completeness of the piece of art as a combination of various visual elements. It is significant in that it allows the overall feeling of a piece of artwork to be conveyed intuitively while retaining the image.

3. User Tests

This user test aimed to improve the recognition rate of the proposed SCC by evaluating and analyzing the cognitive accuracy, intuition, and texture of the sound.

3.1. Chord Coding Colors (CCC)

Fifteen college students took part in the exam, which began after participants spent 15 min learning each color corresponding to the sound of an instrument. Two 7” 2-Way Active Studio Monitor speakers were used to play MIDI sound clips by installing RME TotalMix FX v1.50 to control the hardware mixers and effects on an RME audio interface. Each MIDI sound clip was played to the participants, who were asked to identify the color corresponding to each clip. As the participants listened to the sound clips in random order, they were asked to report on the color hue and color attributes (saturated, light, dark) they associated with each sound. The stringed instruments, such as violin, cello, and viola, were difficult to distinguish from one another in the CCC SCC. Likewise, participants found it difficult to distinguish between the horn and oboe, and the oboe and trumpet.

Moreover, low frequency violins and high frequency cellos were difficult to distinguish. On the other hand, the match between yellow and trumpet was easy to remember. The violin (red) was associated with roughness, and the cello (blue) was deemed the softest. The primary color identification rate for each color was as follows: red (violin): 93%; blue (cello): 80%; yellow (trumpet): 93%. However, it was difficult to distinguish the secondary colors: orange (viola) = 53%; purple (horn) = 47%; green (oboe) = 47%.

The subjective feedback from test participants was as follows:

It would be better to express color with a melody.
It would be better to use a clearly distinguished instrument rather than a similar instrument such as the oboe and horn.
Mechanically regenerated sound is not natural.
If you only hear one sound, it seems difficult to distinguish colors. Listening and comparing multiple sounds at the same time could find a connection.

Thanks to this feedback, we decided that it would be more advantageous to distinguish musical instrument color matching by expressing a melody with a lot of musical personality than to distinguish only by chords.

3.2. CLASSIC SCC V1 vs. VIVLADI SCC

Participants were recruited into two groups: PVI musicians and sighted non-musicians. Eight sighted non-musicians participated in the test. The average age of the sighted participants was 22 years (range 20–25 years). The PVI musicians all had congenital blindness with no color experience. The musicians had all been involved in college-level music studies or had a significant amount of personal education. The non-musicians had not been involved in college-level music research and had no significant amount of personal education.

First, nine congenitally blind musician participants attended the test for CLASSIC SCC V1 and VIVLADI SCC. The average age of the participants was 26 years (range 22–30 years). The participants were given a one-hour tutorial (introduction to color basics and an explanation of how we made the VIVALDI and CLASSIC SCC sets). The sound source and user testimonial questions to be used in the experiment were distributed just before the experiment began. After orientation, participants were presented with sound clips corresponding to 18 colors in each SCC in random order. After listening to each color sound code, the participants evaluated their user experiences. Figure 6 shows the user experience evaluation scores that blind musicians gave the CLASSIC SSC V1 and VIVALDI SCC V1. The overall average scores for the two SCC sets were 3.09 (77%) and 3.47 (87%), respectively. As a result of the paired t-test between CLASSIC SCC v1 and VIVALDI SCC, “ease of use” (t = −4.0, p = 0.04) and “texture”(t = −2.63, p = 0.03), were statistically significant, but “usefulness” (t = −1.51, p = 0.169), “ease of learning” (t = −2.00, p = 0.081), and “satisfaction” (t = −1.51, p = 0.169) were statistically insignificant. As a result of the analysis, both SCC sets received good scores, but it was found that VIVLADI SCC should improve “sound texture” and CLASSIC SCC should further improve “ease of use.” We shall show the improved versions like CLASSIC SCC V2 and VIVLADI SCC V2 in the next sections.

3.3. VIVALDI SCC V1 vs. VIVALDI SCC V2

Table 7 lists the participants’ feedback after reviewing the two versions of VIVLADI SCCs. The user experience scores from the eight sighted participants were high (80%) for the VIVLADI SCC V1. As mentioned above, recall that scores from the nine congenitally blind musician participants were even higher (84%) for the VLIVLADI SCC V1. The sound texture of the VIVALDI SCC V1 is annoying because it was created in MIDI. This motivated us to enhance the quality of sound by recording the SCCs in a soundproof studio with the collaboration of a sound designer, composer, and performers. As seen from Table 7, the negative feedback, “It is difficult to distinguish due to differences between instruments,” became positive: “It was easy to distinguish and fun because only the instruments were different with the same melody.”

The PVI with musical experience may be more sensitive to frequency differences and differences in timbre between two signals and may have better short-term auditory memory than blind people with no musical experience [34]. Also, compared with sighted subjects, early blind subjects showed advantages in auditory spectral and temporal resolution, while late blind subjects showed an advantage in temporal resolution [35].

In the negative evaluations other than the positive user feedback in Table 7, non-musicians who evaluated VIVALDI SCC V1 said, “There are many unfamiliar instruments”; “There are instruments that are difficult to discern”; and “Color transmission is difficult.” Other said, “It’s difficult without appreciating art at the same time”; “In most cases, the sound mood doesn’t match the chosen color.” However, congenitally blind musician participants who participated in the VIVALDI SCC V2 evaluation provided all positive feedback.

The participant who provided negative feedback thought that in the case of a congenitally blind person without color experience, the color name would have a completely different meaning from what is visible, so it is sufficient to convert the image itself into a melody directly without converting it to a melody corresponding to the color of the image. This method has the advantage of enhancing the expressive power of one work, but also has the disadvantage that it cannot be applied consistently to other works.

Moreover, representing the image itself in sound is a much more difficult and complex process than using color names. The feeling that comes from an image or color is very subjective, so deciding which image to express with some sound is very difficult. Therefore, the current method of linking the characteristics of an instrument to the characteristics of color is objective and easy to access. Just as sighted people learn color names, blind people need to learn not only color names, but also musical codes that correspond to colors.

3.4. CLASSIC SCC V1 vs. CLASSIC SCC V2

Table 8 lists the user feedback for the CLASSIC SCC V2 after it had been updated to reflect the user feedback from the test results of CLASSIC SCC V1. User experience scores from the eight sighted participants averaged 79% for the CLASSIC SCC V2. As mentioned above, recall that scores from the nine congenitally blind musician participants were even higher (84%) for the CLASSIC SCC V1.

In addition to the feedback on CLASSIC SCC V1 in Table 8, we got the following: “It is difficult to change the pitch range for each instrument because a specific pitch range is not established”; “Due to the variety of instrumental accompaniment, the color of the instrument is not clear.” To solve this problem as a whole, we replaced the red, orange, and blue sound sources of CLASSIC SCC V1, Table 4, with the sound sources of CLASSIC SCC V2, Table 5.

There was negative feedback about the CLASSIC SCC V2: “The shorter the length of the sound segment representing each color, the better you can remember and distinguish S, L and D.” That is true, but in this case, when creating music that represents the color composition of the entire artwork, we should make sure that each segment is not too short in length to seamlessly connect the sound segments corresponding to each color.

3.5. VIVLADI SCC V2 vs. CLASSIC SCC V2

Three male participants (26 years old on average) with congenital blindness and no experience of color attended the test, in which we wanted to learn how easily the sounds of the VIVALDI SCC V2 and CLASSIC SCC V2 could be identified. Among them, two participants were musicians who played the trombone and guitar, respectively, in a chamber orchestra. After a 30 min tutorial orientation for both SCCs, the subjects were asked to listen to the sound sources threre times to familiarize themselves with the VIVALDI SCC V2. They were then asked to listen to the VIVALDI SCC V2’s 18 color sound codes in a random order, and they were asked to choose one of three color attributes (hue, saturation, or light and dark) that corresponded to each code. Next, the same procedure was used to test the CLASSIC SCC V2. Both SCCs scored 100% on the identification test.

3.6. Vincent van Gogh’s The Starry Night with CLASSIC SCC V2

The user experience score evaluated from eight sighted students for artwork music composition of “Vincent van Gogh’s The Starry Night” applied with CLASSIC SCC V2 was 83.5%.

The feedback on the composition of Vincent van Gogh’s The Starry Night has all been very positive as follows:

The purpose is clearly necessary and useful in expressing color vision through hearing. I really want to recommend it.
It takes a lot of imagination.
It is easy to remember because it can remind us of a rough expression of color in your head. If the rules of the code are maintained but composed of various melodies, the beauty of the work will be conveyed well. It also shows variety.
It is clear and simple, but even looking at one piece of artwork, there seem to be many people making different music with similar pitches.
The division of each part of the artwork was expressed with cymbals, and it was good to appreciate.
When it was arranged with the image of Van Gogh, it was clearer because it was suitable music.
I am satisfied because it goes well with art, and I like music.

4. Conclusions

In this paper, we presented a sound coding color (SCC) design methodology to ascertain whether a person with a visual impairment (PVI) can interact with color through sound without a complex learning process. Although several researchers have done color–sound matchings previously, to the best of our knowledge, no one has experimented with whether PVI can distinguish the color–sound matchings effectively. In this paper, a series of user tests was performed on increasingly refined SCCs developed through user feedback. During the test, the user experience evaluation rate from nine blind musicians for the CLASSIC SSC V1 and VIVALDI SCC V1 was 77% and 84%, respectively. The user experience scores from eight sighted participants were 79% and 80% for CLASSIC SCC V2 and VIVLADI SCC V1, respectively. The music composed to convey the color arrangement of Van Gogh’s starry night using CLASSIC SCC V2 received a user experience rating of 83.5%. Finally, after training three congenitally blind adults in both VIVALDI SCC V2 and CLASSIC SCC for about one hour, the recognition rate for both CLASSIC SCC V2 and VIVLADI SCC V2 was 100%. Therefore, the CLASSIC and VIVALDI SCCs helped participants appreciate the overall color harmony of artwork. In [34], in pitch perception, the congenitally visually impaired group was found to be superior to the acquired visually impaired and sighted groups. The “time change analysis ability” of sound was also found to be superior in the congenitally visually impaired group and the acquired visually impaired group than in the sighted group. In [35], early blindness was linked to enhanced perception of the auditory world, including pitch perception. We shall leave extra experiments on more PVI who are not musicians as future work, along with an experiment to find out the differences in perception among various levels of visually impaired and sighted people regarding our proposed SCC sets. Additionally, as a future study, we shall explore the simultaneous cognition ability of PVI when dealing with both color–temperature code [36] and the color–sound code presented in this paper.

Supplementary Materials

The following are available online at https://www.mdpi.com/2079-9292/9/11/1981/s1.

Author Contributions

Conceptualization, J.D.C. and J.J.; methodology, J.D.C., J.J., H.L. and J.H.K.; validation, J.D.C., H.L. and J.H.K.; formal analysis, J.D.C.; investigation, J.D.C., J.J. and J.H.K.; resources, J.D.C., J.J. and J.H.K.; data curation, J.D.C., J.J. and J.H.K.; writing—original draft preparation, J.D.C. and J.J.; writing—review and editing, J.D.C.; visualization, J.D.C.; supervision, J.D.C.; project administration and funding acquisition, J.D.C. All authors have read and agreed to the published version of the manuscript.

Funding

This research was funded by the Science Technology and Humanity Converging Research Program of the National Research Foundation of Korea, grant number 2018M3C1B6061353.

Acknowledgments

We would like to thank all volunteers for their participation and the reviewers for insights and suggestions.

Conflicts of Interest

The authors declare that they have no conflict of interest.

References

Axel, E.S.; Levent, S.L. Art beyond sight: Guidelines for making tactile diagrams & standard tactile patterns, lines, and icons. Art Education for the Blind; AFB Press: Arlington, VA, USA, 2003. [Google Scholar]
Hayhoe, S. Blind Visitor Experiences at Art Museums; Rowman & Littlefield: New York, NY, USA, 2017. [Google Scholar]
San Diego Museum of Art Talking Tactile Exhibit Panel. Available online: http://touchgraphics.com/portfolio/sdma-exhibit-panel/ (accessed on 22 November 2020).
Bartolomé, J.I.; Quero, L.C.; Kim, S.; Um, M.-Y.; Cho, J. Exploring art with a voice controlled multimodal guide for blind people. In Proceedings of the Thirteenth International Conference on Tangible, Embedded, and Embodied Interaction, Tempe, AZ, USA, 17–20 March 2019; ACM: New York, NY, USA, 2019; pp. 383–390. [Google Scholar] [CrossRef]
Quero, L.C.; Bartolomé, J.I.; Lee, S.; Han, E.; Kim, S.; Cho, J. An interactive multimodal guide to improve art accessibility for blind people. In Proceedings of the 20th International ACM SIGACCESS Conference on Computers and Accessibility, Galway, Ireland, 22–24 October 2018; ACM: New York, NY, USA, 2018; pp. 346–348. [Google Scholar] [CrossRef]
Cho, J.D.; Bartolomé, J.I.; Quero, L.C.; Lee, D.; Oh, U.; Lee, I. Tactile colour pictogram to improve artwork appreciation of people with visual impairments. Color Res. Appl. 2020. [Google Scholar] [CrossRef]
Color Quoted by Paul Gauguin. Available online: https://www.sensationalcolor.com/gauguin-quotes-color/ (accessed on 22 November 2020).
Munsell Color. Available online: http://munsell.com (accessed on 22 November 2020).
Palmer, S.E.; Schloss, K.B. An ecological valence theory of human color preference. Proc. Natl. Acad. Sci. USA 2010, 107, 8877–8882. [Google Scholar] [CrossRef] [PubMed] [Green Version]
Martino, G.; Marks, L.E. Synesthesia: Strong and weak. Curr. Dir. Psychol. Sci. 2001, 10, 61–65. [Google Scholar] [CrossRef]
Marks, L.E. On colored-hearing synesthesia: Cross-modal translations of sensory dimensions. Psychol. Bull. 1975, 82, 303–331. [Google Scholar] [CrossRef] [PubMed]
Odebert, H.S.; Karwoski, T.F.; Eckerson, A.B. Studies in synesthetic thinking: I. Musical and verbal associations of color and mood. J. Gen. Psychol. 1942, 26, 153–173. [Google Scholar] [CrossRef]
Karwoski, T.F.; Odebert, H.S. Color-music. In Psychological Monographs: General and Applied; The Pshychological Review Company: Columbus, OH, USA, 1938. [Google Scholar]
Kandinsky, V. Concerning the Spiritual in Art; Dover Publications: Mineola, NY, USA, 1977. [Google Scholar]
Haack, P.A.; Radocy, R.E. A case study of a chromesthetic. J. Res. Music Educ. 1981, 29, 85–90. [Google Scholar] [CrossRef]
Barbiere, J.M.; Vidal, A.; Zellner, D.A. The color of music: Correspondence through emotion. Empir. Stud. Arts 2007, 25, 193–208. [Google Scholar] [CrossRef]
Jonas, C.; Spiller, M.J.; Hibbard, P. Summation of visual attributes in auditory–visual crossmodal correspondences. Psychon. Bull. Rev. 2017, 24, 1104–1112. [Google Scholar] [CrossRef] [PubMed]
Newton, I. Opticks or a Treatise of the Reflections, Refractions, Inflections & Colors of Light; Dover Publications, Inc.: New York, NY, USA, 1952. [Google Scholar]
Maryon, E. MARCOTONE The Science of Tone-Color; Birchard Hayes & Company: Boston, MA, USA, 1924. [Google Scholar]
Peacock, K. Synesthetic perception: Alexander Scriabin’s color hearing. Music Percept. 1985, 2. [Google Scholar] [CrossRef]
Lavignac, A. Music and Musicians; Henry Holt and Company: New York, NY, USA, 1903. [Google Scholar]
Anikin, A.; Johansson, N. Implicit associations between individual properties of color and sound. Atten. Percept. Psychophys. 2019, 81. [Google Scholar] [CrossRef] [PubMed]
Hamilton-Fletcher, G.; Witzel, C.; Reby, D.; Ward, J. Sound properties associated with equiluminant colors. Multisens. Res. 2017, 30, 337–362. [Google Scholar] [CrossRef] [PubMed]
Giannakis, K. Sound Mosaics: A Graphical User Interface for Sound Synthesis Based on Audio-Visual Associations. Ph.D. Thesis, Middlesex University, London, UK, 2001. [Google Scholar]
Cogan, R.D. Sonic Design: The Nature of Sound and Music; Prentice Hall: Upper Saddle River, NJ, USA, 1976. [Google Scholar]
Marks, L.E.; Hammeal, R.J.; Bornstein, H.M. Perceiving similarity and comprehending metaphor. Monogr. Soc. Res. Child Dev. 1987, 52, 1–102. [Google Scholar] [CrossRef] [PubMed]
Deville, B.; Bologna, G.; Vinckenbosch, M.; Pun, T. See color: Seeing colors with an orchestra. In Human Machine Interaction; Springer: Berlin/Heidelberg, Germany, 2009; pp. 251–279. [Google Scholar]
Cavaco, S.; Henriquesbc, J.T.; Mengucci, M.; Correia, N.; Medeiros, F. Color sonification for the visually impaired. Procedia Technol. 2013, 9, 1048–1057. [Google Scholar] [CrossRef] [Green Version]
Donnell-Kotrozo, C. Intersensory perception of music: Color me trombone. Music Educ. J. 1978, 65, 32–37. [Google Scholar] [CrossRef]
Köhler, W. Gestalt psychology. In An Introduction to New Concepts in Modern Psychology; Liveright Publishing Corporation: New York, NY, USA, 1947. [Google Scholar]
Adeli, M.; Rouat, J.; Molotchnikoff, S. Audiovisual correspondence between musical timbre and visual shapes. Front. Hum. Neurosci. 2014, 8, 252. [Google Scholar] [CrossRef] [PubMed] [Green Version]
Parise, C.V.; Spence, C. Audiovisual crossmodal correspondences and sound symbolism: A study using the implicit association test. Exp. Brain Res. 2012, 220, 319–333. [Google Scholar] [CrossRef] [PubMed]
Reuter, C.; Kruchten, S. Colors and Timbres—Consistency and Tendencies of Color-Timbre Mappings in non-synesthetic Individuals. In Proceedings of the “Musik im audiovisuellen Kontext”—34. Jahrestagung 2018 der Deutschen Gesellschaft für Musikpsychologie (DGM), Universität Gießen, Giessen, Germany, 7–9 September 2017. [Google Scholar]
Bogusz-Witczak, E.; Skrodzka, E.; Turkowska, H. Influence of musical experience of blind and visually impaired young persons on performance in selected auditory tasks. Arch. Acoust. 2015, 40, 337–349. [Google Scholar] [CrossRef] [Green Version]
Shim, H.J.; Go, G.; Lee, H.; Choi, S.W.; Won, J.H. Influence of visual deprivation on auditory spectral resolution, temporal resolution, and speech perception. Front. in Neurosci. 2019, 13. [Google Scholar] [CrossRef] [PubMed] [Green Version]
Iranzo Bartolomé, J.; Cho, J.D.; Cavazos Quero, L.; Jo, S.; Cho, G. Thermal Interaction for Improving Tactile Artwork Depth and Color-Depth Appreciation for Visually Impaired People. Electronics 2020, 9. [Google Scholar]

Figure 1. Red (R), orange (O), yellow (Y), green (G), blue (B), and purple (P) using three color dimensions—saturated (S), light (L), and dark (D).

Figure 2. Saturated (S), light (L), and dark (D) for red in [9].

Figure 3. Original track scores for sound coding saturated (left), light (middle), and dark (right) colors from Vivaldi’s The Four Seasons.

Figure 4. Modified scores for sound coding saturated (left), light (middle), and dark (right) colors played with a violin (same for trumpet and clarinet).

Figure 5. Modified scores for sound coding saturated (left), light (middle), and dark (right) colors played with a guitar (same for piano and organ).

Figure 6. User experience evaluation: strongly disagree (one point)–strongly agree (four points).