Improving Human–Computer Interface Design through Application of Basic Research on Audiovisual Integration and Amplitude Envelope

Abstract

Quality care for patients requires effective communication amongst medical teams. Increasingly, communication is required not only between team members themselves, but between members and the medical devices monitoring and managing patient well-being. Most human–computer interfaces use either auditory or visual displays, and despite significant experimentation, they still elicit well-documented concerns. Curiously, few interfaces explore the benefits of multimodal communication, despite extensive documentation of the brain’s sensitivity to multimodal signals. New approaches built on insights from basic audiovisual integration research hold the potential to improve future human–computer interfaces. In particular, recent discoveries regarding the acoustic property of amplitude envelope illustrate that it can enhance audiovisual integration while also lowering annoyance. Here, we share key insights from recent research with the potential to inform applications related to human–computer interface design. Ultimately, this could lead to a cost-effective way to improve communication in medical contexts—with signification implications for both human health and the burgeoning medical device industry.

1. Introduction

The appropriate design of human–computer interactions plays a crucial role in harnessing the powerful capabilities of electronic devices. Research on visual [1,2], and auditory [3,4] interfaces illustrates the importance of careful attention to the design of unimodal displays. Although relatively little research explores the efficacy of multimodal systems for human–computer interactions, psychologists and neuroscientists routinely illustrate the perceptual benefits of multimodal processing [5]. As a contribution to this special issue, we summarize current theories on these disparate but complementary areas of inquiry, highlighting recent discoveries related to acoustic properties facilitating audiovisual integration. Rather than a comprehensive overview, our goal is to contribute new ideas for interface design by illuminating points of potential interest through the practical applications of basic research.

2. Multimodal Processing

Our brains interpret incoming stimuli from various senses (i.e., cross-modal stimuli) to form a unified perception of our surroundings. One way of formally documenting and exploring this integration is by assessing neural responses to unisensory versus multisensory stimuli. Processing at the level of the neuron is measured by recording changes in neuronal membrane potential in response to external events. When the membrane potential exceeds a certain threshold value, action potentials (APs) are generated. The neuronal response elicited by a stimulus is positively correlated to the firing rate and frequency of APs. Comparing the neuronal response to cross-modal stimuli (e.g., audiovisual) with the neuronal response to its unimodal components (e.g., the auditory and visual components) [6] sheds light into the nature of multimodal integration. Responses to multimodal stimuli larger than their unimodal components indicate multisensory enhancement, and neurons demonstrating such enhancements are considered “multisensory neurons” [7].

Multisensory enhancement is often inversely related to the effectiveness of the unimodal cues [8]. For example, noticing a cat approaching can involve both peripheral sight (a weak visual cue) and hearing soft footsteps (a weak auditory cue). Independently, these weak, unimodal cues are unlikely to attract interest; yet, when presented together (i.e., as a multimodal stimulus), they are more likely to capture attention. This is seen at the neuronal level as weak, unimodal cues combine together to produce a cross-modal response that is much larger than the sum of the responses to each of the unimodal cues—A super-additive response [7,8,9].

Conversely, strong unimodal cues evoke strong responses on their own. For instance, detecting a barking dog running towards you is easy to notice as the barking (a strong auditory cue) is sufficient in grabbing your attention; there is likely no added benefit in observing the dog (a strong visual cue). This is seen at the neuronal level when strong unimodal cues are combined together. Their cross-modal response is likely to be less than the sum of the responses to each of the unimodal cues, referred to as a sub-additive response [7,8,9]. This pattern of multisensory enhancement inversely related to unimodal effectiveness is known as the principle of inverse effectiveness [7]. This holds potential applications in the design of multi-modal alarms in medical settings as keeping individual alarm signals as weak as possible (while still detectable) helps to prevent sensory overload from multiple concurrent systems—An application which may be of use to some members of a medical team but not others.

2.1. Lower-Order Multimodal Integration (Stimulus Orientation)

Sound localization provides a useful example of multimodal signal enhancement. Stimulus orientation and stimulus localization are mediated by the superior colliculus (SC). The SC is a mid-brain structure implicated in multisensory integration and generation of spatial orienting responses. Although many brain regions are implicated in multisensory integration, the convergence of inputs from various sensory modalities (e.g., visual, auditory, somatosensory) [6,10], projections to motor areas [10,11], and the abundance of multisensory neurons [12] make the SC a natural location for illustrating multimodal processing on a neural level. Multisensory enhancement in the SC occurs in many animals, including cats [8,13], rats [14], ferrets [15], and primates [16], decreasing response times for stimulus orientation and localization behaviors. The SC’s role is preserved in humans, mediating multisensory spatial integration [17], thereby aiding faster response generations. Extending beyond simple stimulus orientation, there is some evidence that the SC facilitates primitive social behaviors such as facial mimicry in neonates [18].

2.2. Higher-Order Multimodal Integration (Perceptual Binding)

Lower-order multimodal processes are useful in understanding the neural basis of multimodal integration. However, understanding higher-order multimodal integration is crucial in our interpretation of the complex stimuli encountered in our daily lives and holds useful lessons for the design of multimodal interfaces. For example, verbal communication was once thought to be a purely auditory process; yet, it involves the perceptual binding of watching a speaker’s lip movements (e.g., vision) and hearing the speech sounds produced (e.g., audition) [19]. The well-known McGurk effect [20] clearly illustrates vision’s influence on speech perception, an effect that is magnified when semantic constraints are imposed. When lip movements are incongruent to the speech produced (e.g., lips pronounce ‘bows’ and speech produced is ‘goes’), an intermediate between the lip movements and heard speech (e.g., ‘doze’) is perceived; however, when the visual is removed, this effect disappears and speech sounds are heard accurately [20].

Comparative studies in primates implicate the role of the superior temporal sulcus (STS) in audiovisual integration in response to face and speech sounds [21]. Similarly in humans, the STS has been implicated as the primary site for higher-order audiovisual processing [22,23,24,25]. One study using functional magnetic resonance imaging (fMRI) revealed an increase in the blood oxygen level-dependent signal in the STS in response to temporally aligned audiovisual stimuli but not to audio-only or visual-only stimuli [25], highlighting the multimodal capabilities of the STS. To establish a causal relationship between the STS and audiovisual integration as observed in the McGurk effect, one study used fMRI-guided transcranial magnetic stimulation (TMS) to create temporary virtual lesions while participants observed McGurk and control stimuli [26]. Creating a temporary lesion to the STS significantly decreased the McGurk effect [26]. These results were corroborated by another study using transcranial direct current simulation (tDCS), a non-invasive neuromodulatory technique; cathodal tDCS (decreases excitability) applied over the STS showed a disruption to the McGurk effect, whereas anodal tDCS (increases excitability) applied over the STS showed an increase in the McGurk effect [27].

In addition to its role in speech, the STS also plays a crucial role in the audiovisual integration of non-speech sounds. For example, in the sound-induced fission illusion [28], a single flash is perceived to be multiple flashes when paired with multiple auditory beeps. When anodal tDCS is applied over the STS during the sound-induced fission illusion, there is an increase in perceived fission whereas cathodal tDCS applied over the STS results in a decrease in perceived fission [29]. These studies stress the important role of the STS in higher-order audiovisual processing for both speech and non-speech sounds.

2.3. Gains in Performance from Multisensory Stimulation

Basic research illustrates that multisensory stimulation alters neural and perceptual responses. This raises important questions about the potential benefits of multimodal presentations in human–computer interfaces. Although a full assessment requires explicit tests of future interfaces, previous research exploring generalized improvements in multimodal presentations offers useful insights. Each of the modalities brings different strengths, with audition offering superior temporal resolution and vision providing better spatial resolution [30]. When these modalities are used in tandem, audiovisual interfaces offer greater temporal and spatial resolution than either audio-only or visual-only interfaces. Furthermore, the use of auditory signals (both spatially correlated and non-spatial) during visual search tasks facilitates greater target saliency [31] and faster target identification [32,33,34].

Multisensory stimulation can also enhance the performance of a single primary modality. For example, bimodal presentation of visual and tactile motion stimuli results in faster detection of motion stimuli [35]. These kinds of effects can be used to enhance graphical user interfaces (GUIs), especially for those with disabilities. Scanning input is a visual task where users with physical disabilities (i.e., unable to operate a mouse) identify a required target by visually scanning items in the form of a grid. One study found that a sonically enhanced version of scanning input increased motivation and user engagement, with the potential to also increase scanning speed [36]. Additionally, the use of multimodal presentations in common GUI functions (e.g., drag and drop) have the potential to reduce perceived mental workload without affecting performance [37,38].

3. Amplitude Envelope and Alarm Design

Literature on multimodal processing illustrates the potential for improving human–computer interface design by drawing upon multiple modalities of presentation. However, this will require designers to consider the best ways to organize information across modalities in a way conducive to effective integration. To that end, we see potential for building on recent discoveries from our team with respect to amplitude envelope. Amplitude envelope (henceforth “envelope”) refers to a sound’s shape—i.e., its amplitude profile over time. Sounds with flat envelopes (Figure 1, right panel) feature abrupt offsets offering little information about the materials (i.e., metallic/wooden) involved in sound producing events. In contrast, percussive sounds originating from impact events often exhibit relatively long, decaying offsets informing listeners about the materials involved in the event, such as the hollowness of the struck material [39]. Many auditory interfaces in medical devices, such as those mandated by the International Electrotechnical Commission (IEC), involve melodic alarm tones that use sounds with flat envelopes [40], and these alarm systems are widely recognized as problematic [40,41].

Although tradeoffs between envelope and other properties of tones have long fascinated musicians given the complexity of synthesizing musically satisfying sounds [42], psychologists have generally opted for study and manipulation of isolated parameters such as duration, frequency, and amplitude [43]. These parameters are more well-suited for careful control and manipulation, particularly with the advent of modern computers [44]. This careful focus on easily controlled parameters has been helpful in clarifying the importance of some low-level properties of sound, such as onset [45,46]. Onset plays an important role in distinguishing between musical timbres [47], to the point where removing the onset entirely renders otherwise distinct instruments indistinguishable [48]. Additionally, a lack of sensitivity to tone onset predicts deficits in reading [49].

Curiously, onset is the one temporal parameter that is relatively consistent between flat and percussive tones, which both feature abrupt rises. These tones differ in their sustain and most notably their offset segments (Figure 1), a parameter that has not often been overlooked in importance within the field of auditory psychophysics [50]. Nonetheless, differences in offset can trigger qualitatively different patterns regarding the perception of duration [51,52,53,54,55], loudness [56,57,58], and loudness change [59,60].

3.1. A Demonstration of Envelope Affecting Audiovisual Integration

One example of envelope’s role in the assessment of event duration is particularly pertinent to multimodal interfaces. Research on audiovisual integration using flat tones has generally concluded that vision rarely influences auditory evaluations of duration (provided that the acoustic information is of sufficient quality [30,61,62]. However, a novel musical illusion illustrates that this long-standing conclusion does not hold in some contexts involving percussive sounds. For example, percussionists are able to manipulate audience perception of note duration by using long versus short motions when striking their instrument. Although these gestures fail to alter notes’ acoustic structures, audiences observing these long or short striking motions perceive these notes to sound either long or short [63]. Aside from resolving a long-running debate in the percussion community [64], this finding illustrates a novel documentation regarding a visual influence on the auditory perception of duration. This finding contrasts markedly with previous findings that vision does not affect auditory duration assessments [30,61,62].

This novel pattern of processing led to a new understanding of audiovisual integration when using sounds with percussive envelopes. For example, this variation of this illusion illustrates it holds with other sounds produced by impact events (i.e., striking an instrument), but not sustained events such as blowing into a mouthpiece [65]. Although several factors play a role in this binding, one key factor is the amplitude envelope of notes produced by the marimba—A percussion instrument similar to a xylophone made of wooden bars struck by performers using mallets. These percussive sounds bind with striking gestures (producing impact sounds), whereas other sustained sounds such as those produced by bowing a cello do not [66]. Crucially for the design of interfaces with synthesized sounds, even simple pure tones (sine waves) shaped with decaying envelopes appear to trigger this privileged binding [67]. In contrast, pure tones shaped with flat envelopes fail to integrate with the same visual information [68].

Extensions to this research illustrate that short sound sequences (similar to those used in auditory alarms) shaped with percussive envelopes are easier to associate with everyday objects [69]. These associations appear to be both learned and retained better when using percussive instead of flat sounds. Although that study involved physical objects, the general question of which types of sounds are best associated is highly relevant to the design of effective auditory alarms as users need to learn and retain associations between sound sequences and commands.

3.2. Sounds Currently Used in Medical Device Alarms

The bias towards the use of flat tones in auditory interfaces is consistent with a bias towards such sounds in auditory perception research [70]. We suspect this is largely due to their high degree of experimental control [44], as their temporal structure can be easily and precisely specified using a minimum number of parameters.

However, using stimuli lacking the dynamic temporal changes (inherent in natural sounds) is problematic as they may be processed with different underlying processing strategies [71]. This poses major challenges for generalizing from controlled research studies to real-world applications, creating challenges both for theoretical and applied work. For example, theories derived from a large literature of audiovisual integration using flat tones fail to generalize to sounds with natural shapes. Experiments with flat tones have repeatedly concluded that vision does not affect auditory judgments of duration; however, vision can have a substantial influence when the sounds exhibit natural decays [65,67,68].

The use of flat tones also poses challenges in applied contexts when considering human factors related to auditory interfaces. For example, flat tones sound less pleasing than percussive tones, lowering the perceived value of products in which they are used [72]. Additionally, they are perceived as significantly more annoying than similar sequences of percussive tones [73]. These results help explain repeated previous findings that current alarms (based heavily on flat tones) are problematic for medical professionals [74] who hear hundreds of alarms throughout their workday [75]. Although the pleasantness of alarms may seem a secondary issue, it plays an important role in their use as users might disable unpleasant auditory alerts even if they are actually informative [76]—rendering them ineffective as a consequence of auditory aesthetics, rather than the appropriateness of their signals.

Problems with some current alarms are so widely recognized that those involved with their creation have issued formal apologies in the peer-reviewed literature [77]. Many of these alarms by definition require flat envelopes [40] forcing alarm designers into using sounds that are hard to learn [40,41] and not conducive to integration with visual information. In contrast, percussive sounds integrate more readily with visual information [66,68], are aesthetically preferable [72] and are perceived as significantly less annoying [73].

3.3. The Use of Percussive Tones in Multimodal Interfaces

Sounds with percussive envelopes integrate more strongly with visual information when assessing duration [68] and event unity [78], and are more easily associated with everyday objects [69]. Together, these findings hold important implications for the design of auditory alarms, which require users to learn and retain associations between sound sequences and commands. This is intriguing, given well-known problems with learning [40,41], retention [41,79], and confusion [41,80] of many current approaches. As the field evolves to more fully explore the possibilities of multimodal alarms for medical devices, we encourage exploration of envelope as a way to simultaneously increase audiovisual integration, lower annoyance, improve aesthetics, and offer better user experiences with human–computer interfaces.

Given the well-documented perceptual “gains” of stimulation in multiple modalities (Section 2) as well as the benefits of redundancy in important signals related to the critical issue of patient health, we see the pursuit of multimodal alarm systems as a potentially fruitful area for future research. Although these principles could be of use in any human–computer interface, they are of particular relevance for medical alarm design. Although Industry Canada values the medical device market at CAD $6.7 billion, and USD $336 billion annually [81], the design of auditory alarms in these devices is fraught. Yet, such devices’ instruments play an increasingly important role in patient care, with hundreds of alarms sounding per patient per day in busy hospitals [75].

To aid researchers interested in generating dynamically changing sounds, we have posted a free tool online allowing for the synthesis of percussive and flat tones of any duration and frequency useful in experimental contexts [82]. We have used this tool for several experiments in our lab as it offers a simple interface for precise stimulus generation. Although envelope is slowly gaining increased attention as a research topic for theoretical explorations [57,60,72,73,78], its applications in interface design have been relatively underexplored to date. As our interest in this property came directly from observing its surprising importance in audiovisual integration tasks [63,65,68,78], we are intrigued by its potential applications in improving multimodal interfaces in medical devices. Consequently, we are pleased to contribute to this special issue focused on raising awareness of this important topic.