Design Dimensions of Co-Located Multi-Device Audio Experiences

Abstract

The widespread distribution of mobile computing presents new opportunities for the consumption of interactive and immersive media experiences using multiple connected devices. Tools now exist for the creation of these experiences; however, there is still limited understanding of the best design practices and use cases for the technology, especially in the context of audio experiences. In this study, the application space of co-located multi-device audio experiences is explored and documented through a review of the literature and a survey. Using the obtained information, a set of seven design dimensions that can be used to characterise and compare experiences of this type is proposed; these are synchronisation, context, position, relationship, interactivity, organisation, and distribution. A mapping of the current application space is presented where four categories are identified using the design dimensions, these are public performances, interactive music, augmented broadcasting, and social games. Finally, the overlap between co-located multi-device audio and audio-augmented reality (AAR) experiences is highlighted and discussed. This work will contribute to the wider discussion about the role of multiple devices in audio experiences and provide a source of reference for the design of future multi-device audio experiences.

1. Introduction

For many individuals in modern 21st century society, daily life includes many interactions with a number of different mobile computing devices. This is a result of the proliferation of such devices, including smartphones and tablets, over the last two decades. Moreover, an increasing number of individuals now own multiple such devices. New types of ‘smart’ devices are also emerging which diverge from traditional form factors, such as smart-watches and smart-eyewear [1]. While these devices appear physically different, many have similar hardware features which can enable a wide array of applications, especially through software ‘apps’. As a consequence, there is an increasing number of devices that have different form factors but overlap in their possible applications.

The device that is selected for use at any given time depends on the chosen activity and various contextual factors such as location and social surroundings. However, it is increasingly common to use multiple devices at the same time, either for separate activities or in a connected, collaborative capacity. Research exploring the use of many connected devices to accomplish a task has become popular, most notably under the banner of the Internet of Things (IoT) [2], and aims to harness the power of intentional multi-device networking, which can bring valuable applications to users [3].

One such application is the use of multiple connected devices to consume and interact with audiovisual media content. This approach has benefits which includes the provision of additional content and greater interaction possibilities, especially in social contexts. For audio experiences, using multiple commodity devices can be an accessible, cost-effective method for spatial or immersive audio playback and can also enable straightforward participatory interaction within experiences. In the last five years, the way these devices can be utilised in the design of audio experiences has started to be established through case studies and early demonstrations [4,5]. However, in order to develop the application space further, more work is needed on experience characterisation and design to identify new experience opportunities and to increase our conceptual understanding of the experiences. This is necessary, as much of the research on multi-device experience design until now has focused on the visual modality. Furthermore, the work across multi-device audio experiences suffers from an inconsistent use of terminology, making navigating the literature quite difficult. Consequently, this work addresses the following questions.

• What are the defining characteristics of co-located multi-device audio experiences?
• How do these characteristics manifest in different applications?

A survey was carried out to obtain a dataset which included descriptions of co-located multi-device audio experiences. The survey was then analysed to retrieve information on design aspects and attributes. A set of resulting design dimensions was generated and presented to aid the understanding and characterisation of co-located, multi-device audio experiences. In addition, the application space was reviewed and mapped into example categories using a method in which use-cases can be categorised and new instances can be discovered.

This paper is organised as follows. In Section 2, a review of the current state of multi-device experience design is given, alongside a brief summary of the development of the multi-device audio medium, including relevant production tools and applications. Within Section 3 and Section 4, the methodology for the survey and the results are detailed. The design dimensions are presented and discussed in Section 5, followed by an analysis and review of the applications in the dataset in Section 6. In Section 7, the use of the design dimensions, and the overlap between the applications and audio-augmented reality (AAR) applications is discussed. Finally, a summary of the work is provided and future work discussed in Section 8.

2. Related Work

Multi-device ecosystems and their applications are complex and encounter many additional challenges compared to their single-device counterparts. To tackle this complexity, and to enable the creation of more meaningful multi-device experiences, research has been conducted to increase our understanding, more broadly, of multi-device interactions and user behaviours.

Levin [6] presents a set of high-level design approaches to multi-device experiences, named the 3Cs, which consist of consistent, continuous, and complementary methods. These act as a practical guide for designing multi-device interactions for media experiences and provide a useful introduction for newcomers to the research space. As this research area has progressed and new prototypes have emerged, efforts have been made to consolidate the work through surveys and taxonomies. An excellent example of this is presented by Brudy et al. [3], who report on design characteristics, applications, tracking systems, and interaction techniques for multi-device systems. Moreover, for contemporary multi-device audiovisual systems, Bown and Ferguson [7] propose the term ‘media multiplicities’ to describe coordinating systems of devices that interact to manifest media content and present four key affordances: spatial, where the devices are distributed in physical space; scatterable, where devices can move around and be reconfigured; sensing, devices can interact and coordinate using sensors; and scalable, where devices can be added and removed freely.

While many of the discussed concepts can be applied in audio-only applications, most of the discussion is centered around multi-screen applications. This follows a common trend of research focusing on the visual modality, exploring applications such as ‘second screen’ viewing [8,9,10] and augmented-reality TV [11,12]. In addition, while many experiences are comprised of both visual and auditory stimuli, it is also valid to consider audio and visual experience aspects separately, due to the differences via which both of these modalities are perceived by the human body.

The use of multiple devices for audio reproduction is most commonly applied in music-based performances, for instance, where the audience’s devices are used as an array of loudspeakers. The bulk of this research occurred over the last decade with the rapid uptake of smartphones; however, this practice can be observed as far back as the 1970s, where analogue radios and cassette players have been used in experimental sound–art installations and exhibits [13,14,15]. Since then, mobile/laptop orchestras have been a popular avenue for speaker-array experimentation [4,16]. Dahl [17] described designing these experiences as a ‘wicked problem’, where creators must design the instrument and the interactions between devices, as well as composing the music. To address this complexity, a prototyping and iteration design approach is commonly chosen [18].

With the proliferation of personal computing devices, a number of tools have been developed to reduce the difficulty in creating effective, cross-platform, multi-device audio experiences. Soundworks, for example, is a JavaScript framework for creating multi-device audio experiences for consumer devices and has been used in many demonstrations of participatory sound and music experiences [19]. Audio Orchestrator is a production tool which allows producers to create multi-device audio experiences by facilitating audio routing to any number of devices through an allocation algorithm, which can be controlled through defined behaviours [20]. Using this tool, a number of trial productions have been made ranging from immersive audio dramas [21] to sports programs [22]. A final example is HappyBrackets, a programming environment for the creative coding of multi-device systems, that can also be used as a composition and performance tool for audio experiences [23].

Similar to many other developing research areas, the disjointed use of terminology is also a problem in this field, where many different terms are used to describe similar ideas or concepts. For example, device orchestration [21], media multiplicities [7], and mobile multi-speaker audio [24] can all refer to the use of multiple devices for audio reproduction. Finding some consensus on this issue would help to improve literature accessibility and may lead to more fruitful collaboration across this discipline. As noted above, the research into multi-device experiences has focused mainly on multi-display applications and includes a number of review papers capturing the breadth of the work [3,9,25]. In contrast, far fewer similar studies have been conducted in the audio domain, aside from Taylor’s review of speaker array applications [15]. More effort is therefore necessary to capture and consolidate the knowledge around the applications of multiple devices for audio experiences.

3. Methodology

3.1. Scope of the Research

This paper is concerned only with co-located experiences, where the active devices are within the same physical space or group of adjoining spaces as the listener/audience. This excludes typically remote applications such as teleconferencing and streaming parties. Furthermore, a subset of multi-device audio experiences, where one or more devices control the audio output of a single device (such as the use of an additional controller with a smart instrument [26] or streaming media from one device to another) are also outside the scope of this paper.

During this period of work, attempts have been made to specifically define the word ‘device’ in the context of this work. However, no satisfactory definition was found; they were either too encompassing or too specific. Nonetheless, the loose criteria for devices in this work are that they are standalone, reproduce audio, and offer some form of interface for interaction. For this reason, multi-channel surround experiences have not been considered as co-located, multi-device audio experiences for this study, as they are essentially deemed to constitute a single device. A single loudspeaker may be used in isolation; however, they are designed to be used in multiples. A particular use case, somewhere near the definition boundary of these experiences, that was included in this study was silent films or discos. This type of experience differs from multi-channel audio experiences in two ways. Firstly, the loudspeakers in a multi-channel system are specifically designed to be used in a collective, where the disruption of the number of speakers or the arrangement would likely diminish the experience. In the silent film/disco case, each set of headphones is independent, where the individual listening experience is approximately the same, regardless of whether there are 50 listeners or 500 listeners. Secondly, multi-channel speakers are purely static output objects with no intentional interaction patterns, whereas in silent films/discos, there is typically some form of device interaction, namely personal volume adjustment or selection of a different radio channel. For other research, there may be a legitimate case for including a different range of experiences in this study.

3.2. Survey

To address the first research question What are the defining characteristics of co-located, multi-device audio experiences?, a short survey was conducted to obtain information on both experiences and production tools and systems. A survey was selected as the chosen methodology in order to obtain as large a sample size as possible and was deemed more suitable for purpose versus other qualitative methods such as interviews and focus groups. Researchers, technologists, and creative practitioners were contacted through academic networks, mailing lists, and social media to contribute their knowledge. The survey was delivered using the Qualtrics platform [27]. To specify the types of experiences desired for this study, a set of inclusion criteria was defined at the beginning of the survey.

1.

The platform/experience must employ multiple devices with loudspeakers.

2.

The audio content must be distributed across the devices.

3.

The devices must be co-located in a single space or group of adjoining spaces.

‘Platform’ refers to the underlying technologies and systems which support and deliver multi-device audio experiences, whereas ‘experiences’ are the single applications of those technologies which include the audio content. The survey consisted of open questions with free-text fields where participants could describe the experience and its audio content, as well as multiple choice questions capturing the types of devices used, the modes of interaction, whether the number of devices is variable, and the importance of the devices in the ecosystem. Participants were able to respond to the questionnaire multiple times if necessary. The full questionnaire can be viewed in Appendix A.

3.3. Systematic Literature Search

To supplement the survey, a systematic literature search was completed to locate relevant papers and capture more instances of co-located, multi-device audio platforms and experiences. This was achieved using the PRISMA methodology [28]. Both the Association for Computing Machinery (ACM) Digital Library [29] and the Audio Engineering Society (AES) E-Library [30] were targeted for the search. For both databases, the following query string was used to search paper titles, abstracts, and key words.

((“multi-device" OR “cross-device" OR “distributed" OR “orchestra*" OR “multiple devices") AND (“audio" OR “sound" OR “music"))

The search was conducted by the first author and constrained to papers published between 2000 and 2021. Using the same experience-inclusion criteria as the survey, the initial pool of 361 papers was preliminarily screened through reading the titles and abstracts. This screening stage resulted in 290 paper exclusions and 71 inclusions. These 71 papers were further screened by reading the full texts. Following this process, 11 papers were identified as containing application instances of multi-device audio platforms and experiences that fit into the survey inclusion criteria mentioned above. These instances were then added to the survey by the first author, and the resulting dataset was analysed (see Section 4).

3.4. Thematic Analysis

The free-text data was analysed using a thematic analysis approach, based upon Braun and Clarke’s methods [31] and qualitative descriptive analysis [32], due to the descriptive nature of the data. The purpose of this analysis was to uncover defining characteristics for these types of experiences and to use them to develop a framework for understanding and comparing these seemingly related experiences. The analysis was conducted in three distinct stages. In the first stage, a panel of three experts (including two of the authors) individually assigned codes to each of the text responses. Themes were generated in the second stage, which involved the experts discussing and clustering the codes over two 1-hour group sessions, using a shared Miro board, and which were facilitated by the first author. In the final stage, three non-experts were recruited and given the same text responses as given to the experts to assign codes to. The non-expert codes were then compared and placed into the themes generated by the experts as a form of validation exercise.

4. Results

The survey received 43 responses in total (including responses from the authors and those discovered during the literature search described in Section 3). All multiple choice questions and responses can be seen in Appendix B, and a full list of experiences in the dataset can be seen in Appendix C. Figure 1a shows the frequency of device types from the platforms and experiences described in the responses. An observable trend is that the occurrence of each type largely relates to the size and mobility of the devices, with mobile phones being the most common (33%), and TV/radio being the least common (8%). This result roughly reflects the relative market saturation of each type of device [33] but is also due to the increasing number of experiences created through specific multi-device audio production tools such as Soundworks and Audio Orchestrator, which enable the easy inclusion of mobile devices. The ‘Other’ category, which received 23% of the selections, included Raspberry-Pi-based embedded audio devices and cassette players. Audio-only devices are far less common than those with a visual display. This can be attributed in part to integration challenges for these devices, where many of the modern experiences implement delivery through a web URL that can be typed or accessed via a QR code.

The frequency of various interaction modes is shown in Figure 1b. Tactile input and touch interfaces are most common (63%), followed by motion-gesturing interaction (19%), both of which smartphones and tablets commonly possess. The ‘Other’ category includes, but is not limited to, proximity-based control through the use of RFID or Bluetooth technologies. Out of all 43 entries, 16% of them are labelled as non-interactive. The number of devices was reported to be variable in 84% of entries. The data collected from the responses to the question are insufficient to determine whether the experience could vary the number of devices during or between instances of the experiences. Despite this, the results still demonstrate the flexible nature of these experiences and tend to display configuration agnosticism, unlike traditional multi-channel audio experiences. Finally, in answer to the question about device importance, 58% of responses indicated that the devices were perceived to be equally important, whereas 26% reported some form of hierarchical device structure in the ecosystem.

Through the thematic analysis process described in Section 3, 17 themes were generated (

Table 1. The 17 themes generated from the thematic analysis of the free-text survey responses, their descriptions, and associated parent themes.

Parent Theme	Theme	Description
Devices	Device position	The position of the devices
	Synchronisation	The existence of and the accuracy of synchronisation in use
	Device hierarchy	The existence of and the configuration of the device hierarchy
	Types of device	The types of devices in use
	Number of devices	The number of devices in use
Listeners	Number of listeners	The number of listeners present
	Listener position	The position of the listeners
	Listener role	The role of the listener in the experience, including the extent of contribution
	Social interaction	The type and extent of social interaction during the experience
Content	Content interactivity	The extent of interaction with the content, including device input modalities
	Content hierarchy	The existence of and the configuration of the content hierarchy
	Content distribution	How the content is distributed across the devices
	Content genre	What does the audio content consist of i.e., music, soundscape
Other	Scale	The overall size and scale of the experience
	Flexible distribution	The ability for reconfiguration of devices and redistribution of audio
	Immersive	Experiences aiming to possess an immersive quality
	Type of experience	The format of the experience, i.e., performance, game, etc.

) which were captured under four parent themes: devices, listeners, content, and other. These themes were validated by a set of 300 non-expert codes, with 92% of the non-expert codes being represented within the 17 themes defined by the experts.

5. Design Dimensions

This section describes a set of seven design dimensions, derived from further analysis and refinement of the themes from the thematic analysis, with reference to relevant literature. This process involved re-analysing the themes and aggregating related themes into encompassing design aspect groups, which are different from the parent themes. For example, the Listener role, Social interaction, and Content interactivity are combined into an Interactivity design aspect, despite all being under the same parent theme. In other cases, larger individual themes were assigned their own design aspect. Once all the high-level design aspects were established, the design dimensions were formed by identifying all the distinct design choices associated with that aspect. Figure 2 illustrates the process of generating the design dimensions from the original data.

These dimensions aim to represent the key design considerations for co-located multi-device audio experiences. Within each dimension are a small number of categories which constitute the different design choices available for that dimension. For each dimension, examples from the literature are provided, and how the dimension affects various aspects of audio experiences is discussed. A few of the following dimensions have been adapted from or are related to Brudy’s cross-device taxonomy [3], as many of the important design considerations are applicable to multi-device experiences beyond audio-only. Nevertheless, any adaptations made are intended to make the considerations more relevant to audio experiences. These dimensions relate to:

• the degree of audio synchronisation,
• the social and environmental context of the experience,
• the physical positioning of the devices,
• the interaction relationship between audience members and devices,
• the degree of audience interactivity,
• the roles of the devices and how they are organised in the ecosystem, and
• the distribution of audio content among the devices.

5.1. Synchronisation

Synchronisation is defined as the extent to which the devices in the ecosystem are synchronised in their reproduction of audio. This dimension appropriates and extends Brudy’s temporal dimension [3] for synchronous, multi-device interactions through three categories of synchronisation for audio content.

1.

Asynchronous —In this instance, the devices are not synchronised in any form, acting completely independently from each other. The applications are limited, but asynchronous systems can be used for experimental art installations [35] and enchanted object experiences [36].

2.

Loose synchronisation—Loosely synchronised audio can be achieved through artistic composition, without the use of device networking. The audio material is designed to be played at approximately the same time but without the need for precise timing. Many early multi-device audio experiences were limited to this approach, which typically involved listeners pressing play on devices at the same time [37] or other manual triggering of audio sources [38]. The synchronisation accuracy between devices is typically between 100 ms and 1 s.

3.

Tight synchronisation—This category captures experiences where devices are connected and synchronised using a network. This approach is common in more modern experiences and enables musical performance applications [39] and narrative-driven experiences such as audio dramas [21]. For such applications, experiences in this category require device synchronisation of <100 ms or from word-level accuracy down to approximately frame-level synchronisation [8]. Typical synchronisation solutions for these experiences obtain a sync accuracy of around 20 ms for heterogeneous devices after calibration [40].

5.2. Context

This dimension captures the social and environmental context of the experience, such as the scope for social interaction and the physical space it takes place in. The three categories generally relate to the number of people and devices present. According to Bown’s media multiplicities and their “significance of numbers” [7], the relationship between the number of loudspeakers and individual audio elements also affects how the audio is perceived.

1.

Personal—Personal multi-device audio is an emerging class of experience involving a single listener and a small number of devices. The experiences usually have limited interactivity due to the impracticality of individually controlling multiple devices. Applications therefore tend to consist of overlaying additional audio content to a program [22] or enhancing the spatial audio image through placing devices in different positions [41].

2.

Social—Experiences comprised of small to medium-sized groups of listeners with multiple devices are classified as Social. In this context, rich social interaction is possible and audio scenes with increasing complexity can be created, while it is still possible to perceive the individual contribution of the devices in the system [42]. Experiences of this kind can take place in both small domestic environments and larger spaces, depending on the exact number of involved devices and listeners.

3.

Public—Public represents the larger-scale experiences with many listeners and devices. Experiences in this category commonly have more than 20 listeners and a large number of devices. In this context, communication or coordination is more difficult and commonly requires a central composer or figure to orchestrate the experience. Individual device contributions are typically displaced by the larger aggregate audio image. Many use-cases, such as art installations [35] and performances [13], fall into this category and tend to take place in large public venues.

5.3. Position

Position is concerned with the physical configuration and positions of the devices in the experience, and is similar to Brudy’s dynamics dimension [3]. Three categories are identified.

1.

Fixed—Device positions are typically static and pre-determined, similar to traditional channel-based speaker systems. The number of devices in the system tends to be known, and device placements are optimised for the focal aspect of the particular experience, i.e., the spatial audio image. For example, Tsui et al. designed a one-person orchestra experience [43], where device positions are carefully calibrated with motion controllers to enable audio control through motion gestures.

2.

Arbitrary—Many modern tools and frameworks enable flexible, ad hoc device ecosystems where devices can join and leave at will. This ambiguity in the number of devices makes the control of device positions very difficult. With this flexible positioning, devices can be placed in arbitrary positions. Often, these positions are the same as the listener position if personal devices are used [44,45]. However, this tends to remove the control of the position of individual sound sources in the spatial audio image.

3.

Dynamic—In a few cases, experiences are designed with intentional device movement by utilising Bluetooth or RFID technologies, which can result in ever-changing spatial audio images, which can change over time [46,47].

5.4. Relationship

A number of different people-to-device interaction relationships exist with multi-device ecosystems. These are represented by three categories, similarly adopted from Brudy’s relationship dimension [3].

1.

One-to-many—The first category captures use-cases where an individual utilises many devices to expand or augment a single device listening experience [48,49]. This is aligned with the ‘personal’ value of the context dimension.

2.

Multiple one-to-one—Many experiences implement a ‘use your own device’ approach, which is an easy and accessible method for delivering shared audio experiences to a group. Each listener interacts with their own device, therefore known as a multiple one-to-one relationship [5,38,50]. This relationship lends itself well to interactive experiences, where audience members can influence the experience.

3.

Many-to-many—Other shared or collaborative settings where the numbers of devices and people are not equal can be associated under this category. In this case, there is no link between the number of listeners and the number of devices, where individuals may interact with different devices over the duration of the experience. Examples of this are commonly seen in art installations [35] and music performances [37].

5.5. Interactivity

While traditional, passive media experiences, such as TV and film, still have their place, modern technology now allows for experiences with varying degrees of audience interactivity. Consequently, this affects and broadens the possible roles of audience members in an experience, where an audience member can be described as a bystander, spectator, customer, participant, or player [34]. This dimension is split into three categories, a simplification of Striner’s [34] spectrum of audience interactivity for entertainment domains.

1.

Passive—Passive experiences capture more traditional media experience formats, where audiences are distinctly passive and have no influence on the media content. Many multi-device audio performances are of this nature [51] but so are storytelling experiences typical of radio or TV content [52].

2.

Influence—This category includes application instances where the audience has some limited form of influence or control over the media content, but the experience creator retains all creative control over the experience. This could be personalisation through the selection of audio assets [22] or other influence, perhaps through low-impact manipulation of the audio signal [36].

3.

Create—Highly interactive experiences where audience members could be better described as players or performers, actively creating the audio content of the experience, are situated in this final category [44,53]. For these experiences, the experience creator passes some of the creative control onto the audience members.

5.6. Organisation

Ecosystems of multiple devices introduce new questions around device organisation, from technical, social, and content delivery points of view. This dimension is focused on how the devices are configured for audio reproduction, and what the roles of each of the devices in the delivery of the experience are. This is linked to the interaction topologies presented by Matuszewski et al. for mobile-based music systems [54].

1.

Non-hierarchical—The simplest approach is to assign all devices to have equal responsibility in the experience. In this configuration, each device contributes to the audio experience equally and is regularly observed in social experiences, where personal devices are used [44,47,50]. This can be achieved in practice through the disconnected graph or circular interaction topologies [54].

2.

Hierarchical—In some cases, it may be better to organise devices into two or more tiers, with differing responsibilities and affordances. This is effective where the audio content can be organised into an importance hierarchy [21,55], e.g., dialogue and sound effects in TV and film, or where mobile devices cannot adequately reproduce certain frequencies or amplitudes and need support from an additional device [5,19]. A common implementation of this design choice is assigning a single device as the ‘main’ device and assigning other devices as ‘auxiliary’ devices [20]. These device types can be controlled separately and can be assigned different audio assets if necessary. Other configuration examples include the star and forest interaction topology graphs [54].

5.7. Distribution

The distribution dimension captures different approaches for the distribution of audio to the devices in an experience. This can be linked to the previous dimension, where audio distribution can be determined based on the roles of each of the devices in the experience. There are three categories.

1.

Mirrored—The simplest case is where the same audio content is allocated to every device in the system. This mirrored approach is useful when a simple amplification or duplication of audio is required, e.g., in a silent disco or silent film [56], or in commercial multi-room speaker ecosystems such as Sonos [57].

2.

Distinct—Here, the audio assets on each device are unique, where no two devices are playing the same audio. This approach is prevalent in interactive, music-based experiences where each device is allocated a unique instrument or sample, allowing each participant to have an equal yet distinct contribution to the experience [39].

3.

Hybrid—In this instance, devices are allocated a mixture of shared audio material as well as unique assets for individual or groups of devices. Audio allocation here is determined through a pre-defined set of rules, which can take into account many factors such as device numbers, types of devices, and device locations. Examples of this include larger audio performances with many devices, where there are a number clusters of devices playing different content [13,38].

5.8. Summary

The themes presented in Section 4 provide an overview of the low-level design aspects of co-located, multi-device audio experiences; however, the number of themes and the complexity of the relationships between them can be overwhelming and, therefore, less practical. These design dimensions aim to provide a simplified framework for the core design considerations for these experiences by abstracting away some of the aforementioned complexity. As a result, the dimensions are likely best utilised in the early stages of experience design.

6. Application Patterns

The dataset obtained from the survey was re-analysed to explore the various current applications of co-located, multi-device audio experiences. For this exercise, each experience was assessed against each design dimension and given a respective category. The experiences and their categories were then compared to see how the different design choices affected the type of experience. Using this method, four example application categories were identified, where the experiences in each category share a common combination of categories. For each experience in the dataset, their descriptions, and corresponding application types, please refer to Appendix C. Additionally, in the dimension breakdown table for each of the experience types, the key dimensions are highlighted in bold.

6.1. Public Performances

Key attributes

Synchronisation	Asynchronous/Loose/Tight
Context	Public
Position	Fixed/Flexible/Dynamic
Relationship	Many one-to-one/Many-to-many
Interactivity	Passive
Organisation	Non-hierarchical
Distribution	Mirrored/Hybrid

The largest category in the dataset consists of large-scale performances/installations in a public space. These experiences are akin to more traditional media experiences with largely passive audiences. These types of experiences mark the early experimental years of multi-device audio as a medium, many occurring prior to the 21st century, before ‘personal’ devices [13,37]. However, this is not always the case, notably where new forms of audio devices are being explored for their creative potential [51].

6.2. Interactive Music

Key attributes

Synchronisation	Asynchronous/Loose/Tight
Context	Social
Position	Flexible/Dynamic
Relationship	Many one-to-one/Many-to-many
Interactivity	Influence/Create
Organisation	Non-hierarchical
Distribution	Distinct

Interactive music contains interactive experiences that involve audience participation in music composition or performance. Most of the experiences exhibit either loose or tight synchronisation, as expected for the content medium, although a small number do display asynchronicity [53,58], where the musical content has less of an emphasis on rhythm and structure; for example, ambient music or soundscapes. Additionally, most of the experiences employ a non-hierarchical organisation of devices, ensuring that all participating listeners have the opportunity to make equal contribution to the experience. Where a hierarchical structure is utilised, it is typically in the form of a supporting loudspeaker system to provide better bass frequency reproduction, as seen, for example, in Drops [19]. Many of these experiences have been developed using the Soundworks framework [19], which provides tools for developing interactive experiences.

6.3. Augmented Broadcasting

Key attributes

Synchronisation	Tight Synchronisation
Context	Personal/Social
Position	Arbitrary
Relationship	One-to-many/Many one-to-one
Interactivity	Passive/Influence
Organisation	Hierarchical
Distribution	Hybrid/Distinct

A relatively new set of experiences has emerged over the last decade, that tend to extend a conventional, domestic TV or radio listening experience with additional devices for more immersive spatial audio [21,22,52,55] or to allow the personalisation of content [48,49]. These experiences leverage the ubiquity of personal devices in the home, such as smartphones, tablets, and even newer wearables such as the Bose Frames [59]. Many of these experiences have been created through the Audio Orchestrator production tool, which allows separate audio allocation to the ‘main’ and ‘auxiliary’ devices via a configurable set of ‘behaviours’. Elsewhere, German broadcaster Beyerischer Rundfunk (BR) has developed a smartphone app which communicates with smart TVs to allow the selection of audio description for selected programs, which can be heard through headphones connected to the smartphone [49].

6.4. Social Games

Key attributes

Synchronisation	Asynchronous/Loose
Context	Social/Public
Position	Dynamic
Interactivity	Influence/Create
Relationship	Many one-to-one
Organisation	Non-hierarchical
Distribution	Mirrored/Distinct/Hybrid

Finally, the last and smallest application category seen in the dataset contains shared, interactive audio games. These experiences generally involve listeners or players moving around with devices in a physical space and demonstrate audio-augmented reality (AAR) characteristics through the augmentation of a game with synthesised sound. Audio assets can be triggered and modified through positional and motion-based interactions. These experiences are limited to research studies for now, utilising bespoke, custom-made devices, but represent a vision for future possibilities. Examples include Bown and Ferguson’s work with DIADS [36]; SoundWear [47], a study investigating the effect of non-speech sound augmentation on children’s play; and Please Confirm you are not a Robot [60], a narrative-focused AAR game.

7. Discussion

7.1. Dimension Categories and Combinations

When characterising experiences using these dimensions, it is sometimes difficult (and counterproductive) to assign one given category. Many modern experiences are built with technologies that enable them to traverse between the categories of given dimensions, even during the experience. For example, some experiences could equally be assigned any of the categories in the Context and Relationship dimensions, due to the affordance of being able to add and remove devices. It may or may not be better to consider each of these variations separately; however, it is valuable to recognise the possibility that experiences can move along dimensions and to understand the capabilities of the technologies chosen to build a given experience that may enable this behaviour.

The design dimensions also contain a number of invalid category combinations, of which five have been identified. These are outlined below, where the dimension category is given alongside the dimension name in brackets.

• One-to-many (Relationship) and Social (Context)
• One-to-many (Relationship) and Public (Context)
• Personal (Context) and Multiple one-to-one (Relationship)
• Personal (Context ) and Many-to-many (Relationship)
• Mirrored (Distribution) and Create (Interactivity)

Combinations one to four exist due to both dimensions containing categories which represent a single listener experience and are incompatible with the shared experience categories of the comparative dimension. For combination five, if the audio content is dependent on the creative process of an audience member, it is highly unlikely that the audio output of each device will be identical. Knowledge of these invalid combinations can help steer users of the design dimensions toward more useful outcomes.

7.2. Relationship with Audio-Augmented Reality

Another point of interest is the apparent application overlap between co-located, multi-device audio experiences and audio-augmented reality (AAR) experiences, when comparing with Krzyzaniak’s six categories of AAR [61]. Specifically, experiences in the Augmented broadcasting and Social games experience groups illustrate AAR through two forms. Augmented broadcasting experiences tend to exhibit AAR through the provision of extra context-relevant audio content. In contrast, Social games use an enchanted objects approach to AAR by adding extra purposeful sound to typically silent objects. Given the understanding that AAR can be defined as real-world experiences that are accompanied by additional layers of sound under either form mentioned above, there is an inherent requirement for two or more sound sources, where either, some, or all of those sources could be audio-capable devices. The question then is: When are co-located multi-device audio experiences considered to be AAR experiences?

A good starting point is to evaluate the Organisation dimension, to consider the roles of the devices in the experience. In most ‘non-hierarchical’ experiences, where each device contributes to the audio equally, the experience content can be considered to be a sum of the output of all the devices. Another perspective is that the experience cannot be ‘complete’ without the array of devices, so there is not augmentation but equal complementation. Conversely, in many ‘hierarchical’ experiences, the core experience can be listened to on a single device, and the addition of new devices can introduce enhancements such as spatialised audio and additional audio content, which is consistent with AAR characteristics. An example of this is demonstrated in the BBC trial ‘Immersive Six Nations’ where the original program can be viewed on a main device, but additional devices grant access to extra audio assets such as on-pitch sound effects [22]. This is applicable for Augmented Broadcasting use-cases but is less useful for Social games, as is illustrated by Soundwear [47], where the device organisation is ‘non-hierarchical’, yet the purpose of the devices is to add additional audio to an otherwise ordinary outdoor play experience. Thus, when trying to characterise new experiences, it is proposed that the Organisation dimension should be used as a guide to determine whether a co-located, multi-device audio experience is an AAR experience. Consider, however the roles of the devices on an individual experience basis, as well as evaluating which form of AAR is exhibited in the experience.

7.3. Research Limitations

It is important to note that the dimensions presented here should not be treated as a fixed model, accounting for all possible design considerations, but should be considered as a helpful starting point for further discussion and refinement, through highlighting the most important current design factors. As research in this area continues, it is possible that new categories of experiences will emerge that cannot be sufficiently characterised by the proposed model, and so the design dimensions will have to be revised accordingly. This is in part due to the fact that the dataset of experiences obtained in this study is not an exhaustive list, and it is likely that there are other experiences, academic, commercial, or otherwise, that might be considered to be multi-device audio experiences under the inclusion criteria used in this research. However, this dataset represents the largest available sample of these specific experiences to date.

8. Conclusions

Research on multi-device experiences is steadily maturing, although, only a small proportion of it is focused on how multiple devices can be used to facilitate more immersive and engaging audio experiences. In addition, research efforts concerning audio tend to be disconnected, which is shown by the lack of consensus on the use of terminology. In this research, an attempt has been made to link these research threads and find common ground through the evaluation of current co-located, multi-device audio experiences, driven by two research questions.

• What are the defining characteristics of co-located, multi-device audio experiences?
• How do these characteristics manifest in different applications?

A survey and structured literature review were used to produce data on which a thematic analysis was performed to determine a set of seven design dimensions. These are synchronisation, context, position, relationship, interactivity, organisation, and distribution. Furthermore, these dimensions were used to characterise and compare experiences in the dataset, leading to the identification of four example use-case categories of co-located multi-device audio experiences. These are public performances, interactive music, augmented broadcasting, and social games. More generally, this work provides an overview of research efforts from a practical use-case perspective, as well as acting as a guide for researchers and creative practitioners in furthering the development of these experiences.

Future Work

More work is necessary to test and validate the design dimensions and is likely to be an ongoing process. This can be achieved through assessing more experiences against the design dimensions and using the dimensions to explore new edge-case and uncharted applications by investigating unexplored combinations of the design dimension categories.

One of the major benefits of using personal devices for multi-device audio experiences is the accessibility afforded for shared, interactive audio experiences. In many instances of these, the in-built speakers of the devices are used, whereas headphones are rarely used. This could be due to the acoustic isolation effect on the listener, which may inhibit social interaction. In experiences with headphones, they are used predominantly for this acoustic isolation property but can be used with good effect in some social situations, where collaboration is involved, as evidenced in Schminky [50] and ProXoMix [46]. Future work could explore the implications of the use of different types of headphones (i.e., closed-back, open-back, bone conduction, transparent-mode) and their effect on social interaction, for various types of multi-device audio experiences.

Another potential area yet to be explored is the use of smart speakers, and voice interaction more generally, in these experiences. This may be due to the difficulties encountered by speech-recognition software when in the presence of multiple, competing sound sources. There may also be more development difficulties associated with smart speakers than compared to an open-source ecosystem such as Raspberry Pi. Nevertheless, future work could explore the creative use and effectiveness of voice interaction in co-located, multi-device audio experiences.