Customizing and Evaluating Accessible Multisensory Music Experiences with Pre-Verbal Children—A Case Study on the Perception of Musical Haptics Using Participatory Design with Proxies

Abstract

Research on Accessible Digital Musical Instruments (ADMIs) has highlighted the need for participatory design methods, i.e., to actively include users as co-designers and informants in the design process. However, very little work has explored how pre-verbal children with Profound and Multiple Disabilities (PMLD) can be involved in such processes. In this paper, we apply in-depth qualitative and mixed methodologies in a case study with four students with PMLD. Using Participatory Design with Proxies (PDwP), we assess how these students can be involved in the customization and evaluation of the design of a multisensory music experience intended for a large-scale ADMI. Results from an experiment focused on communication of musical haptics highlighted the diversity in employed interaction strategies used by the children, accessibility limitations of the current multisensory experience design, and the importance of using a multifaceted variety of qualitative and quantitative methods to arrive at more informed conclusions when applying a design with proxies methodology.

1. Introduction

The field of research dedicated to Accessible Digital Musical Instruments (ADMIs), i.e., accessible musical control interfaces used in electronic music, inclusive music practice, and music therapy settings [1], is growing. Lately there has been a rising interest in work focused on the accessibility of musical expression. For example, the theme of the International Conference on New Interfaces for Musical Expression in 2020 was “Accessibility of Musical Expression”, and selected papers from the conference were published in a special issue of the Computer Music Journal, see [2]. Recent advancements in the music technology field have paved the way for new music interfaces that can be specifically designed, customized, and adapted to each and every musician’s needs. Despite such progress, many people are still largely excluded from active participation in music-making. For example, musicians with disability are still particularly under-represented in the global music community [3]. The systematic review on ADMIs published in 2019 [1] also revealed that most of the reviewed instruments were designed for people with physical disabilities (39.8%), whereas less work had been dedicated to instruments designed for people with complex needs (6.0%), persons who are non-vocal (6.0%), hard of hearing (6.0%) or have a visual impairment (3.6%). Moreover, the majority of the ADMIs in [1] were designed to be used by a single person, and few ADMIs were large-scale instruments encouraging collaborative play.

In this paper, we present a 1.5-year-long study exploring the role of haptic feedback in musical interactions in sound installations for children with Profound and Multiple Learning Disabilities (PMLD). PMLD is commonly used to describe a person with severe learning disabilities who most likely has other complex disabilities and health conditions, although there is no single universally agreed definition of the term [4]. Research reviewing the different PMLD definitions together with those who provide services, support and care for people with PMLD has highlighted that no definition can fully articulate the complexities associated with the term [4]. Other work has challenged the very idea of PMLD, emphasizing the role of ambiguity in articulating the life-worlds of these children [5].

The work described in this paper is part of a larger research project focused on advancing musical frontiers for people with disabilities through the design and customization of music technologies. The aim of this project is to improve access to music-making through the development of novel music interfaces and to explore how interface design and multimodal feedback can contribute to widened participation. The goal of the current study was to promote inclusion and diversity in music-making through the customization of a large-scale Digital Musical Instrument (DMI), thus allowing for rich multisensory experiences and multiple modes of interaction for users with various abilities and needs (as suggested in [6]). We explored how music can be presented not only in an audible form but also as visual and haptic sensations. In particular, we investigated how Participatory Design (PD) methods can aid the design and customization of musical haptics in the Sound Forest installation. Participatory design is a methodology that involves future users of a design as co-designers in a design process [7]. Sound Forest (see Figure 1a and Section 2.1) is a long-term project between researchers in the Sound and Music Computing (SMC) group at KTH Royal Institute of Technology and the Swedish Museum of Performing Arts (see [8]). It is a multisensory installation that enables visitors to create sounds by interacting with light-emitting strings, for example, by hitting or plucking a string using the hand. The visitors can feel the music through the body, while receiving visual feedback. In this paper, we describe a collaboration with four students enrolled at a special school in Sweden and how we designed a customized multisensory music experience in Sound Forest with and for this user group.

Previous research focused on the role of music for children with PMLD has emphasized the importance of music education [9] and musical play [10]. In [11], the authors suggest that there is a lack of conceptual clarity as to what constitutes music education versus music therapy for this group. They propose a new model of music education where activities are undertaken primarily for their intrinsic musical value as opposed to promoting learning and development. A study exploring the role of music within the home lives of young people with PMLD through parental perspectives was described in [12]. Findings outlined the positive role of music in contexts where music was used for enjoyment, to support mood-regulation, and to add structure to the lives of the children. The study also revealed that it was more common to listen to music than to make music in the home environment.

Musical Haptics is an emerging interdisciplinary field that focuses on investigating touch and proprioception in music scenarios from different perspectives [13]. Musical experiences involve both perceiving airborne acoustic waves and vibratory cues conveyed through air and solid media [13]. Traditional acoustic instruments intrinsically provide vibratory feedback during sound production. Playing a musical instrument requires a complex set of skills that depend on the brain’s ability to integrate information from multiple senses [14]. The haptic sense has been shown to play an important role in musical interactions with acoustic instruments such as the piano [15,16,17], violin [18,19], and percussion [20]. Research has also shown that haptic technology can assist musicians in making gestures [21]. Studies have suggested that amplifying certain vibrations in a concert venue or music reproduction system can improve the musical experience [22], that vibrations play a significant role in the perception of music [23], and that whole-body vibrations (the type of haptic feedback used in the Sound Forest installation, see Section 2.1), can have a significant effect on loudness perception [24]. As pointed out in [13], musical haptics could potentially facilitate access to music for persons affected by somatosensory, visual, and hearing impairments. For example, haptic (or visual) feedback has been shown to enhance musical experiences for persons with hearing impairment [25]. The potential of vibroacoustic therapy for persons with disabilities has also been stressed [26]. A review of haptic feedback in sensory substitution systems allowing persons with hearing impairments to experience music through the sense of touch, so-called Haptic Music Players (HMPs), was recently published in [27]. An HMP is a device that can process an audio signal to extract musical information and translate this information into vibratory signals. An example is the haptic metronome, which presents musical beats through vibration onto the skin, see, e.g., [28]. Despite the potential benefits of musical haptics and the importance of perceiving vibrations when interacting with a DMI (see, e.g., [29]), few ADMIs provide active haptic feedback. In [1], only 14.5% of the surveyed instruments provided active feedback through the sense of touch, and the full potential of haptics in ADMI design remains to be explored.

Few studies have explored the role of haptic feedback for people with multiple disabilities (relevant examples include [30,31]), especially in the context of music. However, the potential of multisensory experiences to enrich the lives of people with profound and multiple disabilities has been stressed (see, e.g., [32]). Such experiences can be provided in so-called MultiSensory Environments/Multisensory Rooms (MSE). MSEs are artificially engineered environments with multisensory equipment used to create a specific mood. These environments allow for activities and experiences of a sensory nature. Although not built to be used for music therapy purposes, Sound Forest can be considered a multisensory environment, as well as an ADMI.

When it comes to the design of ADMIs, it is crucial that those who have lived experience of disability are actively involved as co-designers in the process of creating the instruments. Reflections on co-design processes in the context of ADMI development are presented, for example, in [33,34,35,36]. In the systematic review of assistive technology developed through participatory methods published in [37], the authors emphasize that participatory development processes should enhance the voice of the participants, considering their ideas, desires, and needs. In order to fully incorporate users with disablity into user-centered design processes, existing co-design methods may need to be extended and adapted [38]. Such adaptations are particularly important when working with children with PMLD, since the communication between designers and users can be affected. Co-design with children with PMLD should take into account that these children may express themselves through a number of different communication methods, depending on what is most efficient for them at the time. Examples include bodily gestures, nonverbal sounds, pointing, and facial expressions. Alternative methods for augmented communication, such as PODD (Pragmatic Organisation Dynamic Display) [39,40] and TaSSeL (Tactile Signaling for Sensory Learners) [41], can be used to support such co-design processes. Other approaches that could be relevant in this context include methods for assessing the views of users through preference assessment [42,43] and methods to categorize behavioral states [44,45]. In the current work, a core focus has been on the tool PODD. PODD is a way of organizing word and symbol vocabulary in a communication book to support understanding (see Figure 2). In our research, we have explored how PODD can be used in communication supporting the participatory design process. We use Participatory Design with Proxies (PDwP) [46] to enable inclusion of input from different stakeholders, i.e., parents and teachers. This method allows for different proxies to provide valuable insights and feedback to augment direct input from the children. To the authors’ knowledge, little previous work has investigated how children with PMLD can act as informants in design processes focused on multisensory installations and large-scale ADMIs.

DMI evaluation is a persistently debated topic in the field of research dedicated to New Interfaces for Musical Expression (NIME) (see, e.g., [47]). Challenges in the evaluation of bespoke assistive music technology have been discussed in [48,49]. The latter work highlights the shortcomings of frameworks biased towards describing others, with researchers often viewing ADMIs from an external perspective. The authors acknowledge that music-making is a phenomenon that is challenging to understand and that researchers might struggle to describe such ecosystems in their entirety. The term ecosystem in this context refers to the fact that a musical activity exists in a system containing constitutional building blocks of affordances, i.e., how something is perceived directly in terms of its potential for actions [50], and constraints existing between the musician and the socio-cultural environment. For a detailed discussion of ecological perspectives in ADMI design and evaluation, please refer to [48].

Nine properties to consider when designing and evaluating ADMIs were proposed in [6]: expressiveness, playability, longevity, customizability, pleasure, sonic quality, robustness, multimodality and causality. A dimension space for the evaluation of ADMIs was introduced in [51], which listed key aspects related to target users and use contexts (cognitive impairment, sensory impairment, physical impairment, and use context), and design choices (physical channels design novelty, adaptability, and simplification). However, such frameworks have not yet been extensively evaluated. In the context of MSEs, the review presented in [52] highlighted the need for randomized controlled trials to evaluate the short- and long-term effectiveness of multisensory therapy. Going beyond discussions of effectiveness measures [53], relatively little has still been published on how to best evaluate musical interactions taking place in MSEs, especially for children with PMLD. Relevant work in this context has primarily focused on observations and preference assessments (see, e.g., [54,55]). To summarise, there is a need for more rigorous research to assess and evaluate the impact of multisensory rooms on children with PMLD, as suggested in [56].

In this article, we report on a long-term study on the customization and evaluation of a multisensory music experience in the Sound Forest installation. This work was conducted through a participatory design process in which we involved students with PMLD and their parents and teachers at a special education school as proxies. The aim of this work was to create a customized multisensory experience informed by the students’ needs, musical preferences, and abilities. An important aspect of this work was to explore tools and methods that could successfully enable a discussion about music and haptics with the students. For example, how can we talk about concepts such as the perception of touch, using PODD? The final design of the multisensory experience was evaluated through an experiment, with music presented in a haptic versus nonhaptic condition. Findings highlighted the diversity in employed interaction strategies, limitations in terms of the accessibility of the current multisensory experience design, and the importance of using a multifaceted variety of qualitative and quantitative methods to arrive at more informed conclusions when applying a design with proxies methodology. Our work contributes to the field of ADMI research by exploring participatory research methods for pre-verbal children with multifunctional physical and intellectual challenges and how such methods could be used to guide the design of musical haptics, a topic that to date has received little attention.

2. Materials and Methods

This paper describes a long-term collaboration with a student group, a teacher, and a group of teaching assistants at the school Dibber Rullen in Stockholm, Sweden [57]. Dibber Rullen is a special education school for students with intellectual challenges and multifunctional physical challenges. The pedagogy at the school follows a thematic structure in smaller groups. The teaching is based on themes, in this case, countries, which are explored for 6–8 week periods. All research described in this paper was carried out as activities within the school’s standard curriculum during music and craft lessons.

The methods used in the current work are grounded on the Social Model of Disability [58,59]. This model describes disability as a condition that arises from the organization of society, attitudes, and the design of the environment. In other words, a person with a disability is disabled by external factors, not by an impairment. As suggested in [60], the design of technology should make sure that users have control of, and not only are passive recipients of, developed technology. This perspective is summarized in the disability rights movement motto “Nothing about us without us” [61], which highlights that people with disabilities know what is best for them and their communities and always should be included through participatory design processes. In our research, we have tried to include the students as informants in every step of the design process, using a combination of different user-centered [62] and participatory [7] methods with proxies [46]. A key consideration has been to adapt our methodology to the school’s pedagogical practice and make sure that all steps of the research are well adapted to the students’ needs and preferences. Each step of the research project was informed by conclusions drawn from the previous stage. All work was conducted in close dialogue with the teacher and teaching assistants at the school, who in turn communicated directly with the student group and their parents.

Most of the research described in this paper was carried out during the COVID-19 pandemic. This greatly affected the extent to which we could connect with the students; social-distancing measures were required to ensure the safety of the users since they are a risk group. As a consequence, we adapted our methods to a distributed participatory design setting (see, e.g., [63,64,65]). Distributed participatory design is a term that is used to describe situations in which all or most design team members are physically, and perhaps also temporally, dispersed. The need to re-frame activities focused on inclusive music-making or design due to the absence of in-person contact with musicians with physical disabilities during a global pandemic was discussed in [66]. The authors proposed to use the Three Pillars of Inclusive Design (accessibility, usability, and value) as a framework for analysis in such contexts. In our study, all of the prestudies were conducted online using video-conferencing tools or by asking the teacher responsible for the music classes to film the interactions taking place at the school, in the wild. Participatory design methods in the wild take place outside of a lab, in settings for which the developed artifacts are actually envisioned [67]. This may introduce particular challenges (see, e.g., [34,67]). The experiment in Sound Forest was carried out once the restrictions had been lifted. This was the only physical encounter we had with the student group.

2.1. Sound Forest

The Sound Forest installation occupies an entire room. It consists of five light-emitting strings attached from the ceiling to the floor, five loudspeakers, five contact microphones set up to detect sound onsets for the strings, ten tactile transducers, and two mirrored walls. The installation was designed with accessibility in mind; the aim of the design was to provide rich multisensory experiences for all visitors, regardless of abilities. Visitors can interact with the strings in the installation and receive feedback in the form of sounds coming from loudspeakers placed above the strings, light emitted directly from the strings, and vibrations that can be felt through circular platforms cut out in the floor. One can use the hand to pluck or strike one of the strings, thereby generating a sound. For a detailed description of different interaction strategies observed for a prototype version of the Sound Forest installation, please refer to [68]. The platforms in Sound Forest rest on vibration dampers, which enables them to vibrate freely if set into motion by two Clark Synthesis TST239 Silver tactile transducers [69]. The strings are decoupled from the platforms and are thus not affected by the vibrations. There are five platforms in the installation. Four of them have a diameter of 0.60 m, and one has a diameter of 0.75 m. The larger platform is designed to be more accessible for visitors using a wheelchair (see Figure 1b).

In our previous work, children and adults with physical and intellectual disabilities were invited to freely explore the installation, playing on the strings for three-minute intervals [70]. The visitors could use whatever strategy they wanted to set the strings into motion (most participants used their hands). We have also explored how music producers can be supported by an introductory workshop focused on perception of whole-body vibrations when composing musical haptics for Sound Forest [71]. Whole-body vibrations occur when a human is supported by a surface that is shaking, and the vibration affects body parts remote from the site of exposure [72]. Studies have established a relationship between the magnitude, duration, frequency content, and waveform of the vibration signal in this context. However, the interaction between these properties is not trivial and is also confounded by inter- and intra-subject differences [72]. Similar to the concept of a Head-Related Transfer Function (HRTF), which characterizes how an ear receives a sound from a point in space, different bodies have different transfer functions for vibrations, so-called Body Related Transfer Functions (BRTF). While an HRTF response depends on the size and shape of the head, ears, and the ear canal, among other factors, the BRTF depends on individual body properties such as weight [73]. As a result, different individuals may have different perceptual experiences in Sound Forest, even if the same signal is played. This highlights the importance of asking the users what they actually perceive when interacting with the installation.

2.2. Participants

We collaborated with an existing student group at Dibber Rullen. The group consisted of four children (2F, 2M, age 9–15). Student groups at Dibber Rullen are reconfigured at the beginning of each school year. One student left and another student joined the group in 2021. This section describes the final group constellation. The students in the group are mostly pre-verbal, meaning that they do not yet have verbal communication skills. They all have multifunctional physical challenges, with varying motor skills, moderate to severe intellectual challenges, and use wheelchairs. As reported by the parents of P1 (Participant 1), she is hard of hearing, has a visual impairment, a chromosome aberration, reduced mobility, a physical disability, and hypermobility in the joints. The parents of P2 reported that it is not clear whether she is hard of hearing or if she has a visual impairment. They also described that she can lift her arms and grab objects. The parents of P3 reported that they did not know whether he is hard of hearing or has a visual impairment. They described that he could not walk but that he could jump when sitting. The parents of P4 reported that he had no hearing loss or visual impairment. A detailed description of the musical background and music preferences of the participant group, as reported by the parents, is presented in Section 3.1.4.

2.3. Ethics Statement

The research procedure described in this paper was reviewed by the Swedish Ethical Review Authority (application No. 2021-06307-01). The study was carried out in accordance with the declaration of Helsinki. We followed informed consent rules and guidelines for ethical research practices presented in the APA Ethical Principles of Psychologists and Code of Conduct [74], CODEX [75], ALLEA [76], and SATORI [77]. We also followed guidelines for inclusive language use (see, e.g., [78] for an overview of Swedish terms) and recommendations based on the International Classification of Functional Status, Disability and Health (ICF, see [79]). In this paper, we use Person/People First Language (PFL) when writing about disability, as opposed to Identity First Language (IFL), see [80]. All parents gave written informed consent before participation and agreed to the data being collected as described in the consent form. It was important to make sure that all students gave consent to participate at all times. This was ensured through direct communication with the teacher/teaching assistants. The teacher proxies also filled out a consent form and gave written informed consent prior to participation. Management of datasets that include personal information of study participants was compliant with the General Data Protection Regulation (GDPR). Procedures for registration and storage of personal data (including sensitive personal data, Swedish: känsliga personuppgifter) were reviewed and approved by KTH’s data protection officer ([email protected]) and KTH’s Research Data Team. Only the Principal Investigator had access to videos and images of the students; none of this material is published as Supplementary Material. The full Ethics Approval can be obtained from the Swedish Ethical Review Authority or by emailing the first author. The Ethics Approval report includes all consent forms and information for research subjects (Swedish: personuppgiftsinformation), as well as the Data Management Plan (DMP) approved by KTH officials.

2.4. Prestudies

2.4.1. Physical Characterization of Vibrating Platforms

In order to understand the physical characteristics of the platforms in Sound Forest, we first measured the frequency responses of the differently sized platforms. The purpose of these measurements was to obtain frequency responses that could inform the design of the haptic feedback provided by the installation. Measurements were made using a Bruel and Kjær accelerometer type 4393 placed on the surface of the vibrating platforms, using petro wax mounting directly on the wooden floor, while exciting both tactile shakers attached to the platform using a sine sweep going from 10 to 10,000 Hz in 18 s. In this way, the accelerometer moved jointly with the platform, making it possible to determine the vibrations perpendicular to the platform surface. The accelerometer was connected to a Bruel and Kjær charge amplifier type 2635. A standard mini microphone captured the audible output for each measurement. The inputs were routed to an RME BabyFace sound card and recorded using the software Tombstone [81], which also synthesized the sine sweep. The signals were recorded at a sampling rate of 44,100 Hz. The measurement was made with the microphone placed next to the accelerometer, on the floor. The procedure was then repeated for three conditions with one of the researchers (M, age 31, weight 70 kg) (1) sitting in a wheelchair, (2) standing on the platform, and (3) lying down on the platform. For these measurements, the microphone was placed at the ear of the researcher to simulate an actual use case scenario. For comparative purposes, additional measurements were also made for another researcher (F, age 32, weight 46 kg) for condition (1) and (3) for the small platform and for condition (1) and (2) for the large platform.

2.4.2. Observation of Music/Dance Lesson

This step consisted of remotely attending a music lesson at the school. The purpose of this prestudy was to obtain a better understanding of the type of music material that the students usually interacted with during their music classes and the musical interactions taking place during these sessions. The music class lasted 48 min and was video recorded. The session included dancing to music, interacting with a large colorful textile while listening to music, and playing musical instruments. Authors 1 and 2 attended the session and annotated in real-time what they observed, including potential questions that arose. We followed up on these questions through email correspondence with the teacher, focusing on: (1) the overall structure of the music classes at the school, (2) the type of music used, (3) the interplay between teacher/teaching assistants and students through gestures and dance, and (4) the time spent on dancing versus playing musical instruments. We subsequently summarized a set of themes to be further explored in an interview with the teacher and teaching assistants (see Section 2.4.3).

2.4.3. Interview with Teacher Proxies

As a follow-up to the steps outlined in Section 2.4.2, we conducted a semi-structured interview with the special education teacher responsible for the music/dance theme at the school, and two teaching assistants who worked with the music/dance lessons. We discussed what we had observed during the lesson, and how future experiments in Sound Forest could be designed to incorporate similar elements and structure. The overall purpose of this step was to obtain a better understanding of the pedagogical approaches used at the school, their communication tools, and how we best could align our research methods to the students’ needs and the school’s pedagogical practice. The interview started with a short video introducing the teacher and teaching assistants to the Sound Forest installation [82]. We then asked questions about the following themes: (1) thematic structure of the pedagogical practice at the school, (2) the teacher’s and teaching assistants’ interpretation of the students’ communication, (3) properties of the sound and music to be used in a future experiment in Sound Forest, (4) accessibility, e.g., the students’ motor skills and strategies for interaction that could be used in Sound Forest, (5) preferences for visual feedback and lights, and (6) procedure, i.e., how the study should be designed to best account for the students’ needs. The full interview protocol is provided as Supplementary Material. The interview lasted 44 min. It was manually transcribed by author 1. Results were then discussed with author 2 in order to decide how to best move forward.

2.4.4. Questionnaire

In order to obtain a better understanding of the musical background and music preferences of the students, we distributed a questionnaire to the parents of all students at the school (n = 19). A full analysis of all results is beyond the scope of this paper; therefore, we only present the findings for the four children who participated in our study. The purpose of this questionnaire was to investigate (1) if the students played any musical instruments or sing, and which musical instruments and tools they had used, (2) their interest in music overall, (3) what type of music they usually prefer to listen to, and (4) obstacles that may prevent the students from participating in music-making (if any). Most of the questions started off with a question with possible answers in the format “Yes/No/I don’t know”. This was, if applicable, followed by a question in a free-text format. We also included one question where the parents were asked to rate how strongly they agreed with the statement that their child is interested in music on a scale from 0 (strongly disagree) to 10 (strongly agree). The parents were encouraged to fill out the form together with their child. A translated version of the questionnaire is available as Supplementary Material. The answers were analyzed to identify common themes across participants. This analysis was performed separately by authors 1 and 2, who then compared their individual results and conclusions, merging overlapping themes into the summary presented in Section 3.1.4.

2.4.5. Music Listening Sessions

Two main conclusions could be drawn from attending the lesson described in Section 2.4.2: the music used during the class mostly had a clear and prominent rhythm, and dance was an important part of the musical interactions taking place. The questionnaire results also suggested that the students had a broad taste in music overall. These conclusions prompted us to further explore how the students interacted with and responded to different musical elements. In particular, we wanted to investigate how they reacted to music with different rhythmic structures. Informed by these observations, we created two playlists with short excerpts of a range of different sounds that could be played to the students during their music lessons. The music was divided into three different categories: (1) music with clear rhythmic structure (e.g., dances such as waltz), (2) music with more complex rhythmic structure (e.g., an excerpt from a radical jazz improvisation session), and (3) music with the absence of clear rhythmic structure (e.g., ambient or drone music) (see

Table 1. Playlists used during the music listening sessions. The following abbreviations are used: L = playlist number, N = number in sequence, C = category, referring to (1) music with clear rhythmic structure, (2) music with more complex rhythmic structure, and (3) music with the absence of a clear rhythmic structure.

No.	L	N	C	Artist and Title
1	1	1	1	Alfred Schnittke—The Story Of An Unknown Actor: IV. Waltz of Farewell
2	1	2	2	Radiohead—Pyramid Song
3	1	3	3	Top Relax Music—Morgon på floden och vårfåglarnas mjuka sång (soundscape)
4	1	4	1	Diss Reaction—Jiiieehaaaa
5	1	5	2	Frank Zappa—The Black Page #1
6	1	6	3	Julianna Barwick— Inspirit
7	2	1	3	M. Holterback, Julia Eckhardt— Two Stasis Made out of Electricity
8	2	2	2	Brandon Lopez, Chris Corsano, Sam Yulsman—The Mess (Live in Brooklyn)
9	2	3	1	N.W.A.—Straight Outta Compton (Instrumental)
10	2	4	3	Sunn O)))—Alice
11	2	5	2	György Ligeti—Études, Book 2: No. 13, L’escalier du diable
12	2	6	1	Tchaikovsky—Symphony No. 6 in B minor, Op. 74 ’Pathétique’, II. Allegro con grazia

). The total duration of each playlist was ten minutes. This time was selected, after discussion with the teacher, in order to reduce the risk of fatigue. Each playlist consisted of six music excerpts of 1 min and 35 s. All sounds were normalized using LUFS loudness normalization. Breaks of five seconds were interjected after each excerpt to maintain a structure of music versus pauses similar to the one used during the observed music lesson; the music/dance class included many different musical excerpts, with pauses and PODD discussions in between. The two playlists are presented in Table 1. The sounds are also available as YouTube playlists, see [83,84]. Each playlist was played to the students twice, on different occasions, using a Bluetooth speaker. The teacher had initially tried to play both playlists during the same lesson but soon realized that it would be better to split the activity into multiple listening sessions. The teacher and teaching assistants used PODD to ask the students if they liked or disliked the sounds as they were played. For this, they used PODD picture cards displaying a sad versus happy emoji, interpreting the students’ responses in the form of bodily expressions (e.g., pointing at a picture card) and nonverbal sounds. They also asked what the students liked or disliked about the music (e.g., if it was boring, fun, etcetera). The teacher filmed the student’s reactions using a mobile phone and annotated reactions to different sounds. Authors 1 and 2 watched the videos together with the teacher, discussing which sounds were most/least appreciated. The session was recorded and transcribed using an automatic transcription tool [85].

2.4.6. Introducing the Notion of Musical Haptics through PODD

The interview with the teacher and teaching assistants revealed that several students had pages in their PODD books that focused specifically on sound and music. The PODD books are individual and may therefore differ somewhat between students, depending on their interests. To acquire an understanding of the images and words used when communicating about music, we asked for the parents’ permission to let the teacher take pictures of sound and music-related pages in the students’ books. After analysis of these pages, we proposed a set of concepts that could be of interest when discussing musical haptics with the students: (1) places on the body (e.g., belly, chest, head, hands, arms, legs, and feet), (2) descriptions of sensations (pleasant or unpleasant, like or dislike, happy or sad), and (3) intensity (if a vibration should be stronger or softer). It was important to introduce these concepts early on to prepare the students for the experiment in Sound Forest. The teacher brought the above-mentioned concepts to the PODD responsible at the school and returned with a set of pictures that could be used. Examples are presented in Section 3.1.6. The teacher then introduced these concepts to the students through PODD discussions.

2.4.7. Music Listening Sessions with Haptic Music Players (HMPs)

The aim of the last prestudy was to introduce the students to musical haptics on a more practical level. Based on our conversations with the teacher and teaching assistants, we understood that repetition is an important aspect of the school’s pedagogical practice. It was crucial to introduce the sounds that would be used in Sound Forest before the students visited the installation. For these reasons, we used three Haptic Music Players (HMPs) that allowed the students to not only hear sounds but also perceive simultaneous vibrations. The devices used were: HMP 1—a haptic pillow, the HUMU Augmented Audio Cushion [86]; HMP 2—a plush toy backpack with an embedded haptic strap, the Woojer Strap Edge [87]; and HMP 3—a custom-built plush toy with an embedded full-range speaker (Visaton FRS 8-8 Ohm [88]) and tactile transducer (TT25-8 PUCK Tactile Transducer Mini Bass Shaker [89]). The latter visually reminded of the character Chewbacca from Star Wars. The three HMPs are displayed in Figure 3. The Haptic Music Players differed in terms of affordances. The haptic pillow encourages the user to lay their head down on the device. The plush toy backpack is intended to be worn on the back/belly. The custom-built Chewbacca plush toy can be placed on the lap and explored using your hands. The design of the two plush toys was informed by comments made by the teacher during a meeting in which the haptic pillow and the Woojer Strap Edge were demonstrated. This meeting focused on how to best design HMPs for the specific student group and which modes of interaction and bodily locations that might be most appropriate.

The teacher had three months to test out the HMPs with the students. Two weeks before the experiment in Sound Forest, a sketch of the music to be played in the installation was sent to the teacher for testing with the HMPs. This 2.5 min long sound sample is available as Supplementary Material (see “2.4.7.demo-soundforest.wav”). Following the same procedure as for the music listening sessions, the teacher filmed the interactions taking place when the students interacted with the HMPs, annotating observations. We subsequently conducted two semi-structured interviews with the teacher focused on usability and accessibility themes. The results are presented in [90]. The most important findings in relation to the current study are summarized in Section 3.1.7.

2.5. Experiment in Sound Forest

2.5.1. Procedure

The student group and the teacher, as well as three teaching assistants, were invited to Sound Forest to explore the sounds and multisensory experiences provided by the installation. The week before the experiment, we organized a 30-min meeting with the teacher and the teaching assistants (including observer 3, see description below) who would be present during the experiment. The purpose of this meeting was to discuss the appropriateness of the proposed methodology. Adjustments to the procedure were made based on the teacher’s and teaching assistants’ comments. Since we had quite limited time with the student group on-site during the experiment, instructions and questionnaires were distributed on the day before the experiment. Upon arrival in Sound Forest, the students first had a couple of minutes to familiarize themselves with the room and the sounds. This was done with the lights on (usually the installation is dark, with lights only emitted from the strings in the room, and two external LED lamps, see Figure 1a versus Figure 1b). The wheelchairs were placed on top of the respective platform so that the students could reach the string. The students stayed with the same string and platform throughout the experiment. Three of the students used the small platforms, and one student used the larger platform.

Once the students had familiarized themselves with the room and the teaching assistants had taken off their shoes, we went through the procedure. The structure of the procedure was as follows: twenty minutes of music was divided into four conditions. Each condition was five minutes long and was followed by a pause in which the music was turned off (i.e., silence in the room). The lights were turned on during these pauses. The conditions were: (1) sitting in a wheelchair-haptics on: WH; (2) sitting in a wheelchair-haptics off: W; (3) sitting or lying on the floor-haptics on: FH; and (4) sitting or lying on floor-haptics off: F. This condition order was chosen to minimize the number of times that the students had to be lifted. The order was reversed for two students to investigate if there was an effect of starting with haptics (P1 and P4 started with haptics off, whereas P2 and P3 started with haptics on). During haptic feedback conditions, the teacher and teaching assistants asked the students where in the body they could feel the vibrations and if they liked or disliked the sensation(s). The teacher and teaching assistants used the PODD images presented in Figure 4 to ask where the vibrations could be felt. To enable the use of PODD books in the dark, each book was equipped with a small reading lamp. After each haptic condition, the teacher and teaching assistants filled out body map questionnaires (see [91,92]) to highlight where the students had perceived vibrations. A body map can broadly be defined as an image outlining the human body. The body map questionnaire also included one multi-line free text question: “Do you interpret it as though the student liked or disliked the vibrations? Please describe if there was anything in particular that the student expressed that they liked/disliked”.

Author 1 kept track of the timing between conditions and distributed and collected the body maps after each condition. Author 2 managed the on and off switches for the amplifiers controlling the haptic feedback, the audio on and off switch, as well as the lights. When not busy with other tasks, author 1 (observer 1), author 2 (observer 2), and one of the teaching assistants (observer 3), focused on observing the interactions taking place in the room. They were later interviewed about what they had observed during the experiment; see detailed description below. The entire session was filmed using four stationary video cameras (one for each student and platform). In order to capture the students’ nonverbal communication (facial and bodily expressions) when communicating with the teacher and teaching assistants about what they perceived, two researchers (including author 3) from KTH Royal Institute of Technology were invited to serve as mobile camera operators. All audio was registered using a Zoom H4nPro handy recorder device. Alongside the recordings, we recorded timestamps and amplitudes of every detected onset for each string.

Micro phenomenological interviews with the observers were carried out after the experiment. This interview method was used to access the observers’ passive memory (or background information) [93] to elicit detailed first-person descriptions that may otherwise remain hidden to traditional methods [94]. Micro phenomenology focuses on the how instead of the what of the experience, distinguishing it from other qualitative methods [95]. This interview technique has been applied to scientific research for the exploration of pain in fibromyalgia patients [96], the emergence of seizures in epileptic patients [97], and musical experiences [98]. The interview procedure is described as follows:

• The interviewer introduces the objectives of the interview. In this case, the observers (or interviewees) were asked to select a specific moment in the students’ interaction with the Sound Forest installation that they might have found memorable. It is important to highlight that the exploration of the subjective experience of the observer is crucial.
• Next, the interviewer explains the micro phenomenology process (described in step 3, below) and some formalities, including the duration of the interview and the possibility of withdrawing from the session at any time.
• The interviewee is asked to select a specific moment to explore and describe in general terms. Then, the interviewer recapitulates and asks some open questions to access further contextual information (such as what they were seeing, hearing or feeling), aiding the interviewee in accessing their passive memory. The interviewer then asks more specific questions focusing on both the temporal aspects (such as what happens before and after a given description) and the how of the experience. These questions lead to more descriptions, which are further deepened by following the same procedure.
• To close the session, the interviewer summarizes the interview content and asks the interviewee to clarify, comment, and correct if any accounts were misunderstood.

In our research, we used micro phenomenological interviews to explore some potential tensions and overlaps arising from the interpretation of the attitudes and body language of students while interacting with the installation. The three observers all had different roles in the project and could, therefore, contribute with different perspectives: observer 1 (author 1) is a researcher focused on sound and music computing, who was the principal investigator and responsible for the study; observer 2 (author 2) is a composer responsible for the programming, music, and multisensory design; and observer 3 is a Master’s thesis student in engineering, with many years of experience from working as an assistant at Dibber Rullen, currently carrying out a project focused on developing instruments for the students, building on [99].

One week after the experiment, Author 1 conducted an online semi-structured group interview with the teacher and the teaching assistants (excluding observer 3). The interview used a stimulated recall methodology [100]; a replay of videos from the experiments was used to stimulate a commentary upon the thought process at that time. The teacher and teaching assistants were shown one video clip from each condition and student, and asked (1) what happened in that particular instance and (2) how they interpreted the student’s reactions. For haptic conditions, this also involved visual inspection of the body maps. The video clips, which were selected by author 1 after the labeling process described in Section 2.5.2, included interactions with the strings, bodily expressions, instances in which there was little movement or no bodily expression, or PODD discussions. The purpose of this step was to collect richer information about the communication between teacher proxies and students regarding like versus dislike. After commenting on all the videos, the session was concluded with six open-ended questions focused on accessibility:

• How appropriate were the music, lights, and vibrations for the particular user group?
• In general, what worked well/did not work well?
• Do you have any suggestions for how the installation could be improved in order to be more accessible?
• Did the students, overall, have any preferences when it comes to presence versus absence of haptic feedback?
• Did the students, overall, have any preferences when it comes to experiencing the installation sitting in a wheelchair or being on the floor?
• If you would have your own multisensory room at the school, with the possibility to have sound, lights, and vibrations, what would such a room look like and how would it work?

The purpose of this final step was to obtain a better understanding of how future versions of the installation could be improved to be more accessible for the student group, both in terms of multisensory design and mapping strategies.

2.5.2. Analysis

Using the collected onset data, we resynthesized the sounds created by each student for each condition and generated temporal plots representing these interactions (see Figure 5). All recorded videos were segmented for each condition. Author 1 then labeled the following time points in the recordings: interactions with the string, PODD discussions, other gestural expressions (arm and leg movements, e.g., clapping, waving the hands), and non-vocal expressions. Interactions with the strings were labeled based on strategy used to trigger a sound, inspired by the Exploratory Procedures (EPs) in haptic object exploration [101] (e.g., movement patterns such as lateral motion or contour following) and categories identified in [68] (e.g., plucking, pulling, shaking, muting). Annotations regarding the level of interplay between the student and the teacher/teaching assistants versus autonomous play were also made for each video clip. The total number of onsets performed by the student versus the teacher or teaching assistants was counted manually by author 1, using a click counter. This procedure was used since it is not possible to know from the recorded onset data alone which sounds were triggered by the student versus the teacher proxies. Obtained results were compared to the recorded onset data to separate the number of onsets triggered by the student versus the teacher and teaching assistants. The total duration of interaction was calculated for each student and condition; interaction time was defined as the time in between that the student had touched the string until the string was released (if no sound was made) or until the string went into idle mode (if a sound had been triggered). Idle mode is the state when the string has returned to its original color and does not make any sound, i.e., when the system is waiting for a new trigger from the user (see Section 3.2.1).

The micro phenomenological interviews were analyzed by author 3. The interview material was automatically transcribed using a transcription tool [85], then refined manually to ensure precision. The data were analyzed through the lenses of thematic analysis [102], where interviews were read several times in search of patterns. Themes were expected to arise as similarities or differences in how the observers perceived the users’ behaviors. Insights linked to responses found in other datasets (i.e., interviews with the teacher and teaching assistants described below) were also considered.

The final interview with the teacher and teaching assistants was transcribed manually and analyzed by author 1. For the first part of the interview, i.e., stimulate and recall, content analysis [103] was used to group quotes into themes. While thematic analysis provides a purely qualitative account of data, content analysis uses a descriptive approach in both coding of the data and its interpretation of quantitative counts of the codes (see [104,105,106]). As such, an inclusion criterion in which there had to be at least three quotes to be considered as a theme was defined. For the second part of the interview, i.e., the six concluding questions, responses were summarized per question.

3. Results

3.1. Prestudies

3.1.1. Physical Characterization of Vibrating Platforms

All measurement data are available as Supplementary Material. Figure 6 displays measurements when no one is standing on the platform, and the microphone is placed next to the accelerometer. Measurements when the researcher was standing or lying on the platform versus sitting in a wheelchair placed on top of the small platform are displayed in Figure 7. As expected, differences in resonance behavior can be observed between the two platforms. The most prominent acceleration peaks for the small platform are located at 75 and 797 Hz with a “hump” around 180 Hz. Corresponding peaks for the large platform appear at 64, 113 and 237 Hz. Resonances are shifted if the body weight of a human is placed on top of the platform (as can be seen in Figure 7), with different effects if the subject is standing, sitting in a wheelchair, or lying down on the platform.

3.1.2. Observation of Music/Dance Lesson

Since the current theme at the school was “Pakistan”, most of the music played followed that motif. A summary of the structure of the lesson is presented in

Table 2. Activities during the observed music/dance lesson, where W = wheelchair, and F = floor.

Step	Activity	Description	Location
1	Starting song	”Babblarna–Dansa med mig”	W
2	Dance session	Teacher/teaching assistants and students dancing together	W
3	Hide-and-seek game	“Gömma knuten”	W
4	Dance session	Everyone dancing together, after teacher demonstration	W
5	PODD discussion	“Do you want to dance more?”	W
6	Dance session		F
7	PODD discussion		F
8	Dance session		F
9	PODD discussion		F
10	Parachute	Textile moving to the sound of the music	F
11	PODD discussion	“Do you want to dance more with the parachute?”	F
12	Parachute		F
13	PODD discussion		W
14	Dance session	Slower music	W
15	Musical instrument play	Maracas, drums, rainstick, egg shaker	W
16	Final dance session		W

. A lot of focus was on dance, with pauses between each music excerpt and PODD discussions. For example, in step 2 (see Table 2), the teacher and teaching assistants were moving and spinning the wheelchairs, making dance gestures with their hands. The students responded through dance movements performed with their hands, sometimes followed by clapping. After a number of dance sessions, the teacher brought a large “parachute” (a textile) in different colors into the room. This parachute was moved to the sound of the music, see Figure 8. Some students were mostly observing, while others actively interacted with the parachute using their hands or legs. At the end of the lesson, there was a jam session with musical instruments (step 15). The students could then pick an instrument (options were maracas, drums, rainstick, and egg shaker) and play together.

3.1.3. Interview with Teacher Proxies

From this interview, we learned that all music/dance classes started with a song, to announce that the lesson had begun. The music was usually played from a Bluetooth speaker connected to an iPad. Selected music pieces were repeated throughout a thematic period since, as described by the teacher: “Our students need a lot of repetition to learn, and therefore we work thematically and we have more time to listen to a country’s music.” We discussed how they work with multisensory experiences to strengthen the focus or understanding of the music. The teacher described: “The parachute is also that, a bit: it makes the music visible.” One of the teaching assistants had previously described that the parachute (see Figure 8) often was used outdoors as a musical game. She mentioned that the school had visited Sound Forest previously and that they would like to have something similar for the students at the school: “(...) we felt we wanted to return to [the installation] with our students. Because it suited them, really.” The teacher and teaching assistants also described that some of the students at the school have epilepsy but that the lights in Sound Forest had not been an issue when they visited last time.

Regarding different interactions that may take place during the music lessons, the teacher explained: “With arms and hand claps (...) they show using their body language that it is fun, that they participate voluntarily.” One teaching assistant described that they would interrupt at certain points of the music to ask the students if they like the music, using PODD: “(...) and then they can elaborate on the answer, if they want to”. She continued: “The students [may] answer yes or no, but, usually the students have different ways to answer yes (...) [for example] they show it with a smile, with joy (...) if we were to obtain a negative answer, then maybe they would push away with the hand, or turn their face away, they show with the body language that they might not like that music.” The students may also signal their preferences by pointing on pictures in the PODD books or by making sounds. Another teaching assistant filled in: “They still show if they do not want to stay”. When discussing how we could ask the students about their sound and music preferences and adapt the installation to those, the teacher suggested asking the students using PODD. She also proposed to record a dance lesson to see the students’ reactions to certain sounds. The teacher commented: “Especially the students that you observed during the [music/dance] lesson very clearly show their opinion using bodily expressions.” The teacher and teaching assistants also described that the students had very different musical preferences.

We also asked about possibilities for interaction in Sound Forest and how we could adapt the experience to the students’ motor abilities. This involved discussing if it would be interesting to experience the installation through the body through active listening or if emphasis should rather be on active play. One teaching assistant described: “I think that some, those who can pull, can pull and feel if it is vibrating. Those who cannot pull, maybe you can (...) help them so they can feel.” The teacher and teaching assistants agreed that lying down on the platforms to listen to the sounds might suit some students, whereas triggering sounds yourself might suit others. They believed that the student group that we had observed during the music/dance lesson would be able to interact with the strings. Moreover, they also commented that it is important that the strings are not too fragile. When discussing how to adapt the experiment procedure to the student’s needs, the teacher and teaching assistants mentioned that the student group usually cannot concentrate for more than 30 min. Furthermore, they would likely not be bothered by the fact that researchers would be present in the room during the experiment. Finally, the teacher and teaching assistants emphasized the importance of clearly describing to the student what would happen during the experiment in the installation.

3.1.4. Questionnaire

Regarding musical interest (where 0 corresponded to strongly disagreeing, and 10 corresponded to strongly agreeing that the child is interested in music), the parents of P1 and P2 reported a level 4 interest, P3 a level 10, and P4 a level 8. None of the parents reported that their child could play a musical instrument or sing. The parents of P1 described that they could play music to her, but that “(...) it’s hard to know if she appreciates the music or if it’s the instruments themselves that attract her.” Moreover, they mentioned that “she has difficulty coordinating her hands and understanding how to play by herself.” P1 had not tried out any music technology or other electronic music-making tools. The parents explained that “she gets stuck on songs, probably she associates them with something positive that happened, or this is just a coincidence. But [her music interest] spans across many genres.” The parents of P2 reported that she likes some of the musical instruments that are available at the school and that she tries to play on them. They interpreted it as though she would perhaps not be interested in playing a musical instrument but that she would want to “create her own sounds”. They did not know if P2 had tested out any music technology or electronic music-making tools. P2 has a broad taste in music, from children’s songs to hard rock. The parents of P3 reported that he “uses [musical] instruments in his own way and creates sounds and rhythms”. He had also tested a synthesizer. P3 likes loud music with a lot of rhythm, Christmas music for children, children’s music, pop music, music from the Swedish music competition Melodifestivalen [107], and Disney music. In particular, he likes Katy Perry, Tones and I, Laleh, Freddy Kalas, and Pidde P. The parents of P4 reported that he had not tested any music technology. They did not mention any specific musical preferences or preferred artists/songs.

Analysis of all questionnaire replies suggested that the music mentioned by the parents was characterized by a clear rhythmical structure and clear repetitions. We could also conclude that the students had a very diverse taste in music overall.

3.1.5. Music Listening Sessions

Results from the above-outlined steps prompted us to include a test stage to explore the students’ reactions to different musical elements and genres. From the discussion with the teacher, we could conclude that playlist 1 was generally more popular than playlist 2. Through dialogue with the teacher, we understood that it appeared as though the students listened more actively, focusing, when enjoying certain sounds. Some sounds were more popular than others, and several students clearly showed that they liked specific excerpts. The students also communicated like versus dislike using PODD.

Analysis of videos suggested that excerpts 7 (category 3) and 8 (category 2) were not appreciated (see Table 1). We hypothesized that this might be since these sounds shared similar timbral properties (a lot of noise) and that excerpt 7 also was rather static. A conclusion that could be drawn from this was that the rhythm and timbre of the sounds to be used in Sound Forest should evolve significantly over time. The forest excerpt (excerpt 3, a soundscape with bird songs) was popular among the students. Other excerpts that seemed to work well were 4 and 12 (category 1), 5 and 11 (category 2), and 10 (category 3). Based on the observed reactions and the discussion with the teacher, we could not conclude that excerpts with clear rhythms were the only popular ones. Since the forest sound was popular, and complex rhythmical structures seemed to be well received too, we decided that the final sound design could make use of a soundscape recording in the background and that rhythmical elements could be mixed on top of this ambiance.

Finally, we concluded that ten minutes of music listening without breaks might be too long for the students. When it comes to the use of PODD, and if the students liked or disliked the music, the students appeared to reply most of the time. However, the teacher described: “It’s hard to say if they think about the music when they answer, or if they think about their play”, mentioning a situation in which a student responded that he did not like the music, but still “seemed to like it”. The teacher mentioned that other circumstances could influence the answers. She also commented that it is “hard to not be biased” when interpreting the students’ reactions.

3.1.6. Introducing the Notion of Musical Haptics through PODD

Example pages focused on music-related concepts from the PODD books are displayed in Figure 2. The PODD pages dedicated to music described the following concepts: me, you, to play, to listen, to dance, to sing, to increase the volume, to lower the volume, to turn off the sound, instruments, albums, and songs. Images used to describe haptic sensations, i.e., the three themes judged to be important for the experiment in Sound Forest (described in Section 2.5.1), are displayed in Figure 2 and Figure 4. Apart from the pictures presented in these figures, the PODD books used during the experiment also included images that allow you to describe if you like versus dislike something. This was represented by a happy versus sad emoji.

3.1.7. Music Listening Sessions with Haptic Music Players (HMPs)

A conclusion from the conversations with the teacher was that a key design consideration for the HMPs was robustness and durability. It was likely that the devices would sometimes be thrown to the floor. A lot of the design process was thus spent on creating robust embedded cable solutions and customized padded structures that would protect the hardware. As suggested by the teacher, we embedded the hardware of two of the HMPs into plush toys. This way, the objects would encourage and invite the students to touch. A detailed account of the development and evaluation process is presented in [90]. The evaluation interviews revealed that the students managed to actively engage with the devices using different modes of listening, both individually and as a group. For example, they used their hands and faces to explore the vibrations. The teacher confirmed that the design of the objects seemed to work in the sense that the HMPs caught the students’ attention. The students managed to successfully explore musical haptics using the haptic pillow (HMP 1) and the custom-built Chewbacca plush toy (HMP 3). The teacher had experienced technical problems with the plush toy backpack (HMP 2), so this device was only explored without haptic feedback. All HMPs were durable (i.e., did not break as a result of use). The teacher confirmed, after playing the test sample simulating the sound to be heard in Sound Forest using HMP 1 and HMP 3, that the sound worked well for the students.

3.2. Experiment in Sound Forest

3.2.1. Multisensory Experience Design

Informed by all the steps outlined above, author 2 created music for Sound Forest, focusing specifically on what we had learned about the student group and their preferences, needs, and potential challenges. The general sound design idea behind the installation was one of a calm and relaxing forest. It was realized with SuperCollider [108], using both processed pre-recorded sound samples and fully synthesized sounds, i.e., sounds completely generated from the software using audio synthesis techniques. For the interaction with the strings, we used the sound of a Tibetan singing bowl and the one of a synthesized wooden-like percussive instrument. The former is an inverted bowl-shaped bell, often associated with relaxation and meditation. The latter is a synthesized sound built from a bank of eight resonators, tuned to resemble the timbre of a percussive instrument in between the timbre of low marimba or a Lujon. The Lujon is a bass metallophone with a wooden box resonator. When a string is hit, the two sounds are played simultaneously. The recorded sound of a struck Tibetan singing bowl (with fundamental frequency $f=559.8$ Hz, approx. C♯5) is played by the corresponding string’s ceiling speaker. The synthesized sound is played by the actuators in the floor and by the ceiling speaker (with an amplitude scaled to a factor of $0.02$). The amplitude of both sounds is fixed and not mapped to the detected onsets’ amplitude. This design choice was made to allow all students to experience the same sound output and vibration level, regardless of how much force they would use to hit the string. The more frequently the string is hit, i.e., if the time in between detected onsets is decreasing, the higher the pitches of the produced sounds. In particular, the pitches of the Tibetan singing bowls are chosen from a Mixolydian mode with just intonation built on the bowl’s fundamental pitch. The pitches of the synthesized sound are chosen from an array of fixed pitches (ranging from MIDI note 41 to 63, i.e., approx. from 87 to 311 Hz). These pitches were compatible with the pitches of the Tibetan singing bowl and the tech specs of the haptic floor. If the time between two consecutive hits decreases, then the pitches slowly go back to the fundamental ones. All the five strings had a similar structure in terms of interactions and sounds. The only difference between strings was that the starting pitches of the Tibetan singing bowls were different. The different starting pitches all belonged to the same musical mode, thus allowing the students to play together without creating redundant or cacophonous results.

In order to make the experience more immersive, we added two background sounds that accompanied the interactions with the strings. The first background sound was a natural soundscape sample. This sound was chosen not only to enhance the metaphor of the forest but also because we noticed that the students reacted positively to nature sounds (see Section 3.1.5). The soundscape used was an excerpt from a recording of a natural environment in San Martino Valle Caudina (Avellino, Italy), realized by sound artist Giuseppe Pisano [109]. This natural soundscape was filtered with a high pass filter with a cutoff frequency of 500 Hz and played back in a loop only by the ceiling speakers. For the second background sound, we were inspired by the students’ music listening sessions, in which we noticed that they found ambient and drone-like pieces interesting as long as they presented some variation over time. Therefore, trying to fulfill these aspects, we created a generative ambient background from a digital processing of the same sample of the Tibetan singing bowl. This second background sound was played by the ceiling speakers and the floor actuators with a low pass filter set to 400 Hz so as to not interfere with the other sounds. The haptic sensation was playing continuously and independently from the interaction with the string. This guaranteed an evolving vibration from the floor for the haptic conditions, regardless of how much the students would interact with the string. Tuning of the sound levels of Tibetan singing bowls, synthetic marimba-like sounds, and background sounds were realized empirically at the museum. We carried out the tuning with the installation up and running without interrupting the code, i.e., with all sound and vibration elements turned on. The levels were adjusted so that the sounds and vibrations for the small versus the large platform would be comparable. Sound examples of what the interaction at a single string sounds like, with and without the background sounds, are provided as Supplementary Material.

Regarding the light design, we decided to insist on the metaphor of the forest already pursued with the sound design. We decided to have the strings glowing in a green color. When an interaction with the string occurs, the string becomes very bright and purple, eventually fading into green again. The combination of green and purple, together with the change in brightness, was selected in order to make the installation accessible also for those with color blindness. A picture of the light design in idle mode is presented in Figure 9. An example video of an interaction with one of the strings is provided as Supplementary Material.

3.2.2. Video and Log Data

The total number of interactions per condition is presented in

Table 3. Total number of onsets per condition (WH = wheelchair-haptics on, W = wheelchair-haptics off, FH = floor-haptics on, F = floor-haptics off), with onsets triggered by the student presented in brackets. String labels: L = large platform, S = small platform.

Participant	String	Dur	Order	Onsets
				WH	W	FH	F
P1	4 (S)	123	W WH F FH	147 (0)	8 (8)	71 (0)	80 (0)
P2	1 (S)	54	WH W FH F	89 (2)	11 (3)	12 (1)	0 (0)
P3	2 (S)	168	WH W FH F	63 (53)	30 (27)	53 (2)	0 (0)
P4	3 (L)	618	W WH F FH	160 (102)	339 (339)	23 (23)	64 (64)

. The table displays the raw onset data, that is, the total number of sounds triggered by either a student or a teacher/teaching assistant in a specific condition. The sounds triggered by the student were manually counted and are presented in brackets. Synthesized sounds and plots for all students and conditions are available as Supplementary Material. A summary of interaction strategies used by the respective student is presented in

Table 4. Means of interacting with the strings. Stars (*) denote primary strategy (i.e., the method most commonly used by the respective student).

Interaction Strategy	P1	P2	P3	P4
Enclose	X	X *	X	X
Shake (small movement)	X			X
Hold (using two hands)	X *		X	X
Apply pressure (with thumb)	X
Pull		X	X	X *
Contour follow		X
Strike (hand)			X *	X
Strike (head)				X
Bite				X
Push away (with palm)				X

. The change in the number of onsets over time suggested an overall tendency towards fatigue as the experiment progressed, at least for P1, P3 and P4, see

Table 5. Onsets per condition, sorted by sequence order. Condition number 1 corresponds to the first condition that the students were exposed to, whereas condition number 4 corresponds to the last condition.

Participant	Condition Number
	1	2	3	4
P1	8	0	0	0
P2	2	3	1	0
P3	53	27	2	0
P4	339	102	64	23

. No clear difference in onsets between haptic versus nonhaptic conditions could be observed. The haptic condition in wheelchair (WH) had a mean of 39.3 (median = 27.5, SD = 48.5), whereas the nonhaptic condition in wheelchair had a mean of 94.3 (median = 17.5, SD = 163.5). The corresponding numbers for the floor conditions were 6.5 for the haptic condition FH (median = 1.5, SD = 11.0), and 16.0 for the nonhaptic condition F (median = 0.0, SD = 32.0).

P1’s main interaction strategy was to hold the string firmly with two hands. In the first condition (W: wheelchair-haptics off), P1 was exploring the string slowly with careful movements, holding it with one or two hands. She produced eight sounds but spent a large portion of the condition not engaging with the string. For the second condition (WH: wheelchair-haptics on), the teaching assistant showed her how to play, guiding her hand using different methods (e.g., grasping around the string, and striking it gently). Despite the encouragement, P1 did not play herself. In the third condition (F: floor-haptics off), the situation was similar; she did not play herself but the teaching assistant was playing for her. At one point, she decided to lie down instead of sitting. In the fourth condition (FH: floor-haptics on), she was lying down on the floor, actively observing when the teacher was playing.

P2 spontaneously explored the string in the first condition (WH). She triggered two onsets (i.e., sounds) for this condition. Her main interaction strategy was to use one hand and gently touch and pull the string. In the second condition (W), P2 did not spontaneously interact herself, but a teaching assistant encouraged her to touch the string, directing her hand to it. In the third condition (FH), P3 only tried to interact with the string after a teaching assistant had demonstrated how to play on it. However, her hand movements were so careful and soft that the sound was not triggered more than once. In the last condition (F), P2 did not interact with the string at all and instead focused on playing with her legs. In the previous condition, she had described in a PODD discussion that she wanted to do something else since the task was “too easy”. Therefore, the teaching assistant did not encourage her to continue playing.

P3’s main interaction strategy was to strike the string. In the first condition (WH), he mostly focused on hitting the string with one hand, using a lot of energy. Sometimes he also pulled the string back and forth. This session involved quite a bit of interplay, with the teaching assistant playing first, followed by P3 playing. In the second condition (W), P3 spontaneously played by himself. Sometimes he also clapped his hands and made large arm gestures, accidentally striking the string. He also chewed on a toy for quite some time. In the third condition (FH), P3 appeared to listen attentively while feeling the vibrations in the body. He was lying down almost all of the time, appearing concentrated. P3 did not play himself in this condition, but at one point when the teacher stopped playing on the string, he grabbed her hand to ask her to continue playing. In the last condition (F), P3 did not focus on the string. Instead, he was moving around a lot in the room, spinning and socializing, moving towards P4’s platform.

P4 was the one who used the large platform. His main interaction strategy was to pull the string back and forth. He created sounds that were a bit different from the other students’ sounds. This can be explained by the fact that he managed to trigger the onsets with a higher frequency, resulting in several pitches building up after each other, as can be seen in Figure 5. In the first condition (W), he used many different strategies, the most common ones being to shake and pull the string back and forth. P4 also struck the string, whereafter he resumed pulling it. At times he used two hands and was also biting the string at the same time. Finally, he used the string to push himself back or forth so as to navigate his wheelchair. P4 also used nonverbal sounds and hand clapping. In the second condition (WH), he used similar interaction strategies. At the beginning of the condition, he seemed to listen attentively and respond to the teacher playing on the string. He clapped his hands several times during this condition, both in the air and on his legs. He also made large hand gestures accompanied by nonverbal sounds. In the third condition (F), he mostly held the string with two hands and rested his head against it, moving slowly. A couple of times, he used his head to trigger sounds. In this condition, he also clapped his hands and produced nonverbal sounds. He was also drumming on the floor. In the last condition (FH), P4 did not interact very much. Similar to the previous conditions, he leaned his head on the string and used his hands to trigger some sounds. Perhaps he was tired at this point since he had been very engaged already from the beginning of the session. In fact, he was playing vividly with the string as soon as he entered the installation, even before the experiment had started.

3.2.3. Body Maps

Body maps filled out for the haptic condition when sitting in a wheelchair (WH) are presented in Figure 10. These results can be compared to the body maps for the haptic condition when sitting or lying on the floor (FH), as seen in Figure 11. Comparing the two figures, we can see that annotations seem to be different for the two conditions for participants P1, P2, and P4. For P3, the annotations look similar in both conditions.

All students had annotations of the head region for condition WH. An annotation for P4 suggested a sensation in the left ear. P2 also had an annotation for the left foot. Annotations for condition FH involved more regions overall. However, P2–P4 still had annotations of the head region also for this condition. Both P1 and P4 had annotations in the middle region of the body. Upon discussion with the teacher and teaching assistants, we understood that the annotations for P4 referred to the belly, whereas P1’s annotations referred to the behind. P2 had more detailed annotations with descriptive text mentioning the head and the face, the legs, and the knees. P4 had four annotations in total, including both hands. Discussions with the teacher and teaching assistants suggested that the hand annotations might either refer to having the hands on the platform or the sensation of holding the string, possibly feeling vibrations through it. Technically the string is decoupled from the vibrating platform, but perhaps there is some leakage.

Regarding the question of whether the students liked versus disliked the vibrations, all answers for the haptic condition in wheelchair suggested that the students liked the sensations. The teacher and teaching assistants sometimes specified whether these responses were based on a PODD discussion or if it was how they interpreted the situation (or both). For the haptic condition on the floor, the teaching assistant described that P2 answered “did not like” and that this was because of the “bright light and that it was too easy”. P3 used PODD to reply that he did not like the vibrations, but the teacher interpreted it as though he liked them. Both P1 and P4 reportedly liked the vibrations for the floor condition.

3.2.4. Micro Phenomenological Interviews with Observers

Through micro phenomenological interviews, observers within the research team focused on specific aspects of the students’ experiences. Observer 1 (O1) focused on the interplay and communication between the students throughout different passages. Observer 2 (O2) primarily focused on P4 (but also briefly mentioned interactions between P1 and a P3). Observer 3 (O3) described her observations of P1’s responses. Analysis of the interviews revealed several instances in which the observers had noticed the same events. Three main themes could be identified. All observers identified events within respective themes.

Theme 1 revealed that the students were able to engage in different types of musicking [110,111]. The term musicking refers to music as a verb rather than a noun, i.e., “to music”. This connects to the idea that the essence of music does not lie in musical works but rather in taking part in the performance, i.e., in social action. The observers identified moments of active listening, which included listening actively to the haptic feedback, as well as social dimensions of the musical experience and musical play. For example, O3 interpreted P1’s bodily expressions as attentively listening to the musical vibrations when lying down on the floor, relaxing. Another example was the observation by O1 and O2 that P3 was actively moving towards the larger platform, where P4 was interacting with the string, perceiving vibrations. This event was also mentioned by the teacher during the final interview (see Section 3.2.5). The teacher interpreted this behavior as P3 (possibly) wanting to move closer to the vibrations. O2 described a moment in which P4 seemed to be deeply immersed in the musical experience, commenting that he was “clapping, smiling, screaming and then repeating [the gesture]”. O2 also elaborated on an instance when P3 seemed to realize that the vibrations had been turned off. This prompted him to play with lower frequency (i.e., fewer onsets per second): “This kid was hitting faster when he also felt the vibration. I think I kind of spotted a little bit of disappointment in his face when we turned the vibration off, and then he was hitting and then vibration was not there”.

Theme 2 highlighted challenges that could arise when comparing PODD responses to observed body language. To stress the relationship between these two sources of information, O3 described how she felt particularly satisfied when noticing an overlap between P1’s responses using PODD and her own observations. She explained that it can take quite a bit of practice to learn how to use PODD, commenting: “I think they find it quite hard to use the pictures and talk with them”. She mentioned one successful example in which P1 used PODD to describe her sensation when lying down on the floor. P1 had PODDed that she felt vibrations in the behind, which made sense to O3 since this was a haptic condition (FH).

Theme 3 focused on discrepancies in interpretation of bodily expressions among different observers. In particular, O1 and O2 both expressed concern about the fact that P1 was not moving much when on the floor. They were worried that she was not enjoying the musical experience. Describing her previous knowledge of P1’s habits, O3 commented how P1 was making “grumpy noises” before being placed on the floor. She stopped making such sounds once the interaction with the string had started. This observation supported her interpretation of P1’s lack of discomfort while on the floor: “[P1] really likes her wheelchair (…) And then the teacher moved [the wheelchair] because she wanted her to focus on their string. And I know that sometimes...when she [P1] starts to be like that, she can be quite hard to...you know, to obtain to want to do the task again. But [when she was put on the floor] she...just stopped [and] seemed very comfortable with the situation”.

3.2.5. Final Interview with Teacher Proxies

Stimulate and Recall

In this section, we refer to the teacher proxies as T1–T3, where T1 corresponds to the teacher responsible for the music theme at the school, and T2 and T3 to the two teaching assistants. A summary of themes discussed during the stimulate and recall session is presented in

Table 6. Themes discussed in the final interview with the teacher and teaching assistants (N = total number of instances).

No.	Theme	N
1	Interpretation based on previously displayed behavior at school	13
2	Interpretation of PODD images and body maps	6
3	Complexity in interpretation (e.g., discrepancies between No. 1 and No. 2)	6
4	Expressions of liking	6
5	Reactions to/strategies for the creation of sounds (in other contexts)	5
6	Expressions of comprehension (here refers to the interaction with the string)	4
7	Expressions of disinterest or fatigue	3

. One of the most commonly used strategies when interpreting the videos was to describe interactions by referring to previously displayed behaviors at the school (Theme No. 1). For example, when watching a video of P2, who did not move very much in the specific video clip, T3 and T2 commented that this is usually how P2 is sitting when waiting for something to happen, or for some new input, play or interaction. Another example was a video in which P3 made large flowing hand gestures. T3 described that the behavior could be caused by seeing himself in one of the mirrors: “He likes it when you stand in front of him and make dance moves, and he wants you to do more. Sometimes when he sees himself in the mirror he can sit like that [waving his hands]”. After watching a video of P3 spinning around on the floor, T2 and T3 described that by spinning and looking around, he was “creating sensations”. In another video, P4 interacted a lot with what was going on around him, smiling, clapping, and making nonverbal sounds. T1 and T3 described that his behavior might reflect his technical interest. They commented that P4 is very interested in wheelchairs, and perhaps also cameras. His reactions could possibly be explained by the fact that he noticed that such objects were coming closer to him.

Another common theme concerned the interpretation of body maps in relation to the PODD images (Theme No. 2). There were situations in which these tools seemed to work well to communicate felt experiences, but also examples of contexts when it was a bit more challenging to interpret the responses. For example, T2 mentioned, when looking at a summary of all body maps for condition WH: “In the face, they said! It is cool that everyone thought the same.” T1 also commented: “What was very interesting, I remember, was that at first [P3] was lying in the other direction, so his head was facing the other side and his feet were facing the vibrations. And then he PODDed that he felt vibrations in his head, and then I turned him over, since his head wasn’t on the platform that was going to vibrate (...) He is very mobile. So if it would have been something that he didn’t like, he would be able to move”. The discussion also highlighted the importance of the exact image representation and text printed on the PODD images. For example, there was a discussion about the image for “foot” in relation to P2’s body map (see Figure 10). Since the PODD image depicted a single foot, students would have had to point to their own feet to describe that they were, in fact, referring to both of them. A similar discussion about singular versus plural arose about the annotation of P4’s ear in Figure 10. In this case, it is also difficult to know if the ear symbolizes hearing or a sensation of vibration. T1 and T2 also commented on the meaning of the annotation of the hands for P4 in Figure 11. They hypothesized that this annotation could either refer to a sensation felt when placing the hands on the floor or a sensation when holding the string. The discussions with the teacher and teaching assistants also highlighted the limitations of the body map representation and the difficulty in providing an unanimous interpretation of such a map. For example, an annotation for P1 in Figure 11 was interpreted as the lower belly region by the authors, but the T3 described that the annotation was, in fact, referring to the behind.

Theme No. 3 focused on the complexity of the interpretation of the students’ behaviors. This refers to instances in which there was a discrepancy between what was said using PODD versus how the teacher proxies interpreted the bodily expressions of the students. For example, in one video where P3 had replied that he did not like the vibrations, T1 described that she had specified in the body map that she “thought he liked it, because he was clapping his hands (...) he seemed to be excited”. She also stressed that her interpretation was based on the fact that he behaved in a way that was similar to how he usually acts when he is excited. T3 confirmed that you usually can see if P3 likes something, but that “Usually when you evaluate, he always goes towards not liking, so I’m thinking if that’s the picture itself, because it’s green. Since the other one is yellow”. T2 then filled in: “We’ve talked a lot about whether it’s adequate what they’re answering. If we as staff put in [our interpretation], you can see that they like it. And yet they answer the opposite. And it’s really hard to know, of course”.

Theme No. 4 concerned gestures expressing that you like something. For example, after watching a video in which P3 was jumping so much that his wheelchair looked like it was gonna fall over, T3 described: “I usually like to interpret it as he is getting this excited when something happens and is just like ’that sound, what’s is that?’ ” Another example was when P3 reached for the hand of T1, to guide her to play for him. T2 commented: “He’s very clear when you’re with him, he likes it when you do this, for example, [shows hand gestures] and then he can kind of come and turn your hands, and trigger them. And we understand (...) He tells us what he wants”. Another example was an interaction between P3 and P4, where P3 was moving towards P4: “And at this point he discovers that it is the other person [P4] who is receiving [vibrations]”. T1 suggested that this movement could possibly have been triggered by wanting to come closer to the vibrations since no haptic feedback was provided for P3 at that point.

Theme No. 5 consisted of quotes focused on strategies for the creation of sounds, preferences for certain sounds, or reactions to sounds. For example, T2 commented regarding P4: “If you drop something and it rattles, then, he thinks that’s really funny”. Another example was a discussion about P1’s interaction with the string in condition WH. She triggered a few sounds but spent approximately 2 min carefully exploring the string. When asked if the string was perhaps not sensitive enough for P1, T3 responded: “Well, maybe. Since what she does here at school is that she can walk up to a favorite chair or a swing and make it bang on the floor. So she wants that sound. So maybe that’s it”.

Theme No. 6 focused on aspects related to understanding versus not understanding how to interact with the strings in order to create sounds. For example, T2 described, after watching a clip where P4 was clapping his hands a lot: “The question is whether he didn’t understand how to get [the string] going. But then he got it. I think he wants something to happen”. Another example was when P3 was holding the string with two hands, biting it. T3 and T1 commented that he usually has things, or toys, in his mouth, and that this is a way for him to explore things.

Finally, Theme No. 7 explored ways of expressing disinterest or fatigue. For example, T3 described that P1 usually wants to lie down when tired. This was the reason why she was lying down on the floor, as opposed to sitting, not moving so much, in the last condition.

Concluding Questions

Question No. 1 focused on how appropriate the music, lights, and vibrations were for this particular user group. The teacher and teaching assistants made positive comments about the interplay between these three feedback types and how this formed a multisensory experience. T3 described: “I know I had a thought when I was sitting there with her [P1], just listening...like you could come here to meditate...and just sit here and pull [the string]”. The teacher proxies were not sitting directly on the vibrating platforms but just next to them, so they did not have the same experience as the students. T2 expressed an interest in visiting the installation to try out the experience herself to know in which body parts she would feel the vibrations. T1 also agreed that this would be useful. T2 mentioned that they initially did not know how the students would react when the lights were turned off in the room, but that this aspect had worked well and that the students seemed to feel safe. When asking if the vibrations were strong enough, the teacher and teaching assistants said that they were not sure but that the vibrations could perhaps have been a bit stronger. Author 1 described that the reason why we did not set a very high vibration level was that we did not want the experience to be too intense, thereby being perceived as unpleasant or scary. T1 then suggested that the strength could change over time: “(…) in some periods it could be soft and in some periods strong (…) so then everyone can adapt, so if they think that it’s too strong then you can just move away”.

Question No. 2 addressed what worked well versus what did not work well during the experiment. The teacher and teaching assistants mentioned that it would be good if some of the strings could be calibrated to another sensitivity level so that those who have less strength in their hands also can have a rich interactive experience. T1 described: “What I’m thinking about is maybe if it’s possible to change some strings, so that some you can just touch them a little bit... [then] something happens... So those who can’t pull that hard, they can play too”. This is also something that was mentioned by T2 during the Stimulate and Recall session.

Question 3 focused on suggestions of how the installation could be improved to be more accessible. The teacher and teaching assistants mentioned that it worked well to enter the room with four wheelchairs but that it would be tight, for example, for a student group of six students to fit. They commented that it would be nice if the room was even bigger since the student groups at the school usually consist of up to six students.

For Question No. 4, the teacher proxies were asked if they thought that the students had any preferences when it comes to the presence of haptic feedback, i.e., if they seemed to prefer having it on or off. T1 mentioned that it appeared to be more fun for the students when more sensory stimulation was provided: “It seemed to be more fun when they... when you pull the string and then something happens”. The others agreed with this statement. T1 followed up by suggesting that when there are no vibrations, there could be more of other types of sensory stimulation (e.g., a change in lights). There was also a discussion about whether it would be possible to ask this student group whether they preferred the vibrations to be on and off. T2 commented (and T1 agreed): “I think it would be very difficult to ask [the students] this question (…) We always try, when we evaluate, that you do it, either during, or right after. So that you have a chance to [get a response]. So maybe some of our other students [at the school] could have answered whether it was better with or without [vibrations], but I don’t think any of these students could”.

Question No. 5 focused on preferences for experiencing the installation by sitting in a wheelchair or being on the floor. The teacher proxies all agreed that preferences vary among different students. T2 mentioned that P3 (who likes to move around) and P1 (who likes to sit down) might have preferred to be on the floor, but that it probably depends on the ability to move. Regarding P1, T3 confirmed: “If she is more energetic, she likes to crawl around”. T2 continued: “Then you would have seen if she wanted to be on the vibrations or not. Because then you could choose to move away or to come closer [to the vibrating platforms]”. T2 also suggested that allowing for free exploration on the floor already from the beginning of the experimental session would maybe enable us to see if there is a preference for having haptics on or off.

For Question No. 6, we asked what an ideal multisensory room at their school would look like and how it would work. T2 stressed the importance of providing tactile experiences, e.g., by introducing objects that can be touched: “We also have some [pupils at the school] who like to discover using their mouths, which might be difficult in this context, but anyway, we have (…) paintings that you can hang on the wall, painting that you can touch”. T3 commented that this also allows for interaction for those students who cannot see. Another suggestion was to invite the students to Sound Forest and let them explore different places in the installation freely (with the help of the teachers, if needed). T2 commented on the role of the teacher in this context: “We are kind of co-discoverers in this context, it’s kind of part of the job, to rediscover things”. Finally, one conclusion that could be drawn was that 20 min is a pretty long time for sensory stimulation. Sound Forest is located outside of the school in a new place that the students are not used to. T2 described that if the installation had been located at the school, it would have been different: “Then you are already safe here, and you can go in and experience it and then come back later (...) that’s the advantage”.

4. Discussion

The prestudies described in Section 3.1 all provided information that was crucial to the planning and execution of the next steps of the project. For example, the physical characterization of the platforms provided valuable information about frequency responses. The results highlighted differences in resonances based not only on platform size and if the subject was sitting in a wheelchair, standing, or lying on the platform, but also based on different bodily properties. Of course, this affected how the multisensory experience was perceived by different students in the different conditions. The observation of the dance/music lesson enabled us to obtain insights into the music practice at Dibber Rullen and the ways in which the students danced to and interacted with music and musical instruments. We observed considerable differences among students in terms of how much movement that was performed to the music. We also concluded that most students used hand gestures and upper body movements to express themselves, although some also used their legs. The initial interview with the teacher and teaching assistants provided valuable insights into the pedagogical structure at the school and the thematic structure of the education. The questionnaire introduced the parents as proxies. This was an important step since it allowed us to obtain a better understanding of the students’ musical background and interests, going beyond what had been observed at the school. Results from the music listening sessions informed our sound design process. We noticed that although there had been a strong focus on dance and clear rhythms during the observed music/dance lesson, complex structures and soundscape sounds were also popular among the students. For the final sound design, we tried to avoid sounds with properties similar to the excerpts that were less appreciated (excerpts 7 and 8). However, it should be stressed that it is difficult to know why certain sounds appeared to be more popular than others, and fatigue also seemed to play a part in this context. Finally, we used Haptic Music Players to introduce the students to the PODD pictures used to describe musical haptics. This also allowed us to test the first version of the sound design for Sound Forest, before the actual experiment. Overall, the haptic pillow (HMP 1) and the customized plush toy (HMP 3) worked well, possibly since they had built-in speakers. However, the plush toy backpack (HMP 2) had to be connected to external headphones, thereby adding another step to the setup process. As such, it was more difficult to debug.

A number of conclusions could be drawn from the experiment in Sound Forest (see Section 3.2). First of all, the sound design used in the experiment appeared to work well in terms of creating a relaxing and ever-changing sonic experience, with a constant yet not overwhelming vibration coming from the haptic floor. Based on some of the feedback, the vibrations could perhaps have been even stronger. Analysis of video and onset data did not suggest any clear difference between haptic and nonhaptic conditions. Rather, we observed large interpersonal differences. This was reflected both in the interaction strategies used (see Table 4), duration of play (ranging from approximately 1 min for P2 to 10 min for P4), and the total number of sound-producing onsets (ranging from 6 to 528 onsets, see Table 3). Of course, the different sound excitation styles resulted in different sonic outputs for different users. For sound examples, please refer to the Supplementary Material, which includes resynthesized sound files for each student and condition. When it comes to the interaction, it should be stressed that we observed a tendency towards fatigue as the experiment progressed (at least for P1, P3 and P4). For P1, this effect was also confirmed by the teacher proxies in the final interview. This further complicates attempts to draw conclusions about differences between conditions.

Overall, using PODD and body maps to communicate about musical haptics with the students seemed to work well (see Section 3.2.3). The chosen strategy, which involved presenting a selection of bodily locations as images and then asking a follow-up question about like versus dislike, was mostly successful. There was indeed a correspondence between what the proxies observed and what was described using PODD, perhaps since the students had already practiced using the tools at the school. However, the interview with the teacher and teaching assistants also highlighted limitations of the employed methods. These limitations mainly concerned the translation between PODD images and body maps and the influence of the used text and picture representation. An example of a situation where the PODD images lacked level of detail was for communication of singular versus plural (e.g. “one foot” versus “two feet”). Pointing could be used as a method of clarification in such situations. Moreover, we also concluded that it would be good if the body map displayed not only the front, but also the back of the body.

Analysis of the micro-phenomenological interviews (see Section 3.2.4) revealed that the observers could identify moments of active listening, including active listening to haptic feedback. The observers also discussed and identified social dimensions of the musical experience and musical play. Moreover, these interviews also brought forward challenges that arose when comparing PODD responses to observed body language, as well as discrepancies in interpretation between different observers.

As can be seen in Table 6, a main conclusion from the Stimulate and Recall part of the final interview with the teacher and teaching assistants was that they often used prior knowledge about the students and their previously displayed behaviors at the school to interpret reactions in Sound Forest. Discussion during the Concluding Questions part of the interview also highlighted how the Sound Forest installation could be improved in order to be more accessible for children with limited movement. For example, the installation should ideally include certain elements or interaction points that could be tuned specifically to those who can move very little, in order to provide rich experiences for all. The multisensory design could also be customized to encourage more spontaneous discovery and support different types of felt experiences, depending on personal preference. One example of how this could be achieved, as mentioned by one of the teaching assistants, was to bring in elements that allow for tactile exploration for those who cannot see. For example, tactile elements could be placed on the walls of the installation. Analysis of the videos revealed that several of the students used nonverbal sounds and clapping to express themselves in the installation. We, therefore, believe that detecting such sounds (using, for example, machine learning techniques) and triggering sonic output based on such interactions could be one way of making the installation more accessible to users with limited movement.

Reflecting on the multisensory experience delivered in Sound Forest through the lenses of the three pillars of inclusive design discussed in [66], i.e., accessibility, usability, and value, a couple of conclusions could be drawn. Regarding accessibility, the installation setup appeared to provide access to creative activities for P3 and P4. It allowed for independent use, either immediately (as for P4) or after teacher demonstration (as for P3). However, the creative possibilities were somewhat limited for P1 and P2, who perhaps would have benefited from a higher string sensitivity. The sensitivity of the contact microphones was set to detect gentle and subtle touching. Unfortunately, the level was not high enough to allow all students, i.e., those with less hand strength, to easily trigger sounds. This issue could be solved by increasing the sensitivity of a specific string even more. However, this would add the counter effect of detecting triggers from the other strings, thus resulting in the installation playing sounds without physical interactions. The microphones would then pick up sounds from the room, not only sounds from the interaction with the string in question. These aspects should be considered in future versions of the multisensory design for Sound Forest, conceiving other ways of detecting interactions through sensing that could make the installation more accessible. In terms of usability, the interactions appeared to be rather intuitive for P3 and P4, whereas they were less so for P1. P2 specifically told the teaching assistant that it was too easy to play on the string, suggesting that she found the interaction intuitive. However, although she perceived the interaction as easy, she did not play so much. Regarding the creative value, it appeared as though P4 managed to explore a wide range of different sonic outcomes, using many different types of gestural techniques. P3 was also rather expressive in this context, playing on the string, making dance gestures, spinning around in the multisensory experience, and interacting with others who played. Interestingly, what we observed goes in line with the high levels of musical interest reported by the parents (a level 8 interest for P4 and a level 10 for P3, see Section 3.1.4). For P1 and P2, the expressivity of the string interaction could indeed be improved. However, it should also be taken into account that both P1’s and P2’s parents reported a level 4 of musical interest. In addition to that, P2’s parents commented that they did not think she would be interested in playing a musical instrument (but perhaps in creating “her own sounds”, see Section 3.1.4). One example of how the musical output could be made more rewarding in general would be to reduce the time required between onsets to achieve a raise in pitch. In the current version of the sound design, this musical effect was mostly explored by P4.

In this study we have effectively explored multisensory music experiences for a group of four students with Profound and Multiple Learning Disabilities (PMLD). We have presented methods that could be used to explore and describe experiences involving both sound and haptics. In our work, we attempted to gather as much information as possible through proxies, meaning that the parents and the teacher/teaching assistants played a vital role when it came to including the students as informants and co-designers of the multisensory experience design. Since the interpretation of what the students experience is complex to access, we decided to use a multifaceted variety of qualitative and quantitative methods to arrive at more informed conclusions. Findings from multiple time points in the research process highlighted the importance of combining different methods, as well as different proxies, both in the design phase as well as the evaluation of our work, in order to understand the complex interactions taking place. A main conclusion from the final interview with the teacher and teaching assistants was that they often used prior knowledge about the students and their previously displayed behaviors at the school to interpret reactions during the experiment. However, even with this prior knowledge developed through their practice, there were several situations in which it was complex for the teacher proxies to interpret some student reactions in relation to the PODD replies. This highlights the importance of using proxies in this context; no observation performed by a researcher, however complex coding schemes that would be used, could replace such input. Nevertheless, it should be noted that working with proxies is never the same as working directly with the users and that this method introduces an additional layer of ambiguity into the research process. Moreover, this methodology, especially when used in the wild at the school (without the researchers being present), puts a lot of responsibility on the proxies to actively engage the users in the co-design process.

4.1. Limitations

Our research relied largely on descriptive narratives provided by proxies. One of the limitations of the current work relates to the assessment of musical preference. Even if evaluation of the aesthetics of music occurs in most cultures, these processes are not yet fully understood [112]. Music preferences are also known to change over time, something that has been stressed in music recommendation literature. Of course, proxy evaluation introduces yet another level of ambiguity in the assessment process. Nevertheless, there are measures that can be taken to aid preference assessments in interactions between teachers and students with disability. For example, the importance of timing, i.e., when you ask a question, has been stressed [113]. In our current work, we have tried to address the above-mentioned limitations by combining several different methods of data collection, using different proxies (parents and teachers).

Another limitation of the presented work is the limited number of participants; results are not generalizable due to the small sample size. Our study design was partly based on research carried out in the wild. As such, the focus has been on obtaining enough experimentation time with the participants rather than on the number of participants, an important aspect of in-the-wild research stressed in [67]. As mentioned in [67], page 59, “It becomes much more difficult, if not impossible, to design an in-the-wild study that can isolate specific effects”. Nevertheless, the outcome of studies carried out in the wild can be most revealing and demonstrate quite different results from those arising out of lab studies [114], showing how people come to understand and appropriate technologies on their own terms and for their own situated purposes [67].

Finally, only a small group of researchers were involved in the qualitative analyses of the material presented in this work. It would, of course, have been better if a larger number of independent researchers would have performed the analyses.

4.2. Future Work

Based on the large interpersonal differences observed in the current study, we believe that future explorations in Sound Forest should include larger groups of participants in order to fully understand the range of interaction strategies that could be explored in the room. Future experiments in Sound Forest could allow for interactive control and adjustment of the haptic experience in real-time. One possibility could be to tune the sensitivity of each string to every child at the beginning of the session. Another option, as suggested by the teacher proxies, would be to program the haptic feedback so that it evolves over time and is different in different places in the room. The installation would, in that sense, invite to spontaneous exploration of the room; if certain users would stay in one place (where there are more vibrations), this could be used as an implicit method to draw conclusions about preferences for haptic feedback. In general, future experiments should allow the students to spend more time exploring the installation freely on their own terms, i.e., in a less controlled experimental setting. However, our results also highlighted that 20 min of music was quite long. Measures should be taken in order to reduce the risk of fatigue when planning time spent in the installation. Since the teachers are also co-discoverers in this context, it would also be good to invite them to explore the installation before the students are able to try out the multisensory experience.

5. Conclusions

This paper presented a 1.5-year-long research study exploring how different methods can be used to explore the concept of musical haptics with pre-verbal children with profound and multiple disabilities. We used a Participatory Design with Proxies (PDwP) methodology, allowing the children to be informants in the wild. The aim of this study was to create a customized multisensory experience informed by the user groups’ needs, musical preferences, and abilities. The designed multisensory experience was explored in an experiment with haptic feedback turned on or off. Results highlighted the diversity in interaction strategies used by the children, limitations in terms of the accessibility of the current multisensory experience design, and the importance of using a multifaceted variety of qualitative and quantitative methods to arrive at more informed conclusions when applying a design with proxies methodology. Our findings shed light on methodological and design considerations that should be taken into account when developing multisensory experiences informed by pre-verbal children.