Next Article in Journal
From Development to Validation: Exploring the Efficiency of Numetrive, a Computerized Adaptive Assessment of Numerical Reasoning
Previous Article in Journal
The Effect of Stress on Depression in Postgraduate Students: Mediating Role of Research Self-Efficacy and Moderating Role of Growth Mindset
Previous Article in Special Issue
The Complex Role Played by the Default Mode Network during Sexual Stimulation: A Cluster-Based fMRI Meta-Analysis
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Using Tangible User Interfaces (TUIs): Preliminary Evidence on Memory and Comprehension Skills in Children with Autism Spectrum Disorder

by
Mariagiovanna De Luca
1,†,
Ciro Rosario Ilardi
2,†,
Pasquale Dolce
3,
Angelo Rega
4,
Raffaele Di Fuccio
5,
Franco Rubinacci
6,
Maria Gallucci
4 and
Paola Marangolo
1,*
1
Department of Humanities Studies, University Federico II, Via Porta di Massa 1, 80133 Naples, Italy
2
Interdepartmental Research Centre on Management and Innovation in Healthcare (CIRMIS), University Federico II, Via Pansini 5, 80131 Naples, Italy
3
Dipartimento di Scienze Mediche Translazionali, University Federico II, Corso Umberto I 40, 80138 Naples, Italy
4
Neapolisanit Rehabilitation Centre, Via Funari, 80044 Naples, Italy
5
Department of Psychology and Educational Sciences, Telematic University of Pegaso, Piazza dei Santi Apostoli 49, 00187 Rome, Italy
6
Smarted srl, Riviera di Chiaia, 256, 80121 Naples, Italy
*
Author to whom correspondence should be addressed.
These authors contributed equally to this work.
Behav. Sci. 2025, 15(3), 267; https://doi.org/10.3390/bs15030267
Submission received: 20 January 2025 / Revised: 21 February 2025 / Accepted: 24 February 2025 / Published: 25 February 2025
(This article belongs to the Special Issue Neural Correlates of Cognitive and Affective Processing)

Abstract

:
Autism spectrum disorder (ASD) is a complex neurodevelopmental condition involving persistent challenges with social communication, as well as memory and language comprehension difficulties. This study investigated the effects of a storytelling paradigm on language comprehension and memory skills in children with ASD. A traditional approach, using an illustrated book to deliver the narrative, was compared to a novel paradigm based on Tangible User Interfaces (TUIs) combined with multisensory stimulation. A group of 28 children (ages between 6 and 10 years old) was asked to listen to a story over four weeks, two times a week, in two different experimental conditions. The experimental group (n = 14) engaged with the story using TUIs, while the control group (n = 14) interacted with a corresponding illustrated book. Pre- and post-intervention assessments were conducted using NEPSY-II subtests on language comprehension and memory. At the end of the intervention, a trend of improved performance was found. In particular, a greater number of subjects benefited from the intervention in the experimental group compared with the control group in instruction comprehension and narrative memory-cued recall. These preliminary findings suggest that TUIs may enhance learning outcomes for children with ASD, warranting further investigation into their potential benefits.

1. Introduction

1.1. Autism Spectrum Disorder (ASD)

Autism spectrum disorder (ASD) is a complex neurodevelopmental disorder marked by difficulties in social interaction and communication and the presence of restricted or repetitive behavior, all of which can greatly affect daily life and overall well-being (Hirota & King, 2023). Early diagnosis and intervention are essential in providing tailored support for individuals with ASD and optimizing their developmental outcomes. Although the exact cause of ASD remains unclear, research indicates that genetic factors are influential, with specific gene variations and mutations linked to an increased likelihood of ASD (McCrimmon & Rostad, 2014). Additionally, non-genetic influences, such as prenatal exposure to environmental toxins, maternal health during pregnancy, and early life experiences, may also play a role in the development of this profile (Landa & Goldberg, 2005; Lord et al., 2018).
The term “spectrum” in ASD reflects the wide range of symptoms and varying levels of severity experienced by ASD individuals. According to the DSM-5, ASD is categorized into three levels based on the severity of symptoms and the degree of support required (Faras et al., 2010; McDougal et al., 2020). Level 1 (Mild ASD), formerly referred to as high-functioning autism, includes those individuals who may face challenges in social communication and interaction. They often display restricted or repetitive behaviors but might also excel in specific areas, such as language and cognitive abilities. While less support is needed than in more severe forms, tailored interventions and assistance could enhance their quality of life. Level 2 (Moderate ASD) individuals experience more pronounced difficulties with social communication and often find it harder to “mask” or blend into social situations. Challenges with understanding social norms and maintaining interactions are common. They may also display repetitive behaviors, strong adherence to routines, and intense interests that stand out from neurotypical behavior. Moderate support is typically required to navigate daily life and social situations. Level 3 (Severe ASD) individuals, often referred to as low-functioning autism, have significant impairments in communication and social interaction. Self-regulation is more challenging, and they are at higher risk for neglect or discrimination. Communication, both verbal and non-verbal, can be difficult, and they may engage in pronounced repetitive behaviors. Social interactions are limited, and people often prefer solitary activities due to difficulty in understanding social cues and norms. Intensive support is necessary to help them engage meaningfully in daily life (Hirota & King, 2023; Tager-Flusberg & Joseph, 2003).
Several studies have shown that ASD individuals often experience difficulties in comprehending task instructions, with a general impairment in narrative comprehension (Brown et al., 2013; Lindgren et al., 2009). In a storytelling task, they may find it difficult to understand the structure and details of the story (Brown et al., 2013; Lindgren et al., 2009; Tárraga-Mínguez et al., 2020). Still, other studies have highlighted that individuals with ASD—whether high- or low-functioning—often exhibit deficits in verbal memory tasks, particularly when recalling word lists (Boucher et al., 2012; Bowler et al., 2004; Cheung et al., 2010; D. L. Williams et al., 2006). These memory troubles become more evident when support cues are absent.

1.2. Traditional Interventions for Autism Spectrum Disorder

Occupational therapy and Applied Behavior Analysis (ABA) are commonly used interventions for ASD that are aimed at empowering academic and functional skills, as well as visuomotor abilities (Fani-Panagiota, 2015). However, conventional occupational approaches, such as shape-matching training, often struggle to sustain engagement and adequately address the complex needs of individuals with ASD (Leonard et al., 2002). Still, while ABA-driven interventions have shown promise, their effectiveness remains debated due to methodological issues and limited generalization to real-life contexts (Yu et al., 2020). Furthermore, the high intervention intensity (up to 40 h per week) poses challenges for both children and their families (Leaf et al., 2022). Also, ethical concerns have been raised by autism rights and neurodiversity activists (Leaf et al., 2022), and uncertainty remains regarding potential conflicts of interest among researchers within the ABA industry (Bottema-Beutel & Crowley, 2021).

1.3. The Role of Technology

A growing body of research has highlighted the significant advantages of integrating computer technologies into autism rehabilitation, especially in fostering motivation, enhancing attentional focus, minimizing maladaptive behaviors, and improving learning outcomes. Furthermore, technology-based interventions provide consistency, predictability, and immediate feedback while establishing clear routines and expectations, all of which satisfy the needs of children with ASD (Silver & Oakes, 2001).
Overall, the most widely used technologies in ASD rehabilitation can be broadly classified into three domains: standard computer systems, robotic technology, and virtual reality. Nevertheless, the use of standard computer systems poses challenges for individuals with limited dexterity and fine motor skills (Sitdhisanguan et al., 2012). As for robotics and virtual reality, both approaches have shown promise in stimulating social skills and motivation (Dautenhahn & Werry, 2004; Leonard et al., 2002); however, they may be less effective in improving cognitive functioning.
Recent studies have emphasized the effectiveness of interventions that leverage multisensory integration (MSI) for individuals facing memory and language comprehension deficits (Parisi et al., 2022). Human communication typically relies on multiple sensory channels—such as facial expressions, affective vocalizations and touch—rather than solely on unimodal signal transmission. MSI is thus essential for enhancing both communicative and social behaviors (Parisi et al., 2022). Among the earliest senses to develop, the human sense of smell retains critical social communicative functions (Kumazaki et al., 2016; Muratori et al., 2017; Sweigert et al., 2020). Furthermore, the interplay between auditory and visual modalities is fundamental for processing speech stimuli (Stevenson et al., 2018; Van der Burg et al., 2008; Vroomen & de Gelder, 2000). Previous neuroimaging meta-analyses have identified a neural network involved in multimodal sensory input processing, which includes fronto-temporal and limbic regions. These brain structures support not only the strictly linguistic aspects of verbal communication but also its connotative features, such as affective and emotional tones (Boccia et al., 2016; Scheliga et al., 2023).
Some MSI-based interventions have been specifically designed for individuals with ASD. Among these, the “Snoezelen” approach (Kaplan et al., 2006) involves the simultaneous stimulation of multiple senses with the aim of reducing stereotyped behaviors (Kaplan et al., 2006). For children with ASD and limited verbal abilities, Parés and coworkers introduced “MEDIATE,” an interactive environment that generates real-time visual and auditory stimuli to promote creativity, exploration, and enjoyment (Parés et al., 2005). Existing research primarily focused on vision and hearing, largely overlooking the senses of touch and smell (Bara et al., 2018; Kumazaki et al., 2016; Muratori et al., 2017).

1.4. Tangible User Interfaces (TUIs) as a Novel Approach

In these last years, Tangible User Interfaces (TUIs) have emerged as a groundbreaking methodology. TUIs are physical interfaces that introduce a new way of “grasping” computer technology. Specifically, they allow users to engage with a computer system through a combination of physical/perceptual and digital actions. An intriguing application of TUIs is in the context of storytelling tasks within a mixed-reality environment (Ishii & Ullmer, 1997). In this scenario, TUIs facilitate user interaction by bridging real objects and digital stimuli within a given narrative framework. As TUIs support multisensory stimulation (e.g., visual, auditory, olfactory feedback), a TUI-mediated storytelling paradigm promoting MSI is expected to boost memory and language comprehension abilities (Di Fuccio et al., 2016; Ponticorvo et al., 2019). Interestingly, research involving typically developing or visually impaired children has shown that olfactory stimulation can positively influence text memorization (Garcia-Ruiz et al., 2021; Sorokowska et al., 2019).
An advantage of TUIs over standard technology protocols lies in their ability to harness physical object manipulation. Unlike traditional computer-based methods relying on symbolic or graphical representations, TUIs streamline human–computer interaction by integrating tangible objects. This feature decreases the cognitive and fine motor demands typically associated with tasks such as drag-and-drop operations using a mouse (Fails et al., 2005). When applied to storytelling tasks, TUIs can incorporate MSI into the learning process, likely producing greater engagement and improving cognitive outcomes. Moreover, by building semi-ecological learning environments that closely resemble real-world scenarios, TUIs may enhance generalizability. Indeed, children may find it easier to apply the skills they learned during training sessions to everyday life (Wynn & Smith, 2003).
In previous research, TUIs have been mainly employed in educational settings (Di Fuccio, 2022; Di Fuccio et al., 2016; Ponticorvo et al., 2019) to foster social competencies (Al Mahmud & Soysa, 2020; Farr et al., 2010; Villafuerte et al., 2012) and improve emotion recognition (Ramos-Aguiar & Álvarez-Rodríguez, 2021). Conversely, little is known about their impact on cognitive functioning (Al Mahmud & Soysa, 2020; Beccaluva et al., 2022; Escobedo et al., 2014; Sitdhisanguan et al., 2012). Applying a similar paradigm within a structured cognitive rehabilitation program may be highly beneficial. Similarly, multisensory stimulation, which activates brain networks essential for both verbal and non-verbal communication (Andrzejewski et al., 2019; Scheliga et al., 2023), may prove effective in improving cognitive skills in individuals with ASD (Rossiter & Garcia, 2010).
Building on these insights, the present study aimed at evaluating the effectiveness of TUIs, integrated with a multisensory approach, in improving verbal memory and instruction comprehension in children with ASD engaged in a storytelling task. These domains are significantly impaired in children with ASD (Boucher et al., 2012; Lindgren et al., 2009). Addressing these deficits might enhance the effectiveness of any subsequent rehabilitation interventions. The TUI-mediated intervention was compared to a traditional approach, where the same story was delivered in a conventional format using an illustrated book. In this regard, previous studies (C. Williams et al., 2002) have shown that children with ASD participating in computer-assisted reading sessions spend more time engaged with reading materials and demonstrated greater improvement in reading abilities, as well as less resistance than with the traditional book-based methods.

2. Materials and Methods

2.1. Participants

Following a convenience sampling method, consecutive individuals with ASD were recruited from the Neapolisanit Rehabilitation Centre in Naples, Italy, for rehabilitative purposes between February and July 2024. The Neapolisanit Centre is an accredited facility within the Italian national health system, specializing in education and rehabilitation for individuals with physical, psychological, and sensory disabilities. Its Autism Division (DAPI) is dedicated to supporting individuals with ASD and their families, aiming to enhance their quality of life and promote social inclusion. The center hosts a dedicated research division focused on autism.
For all eligible participants, clinical records reported a suspected ASD diagnosis with varying levels of severity, supported by the previous administration of the Autism Diagnostic Observation Schedule, Second Edition (ADOS-2) (Lord et al., 2012). This is a standardized assessment tool widely used in ASD diagnostics. Like many other psychometric instruments, the ADOS-2 requires advanced examiner training. Furthermore, its sensitivity may vary across different clinical settings (Lord et al., 2000). While considered a valid and reliable tool, it is most effective when used in conjunction with other forms of assessment and within the context of clinical observation (Gotham et al., 2007). Therefore, in the absence of a formal diagnosis made by an expert child neuropsychiatrist, participants were excluded from this study. Additional exclusion criteria were non-adherence to treatment, excessive concurrent training demands, and limited parental availability. Participants were stratified by age and ASD symptom severity before being randomly assigned to two groups (i.e., experimental group and control group) through a double-blind randomization procedure.

2.2. Apparatus and Stimuli

In this study, a story titled “A Summer in the Garden” was presented. The narrative followed the gardening tasks of the main character, “Pippo”, who seeks help planting a strawberry seedling at his grandmother’s country house. The story could be delivered orally by the examiner, supported by a booklet containing 14 illustrated pages, or through TUIs.
TUIs for digital storytelling are a methodology that leverages digital tools and technologies to convey stories, combining multimedia elements such as images, audio, videos, and objects to create engaging narratives. The digital version of the story, including 14 scenarios, was developed using the same 14 illustrations from the booklet, accompanied by audio narration. Additionally, participants were exposed to tangible objects or olfactory stimuli while listening to the story. The digital story was implemented using the STELT software (Smart Technologies to Enhance Learning and Teaching) (Miglino et al., 2013). In its digital version, the story was presented by means of a standard laptop with a Windows OS (Figure 1a) and two Near Field Communication (NFC) antennas (Figure 1b; (Di Fuccio & Mastroberti, 2018)). NFC technology facilitates interaction with real-world objects. This setup used two types of NFC devices: an NFC Active Antenna, which acted as an interface between the computer and the physical objects, and NFC Tags, functioning as Passive Antennas. The NFC tags embedded in tangible objects held information readable by the active antenna. When the tag approached the active antenna, the antenna emitted an electromagnetic field, allowing the recognition of the corresponding object. Various tangible objects (e.g., a butterfly, watering can, shovel, soil sack, vase, strawberry seeds, and strawberry candy) and scent containers with fragrances like rose, pine, strawberry, and damp soil were incorporated into the story, creating a multisensory experience by linking the tactile and olfactory sensations of the objects and scents with the story’s audio-visual elements (Figure 1d). To enhance participants’ engagement, a 3D-printed figure of the story’s main character, “Pippo”, was also included, allowing participants to interact more closely with the protagonist (Figure 1c).
In designing the apparatus for children with ASD, we adhered to some previously proposed guidelines (Sitdhisanguan et al., 2012):
Size of tangible objects: Tangible objects were designed to be appropriately sized, ensuring ease of grasping for children with limited fine motor skills.
Size of the board containing the NFC antenna: The board was sized to remain fully within the child’s arm’s reach, minimizing unnecessary movements and reducing potential distractions while ensuring comfort during the interaction.
Sound usage: Loud or abrupt sounds, such as buzzers or explosions, were avoided to accommodate the heightened auditory sensitivity often observed in children with ASD. Any warning sounds (e.g., to indicate an error; see Section 2.4) were emitted with soft tones.

2.3. Neuropsychological Testing

On admission, all participants were first screened through the ADOS-2, which is considered the reference standard assessment tool for supporting ASD diagnosis. This battery includes five different assessments: language and communication, mutual social interaction, imagination and creativity, stereotyped behavior and narrow interests, and other abnormal behaviors. For each domain, the appropriate module was selected based on the individual’s language level and age: Module 1 was used for children who did not consistently use verbal language, Module 2 for those who used phrase speech but were not yet fluent, and Module 3 for children and adolescents who spoke fluently. The calibrated severity score (CSS) was employed to standardize the raw ADOS-2 scores, ensuring that ASD symptom severity could be meaningfully compared both across individuals with different developmental levels and over time. The CSS scale ranges from 1 to 10: a score between 1 and 3 indicates the presence of low levels of ASD symptoms (i.e., high functionality); a score between 4 and 5 indicates the presence of mild-to-moderate ASD symptoms (i.e., moderate functionality); and a score between 6 and 10 indicates the presence of moderate-to-severe symptoms (i.e., low functionality) (Shumway et al., 2012).
Subsequently, specific subtests from the NEPSY-II (Cardillo & Mammarella, 2015; Korkman et al., 2012), a neuropsychological battery for preschool and school-age subjects (from 3 to 16 years old), were administered. Overall, the NEPSY-II encompasses five domains: (1) Attention and Executive Functioning, (2) Language, (3) Memory and Learning, (4) Sensorimotor Functioning, and (5) Visuospatial Processing. Its broad range of subtests offers flexibility, allowing the examiner to select tasks best suited to their research aims (Cardillo & Mammarella, 2015). For this study, we chose subtests evaluating the child’s ability to follow instructions (Language domain) and perform properly episodic verbal memory tasks (Memory and Learning domain). For each administered test, the environment was tailored to meet the specific requirements of the task. Only the materials required to complete each subtest were provided, with all other items removed from the experimental setting to minimize distractions. Also, such an arrangement aimed to reduce anxiety in the child, which might otherwise concern about their ability to successfully complete the task. Below is a brief description of each subtest employed in this study.
L1-Comprehension of Instructions (L1-C). This subtest is designed to assess the ability to perceive, process, and follow verbal instructions of increasing syntactic complexity and execute the corresponding actions. Specifically, the examinee is required to point to appropriate stimuli in response to oral instructions. The task materials consist of two pictorial stimuli: the first image features eight rabbits varying in color and size, while the second image displays geometric shapes differing in color and shape. After confirming the examinee’s ability to understand simple instructions (e.g., “touch mouth”, “touch head”), the stimulus images are presented, and instructions of progressively greater complexity are given based on the child’s age. The L1-C task can be administered to individuals aged 3 to 16 years. Its scoring range is 0 to 33, with higher scores indicating greater linguistic comprehension abilities.
M4-List Memory (M4-LM) and M4-List Memory Delayed (M4-LMD). These subtests are designed to assess both immediate and delayed recall of a supraspan word list, the rate of information acquisition, and the impact of proactive interference. In M4-LM, the examiner reads aloud a list of 15 words, and the participant is required to repeat them, recalling the words in any order. There are five learning trials, each followed by an immediate recall. The total number of words correctly recalled across the five trials constitutes the M4-LM raw score. After the fifth trial, the examiner presents a new list of 15 words, which the participant is also required to recall. Following this interference trial, a delay of 25–35 min occurs, during which the participant is not engaged in memory tasks. After the delay, the examinee is asked to recall all the words they remember from the original (first) list. The total number of words correctly recalled from the first list constitutes the M4-LMD raw score. These tasks can be administered to children aged 7 to 12 years. For M4-LM, the total score ranges from 0 to 75, while for M4-LMD, it ranges from 0 to 15. Higher scores reflect better episodic memory skills.
M6-Narrative Memory Free Recall (M6-FR) and M6-Narrative Memory Cued Recall (M6-CR). These subtests assess the ability to recall verbally organized material under conditions of free recall and cued recall. The task begins with the examiner reading a story aloud to the child. They are asked to repeat everything they can remember from the story (M6-FR). Then, the examiner poses specific questions about the story to prompt recall of story-related information (M6-CR). Three different stories are available, selected based on the child’s age. The total M6-FR score ranges from 0 to 20, with one point awarded for each correctly recalled detail. Similarly, in M6-CR, the examiner asks 20 structured questions targeting specific story details, with one point assigned for each accurate response. Overall, higher scores suggest better episodic memory functioning.
The clinical interpretation of NEPSY-II scores follows the framework of the standard scaled score (SS), which represents a child’s performance relative to age-matched peers. The SS framework has a hierarchical structure: SS range 1–3 (Well below expected level), SS range 4–5 (Below expected level), SS range 6–7 (Borderline), SS range 8–12 (At expected level), and SS range 13–19 (Above expected level). In this study, performance improvement was operationally defined as a transition from a lower SS range to a higher SS range. For instance, an increase from SS 2 to SS 3 would not be classified as a significant improvement, as both scores fall within the same SS range. Conversely, a change from SS 6 to SS 10 would be considered clinically meaningful, reflecting a transition from the “Borderline” range to the “At expected level” range.

2.4. Procedure

All participants included in this study underwent two training sessions per week for four weeks, each session lasting one hour. At the beginning of the first session, all children underwent neuropsychological evaluation through the selected NEPSY-II subtests. In the experimental group, TUI-mediated digital storytelling was used. Each participant was asked to interact with objects while listening to the story. Some of these objects were elements of the story, while some others acted as distractors. If the child placed the right object on the keyboard, the story proceeded into the next scene; on the contrary, if it was the wrong object, the story stopped, and a mellow warning sound was emitted by the computer. As previously reported, all objects provide the subject with multisensory stimulation as they can be seen and touched, and some of them can be smelled. In the control group, children were asked to listen to the story read by the examiner while following it in the illustrated booklet. At the end of the treatment, the three NEPSY-II cognitive tests were re-administered to each participant.

2.5. Ethical Approval

The data analyzed in the current study conformed with the Helsinki Declaration. Our named Ethical Committee of the Division for Educational Methodologies and Technologies for Autism of the Neapolisanit Rehabilitation Centre in Naples, Italy, specifically approved this study (protocol number IEC07/2023, February 2023). Prior to participation, the parents of the participants signed informed consent forms.

2.6. Statistical Analysis

Descriptive statistics were expressed as frequency for categorical variables and mean ± standard deviation (SD) for quantitative variables. Following verification of the relevant assumptions (e.g., normal distribution of residuals, homoscedasticity), between-group comparisons were performed using parametric (independent samples Student’s t-test) or non-parametric tests (e.g., 2-way χ2 test, Mann–Whitney U test) to determine whether the experimental and control groups were matched for sociodemographic variables, ADOS-2-related functioning level, and baseline (pre-test) neuropsychological test performance, as appropriate. An Aligned Rank Transform Linear Mixed Model (ART LMM) was constructed to investigate pre-post treatment differences in raw scores for each neuropsychological testing session. Specifically, the ART procedure involved preprocessing raw scores by removing all effects from the dependent variable except for the one being aligned. The aligned data were then ranked, with average ranks assigned in cases of ties. This approach ensures appropriate Type I error rates and adequate statistical power for main effects and interactions (Wobbrock et al., 2011). Subsequently, an LMM was applied as it is more appropriate for handling repeated measures data compared with traditional Analysis of Variance (Cnaan et al., 1997). The LMMs included fixed effects for the group (experimental vs. control) and time (pre-test and post-test), while participants’ ID entered the model as a random effect (Ilardi et al., 2025). The Satterthwaite method was used to approximate degrees of freedom (df) for the fixed effects. Variance components of the random effects were estimated using Restricted Maximum Likelihood (REML). Post hoc pairwise comparisons were conducted using the ART-C procedure (Elkin et al., 2021). Listwise deletion was applied to manage any missing values. Bonferroni’s adjustment was used for multiple comparisons, as appropriate. To further assess the potential benefit of the treatment, the McNemar test was applied to compare the number of participants who, following the treatment, achieved an SS within a higher score range. This non-parametric test is well suited for paired nominal data (McNemar, 1947). Performance improvement was operationalized as a binary (dummy) variable, where 0 represented no improvement, and 1 represented an improvement. Effect sizes were estimated using R-squared statistics for LMMs, Cohen’s d for post hoc analyses, and psi (ψ) for the McNemar test (Cohen, 2013; Edwards et al., 2008; Lehr, 2001). The nominal α level was 0.05. Aligned and ranked responses were generated using ARTool.exe version 2.2.2. LMMS were run using the GAMLj suite for Jamovi. Overall, statistical analyses were conducted using Jamovi v. 2.4.11.

3. Results

3.1. Descriptive Statistics

Fifty children were enrolled. According to the exclusion criteria, the final cohort consisted of 28 participants. The children were randomly assigned to the experimental or control group, each comprising 14 participants. Descriptive statistics for demographic, clinical, and baseline neuropsychological characteristics of the two groups are reported in Table 1.
All participants were males, aged 4 to 13 years, with education levels ranging from 4 to 10 years, starting from kindergarten, according to the Italian schooling system. The ADOS-2 scores revealed a range of different functioning skills across the two groups, reflecting varying degrees of ASD severity: overall, six children demonstrated high functionality, nine moderate functionality, and 13 low functionality. The two groups were matched for age (t = −0.153, df = 26, p = 0.88), education level (t = −0.233, df = 26, p = 0.82), and ASD symptom severity (χ2 = 0.188, df = 2, p = 0.91). To ensure the interpretability of any findings, we verified that the experimental and control groups were comparable at the pre-test stage. The effectiveness of the randomization procedure was confirmed, as the two groups showed no significant difference in any administered NEPSY-II test scores (all p > 0.05).

3.2. Results from ART LMM Analysis

Here is a summary of the results for each NEPSY-II test, highlighting any main effects (group and/or time) and interactions (group × time). Table 2 shows the raw scores achieved by each participant included in this study. The results of ART LMM analyses are graphically represented in Figure 2.
L1-Comprehension of Instructions. Both a main effect of time (R2 = 0.63, F1,26 = 4.867, p = 0.036) and a group × time interaction emerged (R2 = 0.72, F1,26 = 6.912, p = 0.014), while no main effect of group was observed (F1,26 = 1.006, p = 0.325). As for the main effect of time, post hoc analyses revealed a significant improvement in performance from pre- to post-treatment (t = 2.21, df = 26, pBonferroni = 0.036, d = 0.87). This improvement is clearly attributable to the experimental group, although results from post hoc contrasts highlighted only a trend towards significance (experimental group, pre-test vs. post-test: t = −2.604, df = 25, pBonferroni = 0.09, d = 0.87; control group, pre-test vs. post-test: t = −0.035, df = 25, pBonferroni > 0.99).
M4-List Memory. Both the main effect of group (F1,21 = 4.313, p = 0.05) and time (F1,21 = 3.932, p = 0.06) were found to border the nominal statistical level, with better overall performance in the experimental group (t = 2.08, df = 21, pBonferroni = 0.05) and at the end of treatment (t = 1.98, df = 21, pBonferroni = 0.06), respectively. No significant group × time interaction effect was detected (F1,21 = 1.77, p = 0.20).
M4-List Memory Delayed. A statistically significant main effect of the group was highlighted (R2 = 0.40, F1,21 = 9.781, p = 0.005). Specifically, the experimental group outperformed the control group in this task (t = 3.13, df = 21, pBonferroni = 0.005, d = 1.37). Also, a main effect of time (F1,21 = 3.845, p = 0.06) and a group × time interaction effect (F1,21 = 2.99, p = 0.09) were found to approach statistical significance. A trend toward improvement at post-test was found in the experimental group compared with the control group (t = 2.771, df = 20, pBonferroni = 0.07).
M6-Narrative Memory-Free Recall. No main effect of group (F1,25 = 0.780, p = 0.39) or time (F1,25 = 2.949, p = 0.10) nor any time interaction effect (F1,25 = 0.913, p = 0.35) emerged from the model.
M6-Narrative Memory-Cued Recall. The analysis did not reveal significant main effects for the group (F1,25 = 1.49, p = 0.23) or time (F1,25 = 0.187, p = 0.67) nor a significant group × time interaction (F1,25 = 0.696, p = 0.41).

3.3. Proportion of Improvement in NEPSY-II

Given the challenges of fitting a semi-parametric approach with a limited sample size, McNemar’s test was used to compare the number of individuals in each group who improved their performance based on SS ranges. The results of McNemar’s test are depicted in Figure 3. As for L1-C, a significant treatment effect was observed, with 7/14 experimental participants improving at the post-test, while none improved in the control group (χ2 = 7.00, df = 1, p = 0.008, ψ = ∞). While M4-LM SS scores showed equal improvement across groups, with two participants improving in each group (χ2 = 5.33, df = 1, p = 0.021, ψ = 1), a moderate effect in favor of the experimental group was observed in M4-LMD (χ2 = 7.36, df = 1, p = 0.007, ψ = 2). In particular, two participants included in the experimental group improved at post-test compared with only one in the control group. Similarly, while no association between group and improvement was observed for M6-FR (χ2 = 3.00, df = 1, p = 0.08), five participants in the experimental group improved in M6-CR SS scores compared with only one in the control group (χ2 = 5.44, df = 1, p = 0.02, ψ = 5).
Given the small sample size, only a few participants are represented at each ASD severity level based on the ADOS-2 score. This makes it challenging to evaluate intervention efficacy based on aggregated data. However, analyzing individual-level outcomes may provide insight, especially in tasks where the proportion of experimental participants showing improvement was clinically meaningful. For instance, in L1-C, the proportion of children showing improvement was 33.3% (2/6) in the low-functioning group, 60% (3/5) in the moderate-functioning group, and 66.7% (2/3) in the high-functioning group (χ2 = 1.20, df = 2, p = 0.55). In M6-CR, improvement rates were 50% (3/6), 20% (1/5), and 50% (1/2), respectively (χ2 = 1.10, df = 2, p = 0.56). These preliminary findings suggest that the TUI-based intervention may effectively target language and memory deficits in ASD, regardless of the child’s level of functioning.

4. Discussion

This study aimed to investigate whether the use of a multisensory integration (MSI), combined with Tangible User Interfaces (TUIs) during a digital storytelling task, could improve comprehension and memory skills in a group of children with autism spectrum disorder (ASD). To test this hypothesis, the performance of a group of children using TUIs (experimental group) was compared to that of children who followed the story through corresponding images in a book (control group). Although the results are preliminary due to the small sample size, our findings appear promising.
At the end of the intervention, the experimental group demonstrated a trend of improvement in tasks assessing instruction comprehension and narrative memory (cued recall), with a significantly greater proportion of children benefiting compared with the control group. Interestingly, the benefits of TUIs appeared to be evenly distributed across low-, moderate-, and high-functioning children, which suggests that their effectiveness is not contingent on the baseline severity of ASD symptoms. This further supports the potential of TUIs as a scalable and adaptable tool for intervention, capable of addressing core deficits in ASD across a wide range of functional profiles. According to the previous literature, these findings further highlight the potential of MSI for cognitive enhancement in neurodevelopmental disorders, including ASD (Dionne-Dostie et al., 2015). Consistent with the “redundant-target” paradigm (Brang et al., 2022; Cappe et al., 2009), individuals respond faster to multisensory stimuli (e.g., sound and light combined) than to unisensory ones due to the facilitatory effects of MSI (Stein & Stanford, 2008).
Our results are strictly behavioral and do not allow conclusions about the neural substrates involved in the proposed treatment. However, one might hypothesize that regions responsible for early sensory processing and integrating auditory and visual information may have played a key role. Indeed, the primary task required audio-visual integration—listening to the story while viewing images (either on a screen or in a book) (Ross et al., 2022). Accordingly, several studies have shown that audio-visual integration amplifies activity in critical brain regions, such as the primary auditory cortex (Callan et al., 2003; Calvert, 2001), visual motion and prefrontal regions (Puce et al., 2003; Wright et al., 2003), including Broca’s area, and the premotor cortex (Iacoboni, 2008; Meister et al., 2007; Ojanen et al., 2005). Thus, it is plausible that these areas were similarly engaged during the intervention.
Interestingly, while children in both the experimental and control groups showed similar improvements on the free recall memory task, a significantly higher number of children in the experimental group improved on the cued narrative memory task. On the one hand, these findings support the effectiveness of both the experimental and control interventions in improving memory skills. On the other hand, the observed dissociation aligns with previous research (Boucher et al., 2012; Bowler et al., 2004; D. L. Williams et al., 2006) showing that while children were unable to recall the story in full, they could still answer specific questions about it. This suggests that the mnemonic difficulties observed in ASD children do not stem from an inability to encode the information but rather from retrieving it when no facilitation is provided. In other words, the issue may not pertain to the encoding of the memory trace but rather to the effective deployment of retrieval (executive) strategies. From an anatomic-functional standpoint, it has been observed that some areas involved in mnemonic encoding, such as the primary visual and auditory cortices, remain functionally intact in children with ASD, but access to stored information may depend on associative circuits that are less efficient (Yue, 2023).
An innovative aspect of this study was the inclusion of olfactory stimulation alongside tactile, visual, and auditory inputs. Olfaction is unique among sensory modalities as it bypasses the thalamus and directly accesses limbic structures such as the hippocampus and the entorhinal cortex, which are critical for memory and spatial navigation (Zhou et al., 2021). Notably, the entorhinal cortex contains place cells, neurons that play a central role in encoding spatial and contextual information (Salimi et al., 2021). This direct link between olfactory processing and the hippocampal memory system suggests that in children with ASD, who often exhibit deficits in encoding and retrieving episodic memories, olfactory inputs might have supported the encoding of the narrative’s structure, aiding retrieval during the cued recall task (Boucher et al., 2012; Bowler et al., 2004; D. L. Williams et al., 2006). This hypothesis aligns with findings on neurotypical individuals, where odors were found to evoke vivid and emotionally rich memories (Herz, 2016). Indeed, although data on olfactory function in ASD are still controversial (Boudjarane et al., 2017; Dalton et al., 2000; Rozenkrantz et al., 2015; Wicker et al., 2016), it has been shown that relative to typically developing controls, individuals with ASD demonstrated increased functional connectivity of the olfactory network and decreased functional connectivity within the multisensory integration network (Boccia et al., 2016; Scheliga et al., 2023). As the thalamus acts as a relay center for all sensory modalities except olfaction and plays a key role in facilitating MSI during combined stimulation (Froesel et al., 2021; Tyll et al., 2011), we might hypothesize that, in our study, olfactory stimulation may have facilitated the creation of stronger intermodal connections. As a result, this may have enhanced the benefits of multisensory stimulation (Martin & Daniel, 2014; Russo et al., 2010; Stevenson et al., 2018). A recent review (Barros & Soares, 2020) has argued in favor of studying olfactory stimulation, as well as visuo-olfactory integration, given the potential of these approaches to drive effective interventions in ASD.
Beyond clinical perspectives, the potential of TUIs within educational settings should not be overlooked (Di Fuccio, 2022; Di Fuccio & Mastroberti, 2018). Italy boasts a long-standing tradition of inclusive education with decades of legislation aimed at integrating students with disabilities into mainstream schools. Since the 1970s, this framework has led to the elimination of segregated classrooms, the introduction of specialist support teachers, and the implementation of a collaborative model that connects schools, families, and local support services. This ensured that students received coordinated assistance across different settings, benefiting from a holistic approach to education. It is important to stress that not all European countries have embraced a fully inclusive approach to education. In Germany and the Netherlands, for instance, students with disabilities are primarily registered in specialized schools. This policy contrasts with the documented benefits of full inclusion, which has been associated with reduced social isolation, improved interpersonal skills, and enhanced self-esteem and self-efficacy among students with emotional and/or behavioral disorders (The Peacock Hill Working Group et al., 1991).
Although Italy has positioned itself at the forefront of inclusive education policies, conventional educational strategies often struggle to fully engage students with ASD. Standard teaching models may not fully address the diverse sensory and cognitive needs within this population, underscoring the need for innovative tools that support individualized, engaging, and accessible learning experiences within inclusive settings. In this vein, TUIs may offer a structured yet flexible solution. By leveraging physical interaction and minimizing reliance on abstract, verbal-based instruction, TUIs may support an alternative, cutting-edge learning paradigm. For instance, their ability to facilitate engagement through hands-on, multisensory learning experiences, combined with their capacity to foster social interaction—whether with the examiner or within peer groups in cooperative activities—makes them particularly well-suited for inclusive educational environments designed to support individuals with ASD.
The present study is not without limitations. One key constraint is the limited sample size, which restricts the generalizability of our findings. Also, this study lacks a control group of typically developing children, making it difficult to determine whether the observed benefits of the multisensory intervention are specific to children with ASD or reflect broader trends across the general population. Moreover, as a result of the convenience sampling procedure, our study includes only boys. While autism is about four times more prevalent in males than females (Maenner, 2023), making this focus justified, our sample still does not fully capture the epidemiological distribution of the phenomenon. Further studies with balanced groups are needed to explore sex-related differences in autism presentation and to test the invariance of TUIs’ effectiveness as an intervention paradigm.

5. Conclusions

This study suggests that a TUI-mediate multisensory intervention holds promise for enhancing cognitive abilities in children with ASD. This approach could not only improve comprehension and memory but also pave the way for innovative treatments that leverage brain plasticity and functionally preserved areas. Future research should seek to validate these results with larger samples, including neurophysiological measures and advanced neuroimaging techniques for investigating the neural mechanisms involved.

Author Contributions

Conceptualization, M.D.L. and P.M.; methodology, M.D.L., C.R.I., A.R. and M.G.; software, R.D.F. and F.R.; formal analysis, C.R.I. and P.D.; investigation, M.D.L.; data curation, C.R.I. and P.D.; writing—original draft preparation, M.D.L. and P.M.; writing—review and editing, C.R.I. and P.M. All authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding.

Institutional Review Board Statement

This study was conducted in accordance with the Declaration of Helsinki and approved by the Ethics Committee of the Division for Educational Methodologies and Technologies for Autism of the Neapolisanit Rehabilitation Centre, Naples, Italy (protocol number IEC07/2023, approval date: 1 January 2023).

Informed Consent Statement

Written informed consent has been obtained from the parents of the children.

Data Availability Statement

The experimental data that support the findings of this study are available from the corresponding author upon reasonable request.

Conflicts of Interest

Author Franco Rubinacci was employed by the company Smarted srl, Riviera di Chiaia. The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

References

  1. Al Mahmud, A., & Soysa, A. I. (2020). POMA: A tangible user interface to improve social and cognitive skills of Sri Lankan children with ASD. International Journal of Human-Computer Studies, 144, 102486. [Google Scholar] [CrossRef]
  2. Andrzejewski, J. A., Greenberg, T., & Carlson, J. M. (2019). Neural correlates of aversive anticipation: An activation likelihood estimate meta-analysis across multiple sensory modalities. Cognitive, Affective & Behavioral Neuroscience, 19(6), 1379–1390. [Google Scholar] [CrossRef]
  3. Bara, F., Gentaz, E., & Valente, D. (2018). The effect of tactile illustrations on comprehension of storybooks by three children with visual impairments: An exploratory study. Journal of Visual Impairment & Blindness, 112(6), 759–765. [Google Scholar] [CrossRef]
  4. Barros, F., & Soares, S. C. (2020). Giving meaning to the social world in autism spectrum disorders: Olfaction as a missing piece of the puzzle? Neuroscience and Biobehavioral Reviews, 116, 239–250. [Google Scholar] [CrossRef]
  5. Beccaluva, E., Riccardi, F., Gianotti, M., Barbieri, J., & Garzotto, F. (2022). VIC—A tangible user interface to train memory skills in children with intellectual disability. International Journal of Child-Computer Interaction, 32, 100376. [Google Scholar] [CrossRef]
  6. Boccia, M., Barbetti, S., Piccardi, L., Guariglia, C., Ferlazzo, F., Giannini, A. M., & Zaidel, D. W. (2016). Where does brain neural activation in aesthetic responses to visual art occur? Meta-analytic evidence from neuroimaging studies. Neuroscience and Biobehavioral Reviews, 60, 65–71. [Google Scholar] [CrossRef]
  7. Bottema-Beutel, K., & Crowley, S. (2021). Pervasive undisclosed conflicts of interest in applied behavior analysis autism literature. Frontiers in Psychology, 12, 676303. [Google Scholar] [CrossRef]
  8. Boucher, J., Mayes, A., & Bigham, S. (2012). Memory in autistic spectrum disorder. Psychological Bulletin, 138(3), 458–496. [Google Scholar] [CrossRef]
  9. Boudjarane, M. A., Grandgeorge, M., Marianowski, R., Misery, L., & Lemonnier, É. (2017). Perception of odors and tastes in autism spectrum disorders: A systematic review of assessments. Autism Research: Official Journal of the International Society for Autism Research, 10(6), 1045–1057. [Google Scholar] [CrossRef]
  10. Bowler, D. M., Gardiner, J. M., & Berthollier, N. (2004). Source memory in adolescents and adults with Asperger’s syndrome. Journal of Autism and Developmental Disorders, 34(5), 533–542. [Google Scholar] [CrossRef]
  11. Brang, D., Plass, J., Sherman, A., Stacey, W. C., Wasade, V. S., Grabowecky, M., Ahn, E., Towle, V. L., Tao, J. X., Wu, S., Issa, N. P., & Suzuki, S. (2022). Visual cortex responds to sound onset and offset during passive listening. Journal of Neurophysiology, 127(6), 1547–1563. [Google Scholar] [CrossRef] [PubMed]
  12. Brown, H. M., Oram-Cardy, J., & Johnson, A. (2013). A meta-analysis of the reading comprehension skills of individuals on the autism spectrum. Journal of Autism and Developmental Disorders, 43(4), 932–955. [Google Scholar] [CrossRef] [PubMed]
  13. Callan, D. E., Jones, J. A., Munhall, K., Callan, A. M., Kroos, C., & Vatikiotis-Bateson, E. (2003). Neural processes underlying perceptual enhancement by visual speech gestures. Neuroreport, 14(17), 2213–2218. [Google Scholar] [CrossRef] [PubMed]
  14. Calvert, G. A. (2001). Crossmodal processing in the human brain: Insights from functional neuroimaging studies. Cerebral Cortex, 11(12), 1110–1123. [Google Scholar] [CrossRef]
  15. Cappe, C., Morel, A., Barone, P., & Rouiller, E. M. (2009). The thalamocortical projection systems in primate: An anatomical support for multisensory and sensorimotor interplay. Cerebral Cortex, 19(9), 2025–2037. [Google Scholar] [CrossRef]
  16. Cardillo, R., & Mammarella, I. C. (2015). L’utilità della NEPSY-II per la valutazione neuropsicologica nella psicopatologia dello sviluppo. In Psicologia clinica dello sviluppo (pp. 553–560). Il Mulino. [Google Scholar] [CrossRef]
  17. Cheung, M., Chan, A. S., Sze, S. L., Leung, W. W., & To, C. Y. (2010). Verbal memory deficits in relation to organization strategy in high- and low-functioning autistic children. Research in Autism Spectrum Disorders, 4(4), 764–771. [Google Scholar] [CrossRef]
  18. Cnaan, A., Laird, N. M., & Slasor, P. (1997). Using the general linear mixed model to analyse unbalanced repeated measures and longitudinal data. Statistics in Medicine, 16(20), 2349–2380. [Google Scholar] [CrossRef]
  19. Cohen, J. (2013). Statistical power analysis for the behavioral sciences (2nd ed.). Routledge. [Google Scholar] [CrossRef]
  20. Dalton, P., Doolittle, N., Nagata, H., & Breslin, P. A. (2000). The merging of the senses: Integration of subthreshold taste and smell. Nature Neuroscience, 3(5), 431–432. [Google Scholar] [CrossRef]
  21. Dautenhahn, K., & Werry, I. (2004). Towards interactive robots in autism therapy: Background, motivation and challenges. Pragmatics & Cognition, 12(1), 1–35. [Google Scholar] [CrossRef]
  22. Di Fuccio, R. (2022). I sensi nel digitale. Le tangible user interfaces innovano la pratica pedagogica. Progedit–Progetti Editoriali srl. [Google Scholar]
  23. Di Fuccio, R., & Mastroberti, S. (2018). Tangible user interfaces for multisensory storytelling at school: A study of acceptability. QWERTY-Interdisciplinary Journal of Technology, Culture and Education, 13(1), 20–32. [Google Scholar]
  24. Di Fuccio, R., Ponticorvo, M., Ferrara, F., & Miglino, O. (2016). Digital and multisensory storytelling: Narration with smell, taste and touch. Springer International Publishing. [Google Scholar] [CrossRef]
  25. Dionne-Dostie, E., Paquette, N., Lassonde, M., & Gallagher, A. (2015). Multisensory integration and child neurodevelopment. Brain Sciences, 5(1), 32–57. [Google Scholar] [CrossRef] [PubMed]
  26. Edwards, L. J., Muller, K. E., Wolfinger, R. D., Qaqish, B. F., & Schabenberger, O. (2008). An R2 statistic for fixed effects in the linear mixed model. Statistics in Medicine, 27(29), 6137–6157. [Google Scholar] [CrossRef] [PubMed]
  27. Elkin, L. A., Kay, M., Higgins, J. J., & Wobbrock, J. O. (2021, October 10–14). An aligned rank transform procedure for multifactor contrast tests. The 34th Annual ACM Symposium on User Interface Software and Technology (pp. 754–768), Virtual. [Google Scholar] [CrossRef]
  28. Escobedo, L., Ibarra, C., Hernandez, J., Alvelais, M., & Tentori, M. (2014). Smart objects to support the discrimination training of children with autism. Personal and Ubiquitous Computing, 18(6), 1485–1497. [Google Scholar] [CrossRef]
  29. Fails, J. A., Druin, A., Guha, M. L., Chipman, G., Simms, S., & Churaman, W. (2005). Child’s play: A comparison of desktop and physical interactive environments (pp. 48–55). Association for Computing Machinery. [Google Scholar] [CrossRef]
  30. Fani-Panagiota, R. (2015). Teaching strategies for children with autism. Journal of Physical Education and Sport, 15(1), 148. [Google Scholar]
  31. Faras, H., Al Ateeqi, N., & Tidmarsh, L. (2010). Autism spectrum disorders. Annals of Saudi Medicine, 30(4), 295–300. [Google Scholar] [CrossRef]
  32. Farr, W., Yuill, N., & Raffle, H. (2010). Social benefits of a tangible user interface for children with Autistic Spectrum Conditions. Autism, 14(3), 237–252. [Google Scholar] [CrossRef]
  33. Froesel, M., Cappe, C., & Ben Hamed, S. (2021). A multisensory perspective onto primate pulvinar functions. Neuroscience and Biobehavioral Reviews, 125, 231–243. [Google Scholar] [CrossRef]
  34. Garcia-Ruiz, M. A., Kapralos, B., & Rebolledo-Mendez, G. (2021). An overview of olfactory displays in education and training. Multimodal Technologies and Interaction, 5(10), 64. [Google Scholar] [CrossRef]
  35. Gotham, K., Risi, S., Pickles, A., & Lord, C. (2007). The autism diagnostic observation schedule: REVISED algorithms for improved diagnostic validity. Journal of Autism and Developmental Disorders, 37(4), 613–627. [Google Scholar] [CrossRef]
  36. Herz, R. S. (2016). The role of odor-evoked memory in psychological and physiological health. Brain Sciences, 6(3), 22. [Google Scholar] [CrossRef]
  37. Hirota, T., & King, B. H. (2023). Autism spectrum disorder: A review. JAMA, 329(2), 157–168. [Google Scholar] [CrossRef] [PubMed]
  38. Iacoboni, M. (2008). The role of premotor cortex in speech perception: Evidence from fMRI and rTMS. Journal of Physiology, Paris, 102(1–3), 31–34. [Google Scholar] [CrossRef]
  39. Ilardi, C. R., Federico, G., La Marra, M., Amato, R., Iavarone, A., Soricelli, A., Santangelo, G., & Chieffi, S. (2025). Deficits in reaching movements under visual interference as a novel diagnostic marker for mild cognitive impairment. Scientific Reports, 15(1), 1901. [Google Scholar] [CrossRef] [PubMed]
  40. Ishii, H., & Ullmer, B. (1997). Tangible bits: Towards seamless interfaces between people, bits and atoms. In Proceedings of the ACM SIGCHI conference on human factors in computing systems (pp. 234–241). Association for Computing Machinery. [Google Scholar] [CrossRef]
  41. Kaplan, H., Clopton, M., Kaplan, M., Messbauer, L., & McPherson, K. (2006). Snoezelen multi-sensory environments: Task engagement and generalization. Research in Developmental Disabilities, 27(4), 443–455. [Google Scholar] [CrossRef]
  42. Korkman, M., Kirk, U., & Kemp, S. (2012). NEPSY—Second edition [dataset]. APA PsycTests. [Google Scholar] [CrossRef]
  43. Kumazaki, H., Muramatsu, T., Fujisawa, T. X., Miyao, M., Matsuura, E., Okada, K., Kosaka, H., Tomoda, A., & Mimura, M. (2016). Assessment of olfactory detection thresholds in children with autism spectrum disorders using a pulse ejection system. Molecular Autism, 7(1), 6. [Google Scholar] [CrossRef]
  44. Landa, R. J., & Goldberg, M. C. (2005). Language, social, and executive functions in high functioning autism: A continuum of performance. Journal of Autism and Developmental Disorders, 35(5), 557–573. [Google Scholar] [CrossRef]
  45. Leaf, J. B., Cihon, J. H., Leaf, R., McEachin, J., Liu, N., Russell, N., Unumb, L., Shapiro, S., & Khosrowshahi, D. (2022). Concerns about ABA-based intervention: An evaluation and recommendations. Journal of Autism and Developmental Disorders, 52(6), 2838–2853. [Google Scholar] [CrossRef]
  46. Lehr, R. G. (2001). Some practical considerations and a crude formula for estimating sample size for McNemar’s test. Drug Information Journal: DIJ/Drug Information Association, 35(4), 1227–1233. [Google Scholar] [CrossRef]
  47. Leonard, A., Mitchell, P., & Parsons, S. (2002, September 18–20). Finding a place to sit: A preliminary investigation into the effectiveness of virtual environments for social skills training for people with autistic spectrum disorders. 4th International Conference on Disability, Virtual Reality & Associated Technologies, Veszprém, Hungary. [Google Scholar]
  48. Lindgren, K. A., Folstein, S. E., Tomblin, J. B., & Tager-Flusberg, H. (2009). Language and reading abilities of children with autism spectrum disorders and specific language impairment and their first-degree relatives. Autism Research: Official Journal of the International Society for Autism Research, 2(1), 22–38. [Google Scholar] [CrossRef]
  49. Lord, C., Elsabbagh, M., Baird, G., & Veenstra-Vanderweele, J. (2018). Autism spectrum disorder. The Lancet, 392(10146), 508–520. [Google Scholar] [CrossRef]
  50. Lord, C., Risi, S., Lambrecht, L., Cook, E. H., Leventhal, B. L., DiLavore, P. C., Pickles, A., & Rutter, M. (2000). The autism diagnostic observation schedule-generic: A standard measure of social and communication deficits associated with the spectrum of autism. Journal of Autism and Developmental Disorders, 30(3), 205–223. [Google Scholar] [CrossRef]
  51. Lord, C., Rutter, M., DiLavore, P., Risi, S., Gotham, K., Bishop, S. L., Lord, C., Luyster, R. J., Gotham, K., & Guthrie, W. (2012). Autism diagnostic observation schedule, second edition (ADOS-2) (2nd ed.). Pearson. [Google Scholar]
  52. Maenner, M. J. (2023). Prevalence and characteristics of autism spectrum disorder among children aged 8 years—Autism and developmental disabilities monitoring network, 11 sites, United States, 2020. MMWR Surveillance Summaries, 72, 1–14. [Google Scholar] [CrossRef] [PubMed]
  53. Martin, G. N., & Daniel, N. (2014). Autism spectrum disorders and chemoreception: Dead-end or fruitful avenue of inquiry? Frontiers in Psychology, 5, 42. [Google Scholar] [CrossRef]
  54. McCrimmon, A., & Rostad, K. (2014). Test review: Autism diagnostic observation schedule, second edition (ADOS-2) manual (Part II): Toddler module. Journal of Psychoeducational Assessment, 32(1), 88–92. [Google Scholar] [CrossRef]
  55. McDougal, E., Riby, D. M., & Hanley, M. (2020). Profiles of academic achievement and attention in children with and without autism spectrum disorder. Research in Developmental Disabilities, 106, 103749. [Google Scholar] [CrossRef] [PubMed]
  56. McNemar, Q. (1947). Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika, 12(2), 153–157. [Google Scholar] [CrossRef]
  57. Meister, I. G., Wilson, S. M., Deblieck, C., Wu, A. D., & Iacoboni, M. (2007). The essential role of premotor cortex in speech perception. Current Biology: CB, 17(19), 1692–1696. [Google Scholar] [CrossRef] [PubMed]
  58. Miglino, O., Di Ferdinando, A., Massimiliano, C., Angelo, R., & Carlo, R. (2013). STELT (Smart Technologies to Enhance Learning and Teaching): Una piattaforma per realizzare ambienti di realtà aumentata per apprendere, insegnare e giocare. In Sistemi intelligenti (pp. 397–404). Il Mulino. [Google Scholar]
  59. Muratori, F., Tonacci, A., Billeci, L., Catalucci, T., Igliozzi, R., Calderoni, S., & Narzisi, A. (2017). Olfactory processing in male children with autism: Atypical odor threshold and identification. Journal of Autism and Developmental Disorders, 47(10), 3243–3251. [Google Scholar] [CrossRef]
  60. Ojanen, V., Möttönen, R., Pekkola, J., Jääskeläinen, I. P., Joensuu, R., Autti, T., & Sams, M. (2005). Processing of audiovisual speech in Broca’s area. NeuroImage, 25(2), 333–338. [Google Scholar] [CrossRef]
  61. Parés, N., Carreras, A., Durany, J., Ferrer, J., Freixa, P., Gómez, D., Kruglanski, O., Parés, R., Ribas, J. I., Soler, M., & Sanjurjo, À. (2005, June 8–10). Promotion of creative activity in children with severe autism through visuals in an interactive multisensory environment. Proceedings of the 2005 Conference on Interaction Design and Children (pp. 110–116), Boulder, CO, USA. [Google Scholar] [CrossRef]
  62. Parisi, A., Bellinzona, F., Di Lernia, D., Repetto, C., De Gaspari, S., Brizzi, G., Riva, G., & Tuena, C. (2022). Efficacy of Multisensory Technology in Post-Stroke Cognitive Rehabilitation: A Systematic Review. Journal of Clinical Medicine, 11(21), 6324. [Google Scholar] [CrossRef]
  63. Ponticorvo, M., Di Fuccio, R., Ferrara, F., Rega, A., & Miglino, O. (2019). Multisensory Educational Materials: Five Senses to Learn. In T. Di Mascio, P. Vittorini, R. Gennari, F. De la Prieta, S. Rodríguez, M. Temperini, R. Azambuja Silveira, E. Popescu, & L. Lancia (Eds.), Methodologies and intelligent systems for technology enhanced learning, 8th international conference (pp. 45–52). Springer International Publishing. [Google Scholar] [CrossRef]
  64. Puce, A., Syngeniotis, A., Thompson, J. C., Abbott, D. F., Wheaton, K. J., & Castiello, U. (2003). The human temporal lobe integrates facial form and motion: Evidence from fMRI and ERP studies. NeuroImage, 19(3), 861–869. [Google Scholar] [CrossRef] [PubMed]
  65. Ramos-Aguiar, L. R., & Álvarez-Rodríguez, F. J. (2021). Teaching emotions in children with autism spectrum disorder through a computer program with tangible interfaces. IEEE Revista Iberoamericana de Tecnologías Del Aprendizaje, 16(4), 365–371. [Google Scholar] [CrossRef]
  66. Ross, L. A., Molholm, S., Butler, J. S., Bene, V. A. D., & Foxe, J. J. (2022). Neural correlates of multisensory enhancement in audiovisual narrative speech perception: A fMRI investigation. NeuroImage, 263, 119598. [Google Scholar] [CrossRef]
  67. Rossiter, M., & Garcia, P. A. (2010). Digital storytelling: A new player on the narrative field. New Directions for Adult and Continuing Education, 126, 37–48. [Google Scholar] [CrossRef]
  68. Rozenkrantz, L., Zachor, D., Heller, I., Plotkin, A., Weissbrod, A., Snitz, K., Secundo, L., & Sobel, N. (2015). A mechanistic link between olfaction and autism spectrum disorder. Current Biology: CB, 25(14), 1904–1910. [Google Scholar] [CrossRef] [PubMed]
  69. Russo, N., Foxe, J. J., Brandwein, A. B., Altschuler, T., Gomes, H., & Molholm, S. (2010). Multisensory processing in children with autism: High-density electrical mapping of auditory-somatosensory integration. Autism Research: Official Journal of the International Society for Autism Research, 3(5), 253–267. [Google Scholar] [CrossRef]
  70. Salimi, M., Tabasi, F., Nazari, M., Ghazvineh, S., Salimi, A., Jamaati, H., & Raoufy, M. R. (2021). The olfactory bulb modulates entorhinal cortex oscillations during spatial working memory. The Journal of Physiological Sciences: JPS, 71(1), 21. [Google Scholar] [CrossRef] [PubMed]
  71. Scheliga, S., Kellermann, T., Lampert, A., Rolke, R., Spehr, M., & Habel, U. (2023). Neural correlates of multisensory integration in the human brain: An ALE meta-analysis. Reviews in the Neurosciences, 34(2), 223–245. [Google Scholar] [CrossRef]
  72. Shumway, S., Farmer, C., Thurm, A., Joseph, L., Black, D., & Golden, C. (2012). The ADOS calibrated severity score: Relationship to phenotypic variables and stability over time. Autism Research: Official Journal of the International Society for Autism Research, 5(4), 267–276. [Google Scholar] [CrossRef]
  73. Silver, M., & Oakes, P. (2001). Evaluation of a new computer intervention to teach people with autism or Asperger syndrome to recognize and predict emotions in others. Autism: The International Journal of Research and Practice, 5(3), 299–316. [Google Scholar] [CrossRef]
  74. Sitdhisanguan, K., Chotikakamthorn, N., Dechaboon, A., & Out, P. (2012). Using tangible user interfaces in computer-based training systems for low-functioning autistic children. Personal and Ubiquitous Computing, 16(2), 143–155. [Google Scholar] [CrossRef]
  75. Sorokowska, A., Sorokowski, P., Karwowski, M., Larsson, M., & Hummel, T. (2019). Olfactory perception and blindness: A systematic review and meta-analysis. Psychological Research, 83(8), 1595–1611. [Google Scholar] [CrossRef] [PubMed]
  76. Stein, B. E., & Stanford, T. R. (2008). Multisensory integration: Current issues from the perspective of the single neuron. Nature Reviews. Neuroscience, 9(4), 255–266. [Google Scholar] [CrossRef]
  77. Stevenson, R. A., Segers, M., Ncube, B. L., Black, K. R., Bebko, J. M., Ferber, S., & Barense, M. D. (2018). The cascading influence of multisensory processing on speech perception in autism. Autism: The International Journal of Research and Practice, 22(5), 609–624. [Google Scholar] [CrossRef] [PubMed]
  78. Sweigert, J. R., St John, T., Begay, K. K., Davis, G. E., Munson, J., Shankland, E., Estes, A., Dager, S. R., & Kleinhans, N. M. (2020). Characterizing olfactory function in children with autism spectrum disorder and children with sensory processing dysfunction. Brain Sciences, 10(6), 362. [Google Scholar] [CrossRef]
  79. Tager-Flusberg, H., & Joseph, R. M. (2003). Identifying neurocognitive phenotypes in autism. Philosophical Transactions of the Royal Society of London. Series B, Biological Sciences, 358(1430), 303–314. [Google Scholar] [CrossRef]
  80. Tárraga-Mínguez, R., Gómez-Marí, I., & Sanz-Cervera, P. (2020). Interventions for improving reading comprehension in children with ASD: A systematic review. Behavioral Sciences (Basel, Switzerland), 11(1), 3. [Google Scholar] [CrossRef]
  81. The Peacock Hill Working Group, Zabel, R. K., Kauffman, J. M., Lloyd, J. W., Cook, L., Cullinan, D., Epstein, M. H., Hallahan, D. P., Nelson, C. M., Polsgrove, L., Sabornie, E. J., Strain, P. S., & Walker, H. M. (1991). Problems and Promises in Special Education and Related Services for Children and Youth with Emotional or Behavioral Disorders. Behavioral Disorders, 16(4), 299–313. [Google Scholar] [CrossRef]
  82. Tyll, S., Budinger, E., & Noesselt, T. (2011). Thalamic influences on multisensory integration. Communicative & Integrative Biology, 4(4), 378–381. [Google Scholar] [CrossRef]
  83. Van der Burg, E., Olivers, C. N. L., Bronkhorst, A. W., & Theeuwes, J. (2008). Pip and pop: Nonspatial auditory signals improve spatial visual search. Journal of Experimental Psychology. Human Perception and Performance, 34(5), 1053–1065. [Google Scholar] [CrossRef]
  84. Villafuerte, L., Markova, M., & Jorda, S. (2012, May 5–10). Acquisition of social abilities through musical tangible user interface: Children with autism spectrum condition and the reactable. CHI ’12 Extended Abstracts on Human Factors in Computing Systems (pp. 745–760), Austin, TX, USA. [Google Scholar] [CrossRef]
  85. Vroomen, J., & de Gelder, B. (2000). Sound enhances visual perception: Cross-modal effects of auditory organization on vision. Journal of Experimental Psychology. Human Perception and Performance, 26(5), 1583–1590. [Google Scholar] [CrossRef] [PubMed]
  86. Wicker, B., Monfardini, E., & Royet, J.-P. (2016). Olfactory processing in adults with autism spectrum disorders. Molecular Autism, 7, 4. [Google Scholar] [CrossRef] [PubMed]
  87. Williams, C., Wright, B., Callaghan, G., & Coughlan, B. (2002). Do children with autism learn to read more readily by computer assisted instruction or traditional book methods? A pilot study. Autism: The International Journal of Research and Practice, 6(1), 71–91. [Google Scholar] [CrossRef] [PubMed]
  88. Williams, D. L., Goldstein, G., & Minshew, N. J. (2006). The profile of memory function in children with autism. Neuropsychology, 20(1), 21–29. [Google Scholar] [CrossRef]
  89. Wobbrock, J. O., Findlater, L., Gergle, D., & Higgins, J. J. (2011, May 7–12). The aligned rank transform for nonparametric factorial analyses using only anova procedures. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 143–146), Vancouver, BC, Canada. [Google Scholar] [CrossRef]
  90. Wright, T. M., Pelphrey, K. A., Allison, T., McKeown, M. J., & McCarthy, G. (2003). Polysensory interactions along lateral temporal regions evoked by audiovisual speech. Cerebral Cortex, 13(10), 1034–1043. [Google Scholar] [CrossRef]
  91. Wynn, J. W., & Smith, T. (2003). Generalization Between Receptive and Expressive Language in Young Children with Autism. Behavioral Interventions, 18(4), 245–266. [Google Scholar] [CrossRef]
  92. Yu, Q., Li, E., Li, L., & Liang, W. (2020). Efficacy of Interventions Based on Applied Behavior Analysis for Autism Spectrum Disorder: A Meta-Analysis. Psychiatry Investigation, 17(5), 432–443. [Google Scholar] [CrossRef]
  93. Yue, Q. (2023). Maintaining auditory working memory representations beyond sensory cortices makes working memory work. The Journal of Neuroscience: The Official Journal of the Society for Neuroscience, 43(40), 6714–6716. [Google Scholar] [CrossRef]
  94. Zhou, G., Olofsson, J. K., Koubeissi, M. Z., Menelaou, G., Rosenow, J., Schuele, S. U., Xu, P., Voss, J. L., Lane, G., & Zelano, C. (2021). Human hippocampal connectivity is stronger in olfaction than other sensory systems. Progress in Neurobiology, 201, 102027. [Google Scholar] [CrossRef]
Figure 1. Tangible User Interface (TUI) apparatus and multisensory stimuli for digital storytelling. The TUI prototype consisted of (a) a system with a laptop running Windows OS to play the video and (b) a rigid plastic board, called the “Magic board”, connected via USB cable. The Magic board contained an active Near Field Communication (NFC) antenna. Each object used in this study was equipped with a tag embedded with a passive NFC antenna. When an object interacted with the Magic board, the system identified it. If the selected object was correct, the story progressed; otherwise, the story paused, and the computer emitted a warning sound. Among the interactive objects was “Pippo”, the story’s protagonist, represented as a 3D-printed figurine (c). During the task, the selection of objects was guided by their multisensory properties (d). For touch, these included a small hand rake, soil, a watering can, a strawberry plant, and Pippo. The smell was engaged through containers emitting scents such as pine, rose, strawberry, and damp earth. Finally, the taste was stimulated with a strawberry-flavored candy.
Figure 1. Tangible User Interface (TUI) apparatus and multisensory stimuli for digital storytelling. The TUI prototype consisted of (a) a system with a laptop running Windows OS to play the video and (b) a rigid plastic board, called the “Magic board”, connected via USB cable. The Magic board contained an active Near Field Communication (NFC) antenna. Each object used in this study was equipped with a tag embedded with a passive NFC antenna. When an object interacted with the Magic board, the system identified it. If the selected object was correct, the story progressed; otherwise, the story paused, and the computer emitted a warning sound. Among the interactive objects was “Pippo”, the story’s protagonist, represented as a 3D-printed figurine (c). During the task, the selection of objects was guided by their multisensory properties (d). For touch, these included a small hand rake, soil, a watering can, a strawberry plant, and Pippo. The smell was engaged through containers emitting scents such as pine, rose, strawberry, and damp earth. Finally, the taste was stimulated with a strawberry-flavored candy.
Behavsci 15 00267 g001
Figure 2. Box-and-whisker plots illustrating the performance (raw scores) at pre- and post-test of the experimental and control groups across the five NEPSY-II subtests. Tukey’s method was used to plot whiskers and outliers. The ends of the boxes represent the first (Q1) and third (Q3) quartiles, with the interquartile range (IQR) defined as the difference between Q3 and Q1. Values greater than Q3 plus 1.5 times the IQR or less than Q1 minus 1.5 times the IQR are plotted as individual points. The median is displayed as a horizontal line within the box, while the arithmetical mean is marked with a “+” symbol. Note: L1-C: L1-Comprehension of Instructions; M4-LM: M4-List Memory; M4-LMD: M4-List Memory Delayed; M6-FR: M6-Narrative Memory-Free Recall; M6-CR: M6-Narrative Memory-Cued Recall.
Figure 2. Box-and-whisker plots illustrating the performance (raw scores) at pre- and post-test of the experimental and control groups across the five NEPSY-II subtests. Tukey’s method was used to plot whiskers and outliers. The ends of the boxes represent the first (Q1) and third (Q3) quartiles, with the interquartile range (IQR) defined as the difference between Q3 and Q1. Values greater than Q3 plus 1.5 times the IQR or less than Q1 minus 1.5 times the IQR are plotted as individual points. The median is displayed as a horizontal line within the box, while the arithmetical mean is marked with a “+” symbol. Note: L1-C: L1-Comprehension of Instructions; M4-LM: M4-List Memory; M4-LMD: M4-List Memory Delayed; M6-FR: M6-Narrative Memory-Free Recall; M6-CR: M6-Narrative Memory-Cued Recall.
Behavsci 15 00267 g002
Figure 3. Stacked bar charts depicting the proportions of participants who showed improvement in each of the five NEPSI-II subtests following the treatment. Note: L1-C: L1-Comprehension of Instructions; M4-LM: M4-List Memory; M4-LMD: M4-List Memory Delayed; M6-FR: M6-Narrative Memory-Free Recall; M6-CR: M6-Narrative Memory-Cued Recall.
Figure 3. Stacked bar charts depicting the proportions of participants who showed improvement in each of the five NEPSI-II subtests following the treatment. Note: L1-C: L1-Comprehension of Instructions; M4-LM: M4-List Memory; M4-LMD: M4-List Memory Delayed; M6-FR: M6-Narrative Memory-Free Recall; M6-CR: M6-Narrative Memory-Cued Recall.
Behavsci 15 00267 g003
Table 1. Demographic, clinical, and baseline neuropsychological characteristics of the two groups.
Table 1. Demographic, clinical, and baseline neuropsychological characteristics of the two groups.
Experimental Group
(n = 14)
Control Group
(n = 14)
p-Value
Age, years; mean (SD)9.14 (2.80)9.00 (2.07)0.88 a
Education, year; mean (SD)6.21 (2.75)6.00 (2.07)0.82 a
Functional level (ADOS-2)
  Low functioning, CSS range: 6–1067
  Moderate functioning, CSS range: 4–554
  High functioning, CSS range: 1–333
L1-C, raw score; mean (SD)3.29 (3.77)3.43 (4.13)0.92 a
M4-LM, raw score; mean (SD)2.33 (3.45) 0.82 (1.47) ††0.19 a
M4-LMD, raw score; mean (SD)2.17 (3.33) 0.91 (1.45) ††0.21 b
M6-FR, raw score; mean (SD)3.23 (3.81) 2.57 (3.39)0.53 a
M6-CR, raw score; mean (SD)3.46 (3.93) 2.36 (2.90)0.35 a
Note: CSS: Calibrated Severity Scores; L1-C: L1-Comprehension of Instructions; M4-LM: M4-List Memory; M4-LMD: M4-List Memory Delayed; M6-FR: M6-Narrative Memory-Free Recall; M6-CR: M6-Narrative Memory-Cued Recall.  n = 12. †† n = 11. n = 13. a Student’s t-test. b Mann–Whitney U test.
Table 2. Number of correct responses recorded for each NEPSY-II subtest at pre- and post-test.
Table 2. Number of correct responses recorded for each NEPSY-II subtest at pre- and post-test.
IDGroupL1-CM4-LMM4-LMDM6-FRM6-CR
PrePostPrePostPrePostPrePostPrePost
P1EG1113121111
P2EG1211123132
P3EG13141111111013151315
P4EG4555710
P5EG1111110001
P6EG1101000101
P7EG1611112211
P8EG561121107879
P9EG1101010000
P10EG6821114456
P11EG13110191714
P12EG911877661661
P13EG18
P14EG1111110111
P15CG14140010715614
P16CG1111011111
P17CG221113
P18CG1101010000
P19CG125744
P20CG1101110100
P21CG1210001122
P22CG767881
P23CG81510084442
P24CG1100500000
P25CG88181110577
P26CG1100100000
P27CG1100000000
P28CG1111110000
Note: ID: Identification Code; EG: experimental group; CG: control group; L1-C: L1-Comprehension of Instructions; M4-LM: M4-List Memory; M4-LMD: M4-List Memory Delayed; M6-FR: M6-Narrative Memory-Free Recall; M6-CR: M6-Narrative Memory-Cued Recall. In line with the demographic guidelines of the NEPSI-II, participants for whom the task was deemed not applicable were labeled with “•”.
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

De Luca, M.; Ilardi, C.R.; Dolce, P.; Rega, A.; Di Fuccio, R.; Rubinacci, F.; Gallucci, M.; Marangolo, P. Using Tangible User Interfaces (TUIs): Preliminary Evidence on Memory and Comprehension Skills in Children with Autism Spectrum Disorder. Behav. Sci. 2025, 15, 267. https://doi.org/10.3390/bs15030267

AMA Style

De Luca M, Ilardi CR, Dolce P, Rega A, Di Fuccio R, Rubinacci F, Gallucci M, Marangolo P. Using Tangible User Interfaces (TUIs): Preliminary Evidence on Memory and Comprehension Skills in Children with Autism Spectrum Disorder. Behavioral Sciences. 2025; 15(3):267. https://doi.org/10.3390/bs15030267

Chicago/Turabian Style

De Luca, Mariagiovanna, Ciro Rosario Ilardi, Pasquale Dolce, Angelo Rega, Raffaele Di Fuccio, Franco Rubinacci, Maria Gallucci, and Paola Marangolo. 2025. "Using Tangible User Interfaces (TUIs): Preliminary Evidence on Memory and Comprehension Skills in Children with Autism Spectrum Disorder" Behavioral Sciences 15, no. 3: 267. https://doi.org/10.3390/bs15030267

APA Style

De Luca, M., Ilardi, C. R., Dolce, P., Rega, A., Di Fuccio, R., Rubinacci, F., Gallucci, M., & Marangolo, P. (2025). Using Tangible User Interfaces (TUIs): Preliminary Evidence on Memory and Comprehension Skills in Children with Autism Spectrum Disorder. Behavioral Sciences, 15(3), 267. https://doi.org/10.3390/bs15030267

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop