Next Article in Journal
Safe, Smooth, and Fair Rule-Based Cooperative Lane Change Control for Sudden Obstacle Avoidance on a Multi-Lane Road
Next Article in Special Issue
MinerGuard: A Solution to Detect Browser-Based Cryptocurrency Mining through Machine Learning
Previous Article in Journal
Sodium Hypochlorite Accident during Canal Treatment: Report of Four Cases Documented According to New Standards
Previous Article in Special Issue
WPFD: Active User-Side Detection of Evil Twins
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Applied Sciences
Volume 12
Issue 17
10.3390/app12178527
applsci-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Requesting Help Module Interface Design on Key Partial Video with Action and Augmented Reality for Children with Autism Spectrum Disorder

by
Chuan-Po Wang
1,*,
Cheng-Hui Tsai
2,* and
Yann-Long Lee
3
1
Department of Creative Product Design, National Taichung University of Science and Technology, Taichung 404348, Taiwan
2
Center for General Education, National Taichung University of Science and Technology, Taichung 404348, Taiwan
3
Department of Industrial Design, Chaoyang University of Technology, Taichung 41349, Taiwan
*
Authors to whom correspondence should be addressed.
Appl. Sci. 2022, 12(17), 8527; https://doi.org/10.3390/app12178527
Submission received: 22 July 2022 / Revised: 21 August 2022 / Accepted: 22 August 2022 / Published: 26 August 2022
(This article belongs to the Special Issue Human-Computer Interactions 2.0)
Browse Figures
Review Reports Versions Notes

Abstract

:
Children with autism spectrum disorder (ASD) have marked difficulty with vocabulary, lack of language, or shortcomings with their ability to organize their oral expression; thus, they cannot effectively communicate with others. In particular, people with moderate or severe disabilities cannot systematically narrate an incident and cannot follow pragmatic rules provided by others. Their attempts at standard everyday conversation lead to cognitive problems. When children with ASD are faced with difficult circumstances, they are usually unable to seek help from others, which in turn can result in their being unable to communicate effectively. This research focused on three child participants with ASD and language disorders. The goal was to strengthen the effectiveness of their requesting help and to organize their oral expression, to use requesting help modules, to remove static key images, and to use augmented reality (AR) combined with the dynamic video clips in key partial video with action (KPV). This study developed request-assistance training in conjunction with an auto organizational menu (AOM), multiple case studies and withdrawal designs, training-response methods, and a comparison of outcomes. The proposed AR sentence intervention effectively increased the children’s desire to communicate with others and the accuracy rate of their help requests, and increased their level of communication. We conclude that the interface of our requesting help modules is efficacious enough to assist children with ASD at different levels. The proposed AR sentence intervention helps them build scenarios by themselves, helps them organize communication with their peers, and assists them to request help.

1. Introduction

Autism-related language disorders with moderate or severe disabilities, including a lack of sentence-organization ability, make it difficult for those with autism spectrum disorder (ASD) to communicate with others [1,2]. Children with ASD often have marked difficulty with vocabulary, a lack of or shortcomings with oral organizational capacity, and cannot effectively communicate with others. In particular, people with moderate or severe disabilities cannot systematically narrate an incident and cannot follow pragmatic rules provided by others. Their attempts at standard everyday conversation lead to cognitive problems. When children with ASD are faced with difficult circumstances, they are usually unable to communicate effectively and, therefore, often cannot seek help from others [3]. According to some recent reviews [3,4,5,6], the social participation rate of parents of children with ASD is lower than that of children without ASD; thus, children with ASD often lack adequate role models for social interaction.
An augmented and alternative communication (AAC) board is an assistive technology tool used to compensate for the difficulties faced by people with complex communication needs, such as people with Down syndrome, autism spectrum disorders, intellectual disabilities, cerebral palsy, or impaired speech, for example, apraxia or aphasia [7,8]. These tools allow such individuals to communicate by sequentially selecting and arranging pictograms to form sentences, as shown in Figure 1. “Pictogram” is the term used by the AAC community for a photo or a drawing with a label that represents a concept, for example, an action, object, person, animal, description, or place. Such AAC boards enable children and adults with complex communication needs to communicate and participate in a wide range of environments and activities [9,10,11,12]. However, some studies have pointed out the obstacles or difficulties faced by AAC board users [13,14,15,16,17,18,19]. Whether it is the obstacle of children with autism spectrum disorder (ASD) itself, or the auxiliary communication user, the obstacle of the AAC operation interface is the problem to be considered and tried in this research, and it is especially pointed out that the current design of the AAC interface will affect the operation of the ASD, and also proved through experimental verification that there are indeed many difficulties in the operation of ASD.
A few studies have explored the effectiveness of AAC system interface designs, such as hierarchical menus, pull-down menus, message formulation and retrieval mechanisms, and content presentation methods of picture exchange (PE), picture exchange communication system (PECS), voice output communication aids (VOCAs), and speech-generating devices (SGDs) [3,20]. People with moderate ASD, despite their communication problems, usually communicate using existing AAC communication graphics systems such as: 1. Boardmaker, which is selected and used through Picture Communication Symbols (PCS) and uses the following concepts: (1) build on the grid of board, (2) search for your graphs and vocabulary, (3) fill in the grid, (4) it must use the speaking dynamically to speech; 2. Proloquo2Go, which was designed from the ground up for use on the iPhone, iPod touch, and iPad., and provides an easy-to-use, portable, and affordable communication solution for people with speaking difficulties with the following concepts: (1) select the grip size, (2) search your word and photo, (3) use the drop in the grip, (4) execute the board sentence; 3. Picture Master language software, which improves the overall communication ability of the participants, the use of communication equipment and technology, and the independence of functional communication, and uses the following concepts: (1) build up the grip size, (2) choose the word and photo, (3) use the seal to fill in the grip, (4) execute the board; and 4. iCAN, which was designed and implemented in the system as a teaching-assistive tablet application, with the aim of eliminating the burdensome and complex process of creating and handling collections of paper-based picture cards, and for better progress with their cognition, language, and communication learning. Compared to the traditional approach in creating and retrieving paper-based picture cards, users were able to achieve the analogous task in a quarter and a third of the time, respectively [21,22].
However, if their ability to create intelligible sentences is insufficient, they will find AAC communication graphics systems difficult to use. Many people with ASD cannot follow pragmatics, or grammar rules, or even select appropriate sentence structures or word order rules because of their auxiliary cognitive difficulties. An AAC device user might select a picture of a family to express something like “I want to go home”, “I wonder whether my family will take me home”, or “I want to find my family”. Research into organizational design content is particularly important so that we can design an effective requesting-help module interface. Therefore, the goal of this study is:
  • to explore the use of SGDs by non-verbal children with ASD;
  • to compare the effectiveness of augmentative and alternative communication (AAC) with augmented reality (AR) in operating identification;
  • to evaluate whether the intervention of AAC with AR can increase the users’ accuracy in question responses;
  • to assess the difference in the independent completion rate of the interfaces with prompt;
  • to discuss whether the interfaces with prompts could improve users’ communication ability.

2. Related Work

Many students with ASD have significant deficits in social skills and in cognitive skills such as abstraction and comprehension. Recent research in augmented reality (AR) therapy is encouraging because it has shown that the therapy is effective in promoting, supporting, and protecting the health and well-being of children and adolescents with ASD [23,24,25,26]. AR is a real-time, direct, or indirect view of a physical, real-world environment, elements of which are augmented by computer-generated sensory inputs such as sound, video, and graphics. Related research has begun to be applied in education, and there is an opportunity to improve our understanding of people with ASD in a semantic context with environmental elements, because with advanced AR technology, the real-world information around the user becomes interactive, digitized, and accessible [27,28,29]. Artificial information about the environment and its objects can be overlaid on the real world, and it is believed that it can effectively help users with ASD improve their communication and socializing abilities [30,31,32,33,34].
Augmented reality eBooks provide a fun, novel, and interesting reading experience in cultural education applications. For example, Augmented Reality Magic Books, designed by Billinghurst et al. [9], both integrates animated content into and overlays it on physical books. Readers can turn the pages of these AR books when reading, and they can see a virtual model animation, which allows them to undergo the stimulation of a real reading experience. Escobedo et al. [23] also explored how augmented reality could help integrate the physical and digital worlds by mimicking current strategies for attention management in people with ASD. They developed the Moving Object Recognition System (Mobis), a mobile augmented reality application that allows teachers to overlay digital content on top of physical objects. In addition, many places are remote and may be difficult or impossible to reach in person; this is the case in areas inhabited by isolated aboriginal tribes in places like Amazon River jungles. The internet alone cannot provide an immersive feeling. Virtual reality can teach about the time and space related to the scene and even travel through time and space. Individuals can feel the conditions of the scene, but augmented reality can be further interacted with or experienced after the AR interfaces are setup. Shelton and Hedley [24] found that using augmented reality can help teachers use simpler teaching materials. Augmented reality can increase student perception and improve understanding. However, using static or fragmented images is too limited and ecologically ineffective. Dynamic video has advantages, but it is difficult for children with ASD to focus on dynamic video. Because of this, children with ASD are not always attracted to the media; thus, we use AR technology to grab their attention. Virtual reality allows students with ASD to repeat exercises to compensate for their weak abilities and developmental needs in real life, including social skills, daily etiquette, life management, and skills training [24,25,26,27,28]. Furthermore, to reduce their visual stress and load, we selected suitable content that lasted less than seconds, focused on specific social signals, and slightly combined AR technology with video modeling (VM). Other researchers have provided evidence that AR also helps train typically developing (TD) children [29,30,31,32,33,34].

3. Methods

3.1. The Design Process of the Communication Menu

The communication menu was developed in four stages:
  • In the first stage, we conducted preliminary interviews with more than ten children with autism, parents, and teachers. The Ethics Committee of the China Medical University, Taiwan Ministry of Public Health, provided ethical approval for this study. All participants signed a consent form when explaining the designed questionnaire. We used core vocabulary graphics recording 100 sentences and then select ten questions from all sentences (Appendix A and Supplementary File S1), and then made more than ten communication boards according to the grammar (Table 1) for the study and related words gleaned from these interviews to design the requesting-help communication menu [35]. Some parents, class tutors, and language therapists met regularly to discuss the core vocabulary graphics. They compiled a core vocabulary of more than a thousand commonly used communication terms to be used in our communication graphics system. The design team collected the details of how children with ASD learn communication skills by pointing at paper-based picture cards. They used the concept of core vocabulary to create an interface with which participants can learn by repeatedly practicing specific sentence patterns [36,37,38,39].
  • In the second stage, researchers designed textbooks based on core vocabulary and the ISO copyright-free graphics library (available at https://isorepublic.com; accessed on 1 October 2021) to redesign and select the most frequently used words with graphics to make sentences. For example, Table 1 lists subject, verb, object, adverb, or subject, verb, adjective, adverb, and other combinations of sentence elements. The requesting help module provided the main communicative content—the core vocabulary. Because of different needs, the requesting help module and its dozens of vocabularies were organized into ten categories, and the vocabularies were used as marginal vocabularies, so it could provide the use and expression of choice judgments. [40,41]. Researchers used different colors to classify the different parts of speech (for example, yellow for subject words, orange for affirmative and negative words, green for verbs, and so on). They also used a structured layout design to guide participants in using lexical phrases in proper sequence [42,43,44].
  • In the third stage, because the participants sometimes found the interface too complicated to easily use, we realized that we need to further study the differences in how children with ASD understand meaning. We hoped that this stage of intervention would help us to improve the participants’ abilities to engage in further operational communication and to request help. Therefore, we adjusted the interface to use in formulating eight modules of requesting help recognition concepts. The operational communication content of requesting help was then developed. Scenarios were created (Appendix A Table A1 and Supplementary File S1). We then gave our prototype to parents and caregivers to judge its value and verify its usability for children with ASD [45,46].
  • In the fourth stage, the interface refinement and assessment stage, our system was installed onto a tablet device for the three children (the end users) to test. The designers, special education instructors, and speech therapists assessed the effectiveness of the refined interface. The interface showed a table of the parts of speech, subjects, verbs, objects, nouns, adjectives, adverbs, and conjunctions.
Our participants could select images based on the parts of speech in the table. Because many children with ASD find it difficult to create coherent sentences and engage in articulate conversations, previous researchers have created communication assistance methods and devices to allow them to practice and improve their abilities to converse with other people. Some of the effective methods are the Picture Exchange Communication System (PECS), picture exchange (PE), and speech-generating devices (SGDs). We developed a method with an image-based menu that uses word class labels (S, V, O, N, Adj, Adv, Conj, and Q-word) to generate an image grid to teach children with ASD how to create sentences to ask other people for help. When each figure in the image grid of the user’s potential sentence was filled in, a grammatically correct sentence was automatically generated and then uttered by the SGD connected to our menu [47,48].

3.2. Using the Auto Organizational Menu with AR to Generate a Request for Help

The third stage of this study included: (1) using AR and captured KPV with key scenes. These short videos were then used to stimulate our participants to correctly identify the request for help expressed in two different images which included static key images with real people; and (2) evaluating the outcomes of the auto organizational menu (AOM) intervention for improving operational identification and facilitating the improvement of our participants’ coherent expressions (as shown in Figure 1) [33,34,35].

3.2.1. Participants

There were 5 participants in this study (all given pseudonyms to guarantee anonymity): 2 TD children (2 boys) and 3 children who had previously been diagnosed with ASD (2 boys and 1 girl) by clinicians using the multidisciplinary assessment of clinical services in Taiwan; the Wechsler Intelligence Scale for Children (WISC) was used to determine physical and sensory inabilities that might affect their speech and language development.
The 3 children with ASD (mean age = 7 years old; age range: 6–8 years; intelligence quotient [IQ] scores: [a] full scale IQ [FIQ] = 71.33 ± 3.06; [b] verbal IQ [VIQ] = 73.33 ± 4.16; and [c] performance IQ [PIQ] = 67.33 ± 2.89) had sensory abilities within the abnormal range; e.g., the mean FIQ was lower than 80. All participants signed a children’s consent form, and parental consent forms were obtained before the participants were enrolled in the study. All participants with ASD had a disability identification card issued by a medical institution in Taiwan and had been counseled in special education schools and institutes in Taiwan.

3.2.2. Settings

All instructional and probe sessions and other procedures occurred in teacher-selected computer classrooms in Kaohsiung elementary schools. Participants sat at a table and the trainer sat across from them. To begin the intervention test, an experimenter showed the requesting-help question paper to the children and asked them to look at the pictures of each scene. These ten scenes are generally recognized universally and are created by people all over the world. These scene types were chosen based on caregivers’ and teachers’ preferences. The participants answered each test question (Appendix A Table A1 and Supplementary File S1) after looking at the pictures, and then selected an appropriate answer for the KPV requesting-help expressions (see Figure 2). An independent observer was present during all sessions to collect inter-observer agreement data.
We initially used HP Reveal to make AR prototypes and used Unity Vuforia, AR Kit, and MAKAR as our follow-up development platforms, because they provide the most extensive features and functions for making AR at present. Vuforia supports iOS, Android, and Unity 3D, so the platform allows users to write a native application that reaches the majority of users on the widest range of smartphones and tablets.
We selected from the caregivers and teachers ten scene videos that contained requesting-help communication, illustrated the appropriate physical distance with low levels of metaphor, represented distinctive themes, and illustrated the appropriate physical distance in Taiwanese culture. The scenes included significant nonverbal communication (body movements and clear facial expressions) that vividly expressed the request for help. In addition, there were interactions and dialogue among the characters in each story. Excessively complicated and abstract videos were excluded. All scenes depicted a variety of requests for help and were displayed at the same resolution of 1024 × 768 PPI on a 20-inch monitor. Each scene was approximately 10 s long (as shown in Figure 2).

3.2.3. Key Partial Video with Action (KPV)

The Key Partial Video with Action (KPV) was created from frozen images captured from the Request for Help video. In each video, we selected 10 to 20 short video frames based on video context and story development sequence, including help-seeking actions, situations, and body movements, to develop KPV material for our participants. KPV scenarios were created from video footage of images taken from researchers, caregivers, and teachers. The purpose was to capture key short film scenes and use the key short action scenes of requesting assistance to help participants with ASD who repeat the real film images to understand how to request assistance and to recognize matching (as shown in Figure 3).

3.2.4. Operational Definitions

The participants were asked to (a) select the S, V, objects, and N picture symbols that depicted the sentence they wanted (e.g., ‘‘Please help me go to the nearest toilet’’) (Figure 1); (b) select the image that they needed to use any four images at least one idioms arbitrarily, so that the AOM interface could automatically organize those four components into a complete sentence sequentially; (c) play the selected sentence to send the digital voice message; And (d) use the AR function (Figure 2) of the mobile phone to scan the image file to activate the AR effect of Key Partial Video with Action (KPV) to assist the ASD in the process of understanding how to request help (Figure 3).

3.3. Intervention Strategy

This study combined multiple baseline and reversal design: A-B-C-B. A multiple baseline design is a style of research involving the careful measurement of multiple persons, traits, or settings both before and after treatment [14,16]. The AOM interface content was based on KPV with AR basic social cues. The KPV visual strategy is used to attract their attention on those nonverbal facial cues related to situations in emphasis to indicate the concern of their performance. Each session lasted for approximately one hour. Each participant received 2 h of intervention per week for 4 months and was assessed in terms of their target behaviors in the session until these behaviors became stable. After that, intervention for the first participant began, and the others continued their sessions. When the target behavior of the first participant improved about 80% and was stable, intervention for the second participant began, and so on [32]. Based on the expert interviews, the settings included “the need to express oneself” and comprehension tasks. (Appendix A).

3.3.1. How We Selected Standard Answers

Our selection method used the main request help chosen by the TD children as the standard answer for perception judgment. Because the perception judgments and experience of TD children were consistent, and their comprehension of situations was generally accurate (their comprehension scores were 96.43% on the KPV with AR test), the perceptions of the TD children were also used as the selection criteria for the children with ASD. In addition, the request help select answers were agreed on by the experts and therapists.

3.3.2. Correct Judgment Rate of Children with ASD

The request help chosen by the TD children was selected as the standard answers for the KPV with AR tests. The answers chosen by children with ASD were recorded as correct or incorrect, as appropriate.

3.3.3. Intervention and Evaluation of the Procedure

The study was divided into two phases. The first phase included the A1 baseline period and the B1 treatment period, and the second phase included the C1 reversal period and the B2 treatment period (phases 1–50). Each question request was reinforced by KPV with AR recognition, each participant was asked to recognize auxiliary clues found in the Appendix A, and each participant was allowed to sequentially select graphic actions from the AOM and play with them for 20 s. After each reversal session, which the second stage consisted ofis continuously strengthened. In order to assess whether the two interventions in the baseline case preserved the ability to recognize KPV and AR movements and to use AOM to respond to others, a maintenance period was assessed after approximately one month [32,35].

3.3.4. Intervention Definition

The interface was designed to train children with ASD to treat what some scholars consider either an absence or a delay in the development of the theory of mind in people with ASD. To overcome this cognitive deficit, the affected people must change their understanding of how and what others feel. The steps in the process are as follows.
  • A request for help and a transfer cognitive intervention: using AR and capturing KPV images to make participants’ intervention assessments of how to request help using photographs and identifying and recognizing action from schematic drawings, to teach participants with ASD how to understand the correct way to request help. Experimenters asked participants to choose the subjects, nouns, verbs, objects, adjectives, adverbs, and question words to allow the AOM to automatically organize sentences needed for the intervention task.
  • The experimenters told the participants what kind of help to request, to execute the tasks in the Appendix A, to pay attention to social interaction distance and continuity, and to distinguish between two different kinds of real situations and between ways of requesting help. This study used a combination of multiple baseline and reversal designs: A-B-C-B to evaluate the two intervention phases and analyze the outcomes of the intervention.
  • Task execution: the experimenter told the participants what kind of help in the test to request, and to evaluate whether they understood and performed the tasks in the Appendix A. They wanted to use some distance to evaluate the help-seeking activity, and they wanted to evaluate the continuous effectiveness of the training. They used two different real-world situations for the participants to perform the task of asking for help, and they used a combination of multiple baseline and reversal designs (A-B-C-B) to assess the two intervention phases and analyze the outcomes of the intervention.

4. Results

4.1. Results of KPV with AR, AOM Intervention, and Evaluation

4.1.1. Judgments of Perceptions of Others

TD children scored 88% on the KPV with AR test and 98% on the AOM test when using the A-B-C-B designs. On the other hand, the correct judgment rate of the perceptions of others of the children with ASD improved from Baseline 1 at 83% when using the AOM to 98% at Intervention 2 when using KPV with AR.

4.1.2. Situational Comprehension

The situational understanding rate of the three participants with ASD also increased from a Baseline of 34% to a Reversal of 56%. In the AOM intervention, determining the correct expression of KPV expression also significantly (p < 0.05) increased from 69% to 72% (Table 2).
We used paired t-tests to compare the Baseline test values with the Intervention 1, Reversal, Intervention 2, and Maintenance test values. Baseline, Reversal, and Intervention 1, 2 Paired sample t-tests were used to compare whether there was a significant difference in outcomes between the two groups. The resulting p-values were 0.0027, and the tests Intervention 1, and 2 were all significantly (p < 0.05) higher than the Baseline values.

4.2. Interaction Effect

The KPV with AR intervention helped the children improve their ability to judge and determine the relationships between roles and activities, and the AOM helped them manipulate the answers relating to the sentence and communicate with the therapist. This shows that a limited amount of information with structured and specific close-up images helped the children improve their situational awareness and perceptions of others. Although children with ASD might encounter passive barriers, the visual support and structured situational characteristics of the scenes were beneficial to their perceptual awareness, and also helped them to develop their social interaction function.

5. Conclusions

5.1. Discussion

At Baseline, the researchers reported that participants always focused on insignificant parts of the video scene, such as the number or timing of the video. In general, they often focus only on things that interest them, such as repetitive demands to keep playing a certain segment of a movie or an action. In addition, through the researcher’s discussions with the participants about the film’s character’s behavior, we found that they had difficulty comprehending and identifying the purpose of the behavior. Although participants could choose the correct words to describe behaviors, they were usually unable to identify the purpose of those behaviors [13]. This may be why all of the children scored lower at Baseline.
However, during the Intervention phase, using KPV as a learning material guided their attention to key paragraph elements in dynamic video clips. Although sometimes they still focused on irrelevant parts of the entire video scene, KPV often helped them focus their attention on the behavioral meaning of the nonverbal feelings of key passages expressed in specific social situations. They were drawn to the characters and began to ask the researcher a series of questions about why the characters behaved as they did, about their actions, and about related social activities. Based on the results obtained, the combination of KPV with AR technology appears to have had a positive effect on improving the social interaction [27] and verbal and non-verbal communication skills, facial emotion recognition procedures, attention skills, and life functions of children with autism.
In related studies, AR technology has yielded many beneficial outcomes in social interaction, emotion recognition, attention skills, and functional learning [25,26,27,28,29,30,31,32,33,34]. Therefore, it can be seen from our research results that the creative introduction of KPV and combining AR technology and AR applications is beneficial for treating this developmental disorder [13]. First, AR builds on the common strengths of children with ASD, such as visual learning styles or preferences for the visual display of information. In addition, some people with ASD may display a natural tendency to use digital tools, show a fondness for electronic media, and enjoy elements of gaming. KPVs with AR technology can help reduce the stress that often occurs in real-world social situations.
AR can therefore provide engaging learning effects to help children with ASD stay focused, promote participation in activities, facilitate task behaviors, motivation, and positive emotions, and enhance the learning process. Likewise, educators’ perceptions of AR suggest that it is highly inclusive, usable, and engaging [30,31,32,33,34].
Although the training is only based on the content needed for requesting help, it should be combined with life situations and focus on certain situational scenarios through KPV to train children with ASD. It does need more research topics and a better design, with better methodology and more robust results, but we can say that this is an expanding field of research and these are mostly pilot studies. Nonetheless, studies suggest that the combination of KPV and AR technology is necessary to expand the scope of our research [49,50]. The number of children with ASD has been steadily rising over the past quarter century, and efficacious interventions are required to meet their unique individual needs. KPV with AR technology appears to be suitable for the learning style of children with ASD and their interest in visual stimuli. The present study helps us to understand the effectiveness of help-seeking. Our results are promising, but more high-quality studies are needed.

5.2. Limitations

As we all know, the target of autism is almost a small number of ASD, it is difficult to find a large number of suitable ASD in a region to achieve normal distribution, human resources are also limited, related journal research papers are also encountered in this situation, and it is not easy to find suitable ASD under the epidemic (we have conducted a Ministry of Science and Technology study, and the research team has been searching for nearly a year, and currently, only five ASD have been received, one of which is still inappropriate). But our research has made considerable efforts. There are more than 100 related request help contents, and the required planning interface content is produced, the required help videos are shot, and the AR display mode is set. In terms of experiments, we found a few suitable ASD from the inquiry and confirmation of more than ten nearby institutions and primary schools (some cannot control their emotions, cannot operate stably, or cannot meet the WISC standards and must be excluded), and consult parents and teachers. It can only be carried out after consent. On a case-by-case basis, the experimental intervention and interactive evaluation will be carried out in the school every week. For the validation part of the study, because it is a small sample and does not reach the normal distribution, such studies are usually validated with a single-subject applied behavioral analysis, nonparametric statistics, or a t-test.

Supplementary Materials

The following supporting information can be downloaded at: https://www.mdpi.com/article/10.3390/app12178527/s1, File S1: Interactive plot data.

Author Contributions

Conceptualization, C.-P.W.; methodology, C.-P.W. and Y.-L.L.; validation, C.-P.W. and C.-H.T.; formal analysis, C.-P.W. and Y.-L.L.; investigation, C.-P.W.; resources, Y.-L.L. and C.-P.W.; data curation, C.-P.W.; writing—original draft preparation, C.-P.W.; writing—review and editing, C.-P.W.; visualization, C.-P.W. and C.-H.T.; supervision, C.-P.W.; project administration, C.-P.W. and C.-H.T.; funding acquisition, C.-P.W. and C.-H.T. All authors have read and agreed to the published version of the manuscript.

Funding

This work was supported by the Ministry of Science and Technology, Taiwan.

Institutional Review Board Statement

This research was approved by the institutional review board (IRB) of the Ethics Committee of the China Medical University, Taiwan.

Informed Consent Statement

Informed consent was obtained from all participants involved in the study.

Data Availability Statement

Not applicable.

Acknowledgments

We thank all the caregivers who participated in this study, and thank the Autism Society of Taiwan for assisting with data collection. All authors have read and agreed to the published version of the manuscript.

Conflicts of Interest

The Ethics Committee of the China Medical University, Taiwan Ministry of Public Health provided ethical approval for this study.

Appendix A

Table
Table A1. Request help with experimental intervention record form (blank).
Table A1. Request help with experimental intervention record form (blank).
Title Design/SessionThe Correct Rate\Time
Intervention121212
Participant123
Requesting help
1. Please help me go to the nearest convenience store.
2. Please help me go to the nearest toilet.
3. Please take me to the vegetarian restaurant.
4. Please help me go to my teacher’s office.
5. Please help me make a telephone call to my family.
6. Please help me ask my family to be there.
7. Please help me hand in homework to my teacher.
8. Please take me to my school playground.
9. Please take me to the nearest library.
10. Please take me to the nearest health center.
Accuracy & Action%
Examples of Core Vocabulary Questions.

References

  1. Kurtcu, F. An analyze of high school web interface designs in terms of graphic design. Procedia Soc. Behav. Sci. 2012, 46, 5661–5665. [Google Scholar] [CrossRef]
  2. Binger, C.; Light, J. The morphology and syntax of individuals who use AAC: Research review and implications for effective practice. Augment. Altern. Commun. 2008, 24, 123–138. [Google Scholar] [CrossRef] [PubMed]
  3. Beukelman, D.; Mirenda, P. Augmentative and Alternative Communication: Supporting Children and Adults with Complex Communication Needs, 4th ed.; Paul Brookes: Baltimore, MD, USA, 2013. [Google Scholar]
  4. Boesch, M.C.; Wendt, O.; Subramanian, A.; Hsu, N. Comparative efficacy of the Picture Exchange Communication System (PECS) versus a speech-generating device: Effects on social-communicative skills and speech development. Augment. Altern. Commun. 2013, 29, 197–209. [Google Scholar] [CrossRef] [PubMed]
  5. Van der Meer, L.A.J.; Rispoli, M. Communication interventions involving speech-generating devices for children with autism: A review of the literature. Dev. Neurorehabil. 2010, 13, 294–306. [Google Scholar] [CrossRef]
  6. Sigafoos, J.; Green, V.A.; Payne, D.; Son, S.H.; O’Reilly, M.; Lancioni, G.E. A Comparison of Picture Exchange and Speech-Generating Devices: Acquisition, Preference, and Effects on Social Interaction. Augment Altern. Commun. 2009, 25, 99–109. [Google Scholar] [CrossRef]
  7. American Psychiatric Association. Diagnostic and Statistical Manual of Mental Disorders, 5th ed.; American Psychiatric Association: Washington, DC, USA, 2013. [Google Scholar]
  8. Robert, C.; Pasquier, L.; Cohen, D.; Fradin, M.; Canitano, R.; Damaj, L.; Tordjman, S. Role of genetics in the etiology of Autistic Spectrum Disorder: Towards a hierarchical diagnostic strategy. Int. J. Mol. Sci. 2017, 18, 618. [Google Scholar] [CrossRef]
  9. Billinghurst, M.; Hirokazu, K.; Ivan, P. The Magic Book: A Transitional AR Interface. Comput. Graph. 2001, 25, 745–753. [Google Scholar] [CrossRef]
  10. Taipei Autism Children Social Welfare Foundation. Prevalence of Autism in Taiwan. Available online: http://www.ican.org.tw/content.asp?id=2 (accessed on 15 August 2021).
  11. Tick, B.; Bolton, P.; Happé, F.; Rutter, M.; Rijsdijk, F. Heritability of autism spectrum disorders: A meta-analysis of twin studies. J. Child Psychol. Psychiatry 2016, 57, 585–595. [Google Scholar] [CrossRef]
  12. Tordjman, S.; Somogyi, E.; Coulon, N.; Kermarrec, S.; Cohen, D.; Bronsard, G.; Xavier, J. Gene × environment interactions in autism spectrum disorders: Role of epigenetic mechanisms. Front. Psychiatry 2014, 5, 53. [Google Scholar] [CrossRef]
  13. Chen, C.H.; Lee, I.J.; Lin, L.Y. Augmented reality-based video-modeling storybook of nonverbal facial cues for children with autism spectrum disorder to improve their perceptions and judgments of facial expressions and emotions. Comput. Hum. Behav. 2016, 55, 477–485. [Google Scholar] [CrossRef]
  14. Chen, L.; Xu, L.; Dhar, S.; Li, M.; Talwar, D.; Jung, E. Autism spectrum disorders: A qualitative study of attitudes toward prenatal genetic testing and termination decisions of affected pregnancies. Clin. Genet. 2015, 88, 122–128. [Google Scholar] [CrossRef]
  15. Johnston, D.; Hauke, E.; Gavin, K. SoundFields: A virtual reality game designed to address auditory hypersensitivity in individuals with autism spectrum disorder. Appl. Sci. 2020, 10, 2996. [Google Scholar] [CrossRef]
  16. Masi, A.; DeMayo, M.M.; Glozier, N.; Guastella, A.J. An overview of Autism Spectrum Disorder, heterogeneity and treatment options. Neurosci. Bull. 2017, 33, 183–193. [Google Scholar] [CrossRef]
  17. Jiang, Y.; Wang, Y.; Xiu, X.; Choy, K.; Pursley, A.; Cheung, S. Genetic diagnosis of autism spectrum disorders: The opportunity and challenge in the genomics era. Crit. Rev. Clin. Lab. Sci. 2014, 51, 249–262. [Google Scholar] [CrossRef]
  18. Rossi, M.; El-Khechen, D.; Black, M.H.; Hagman, K.D.F.; Tang, S.; Powis, Z. Outcomes of Diagnostic Exome Sequencing in Patients With Diagnosed or Suspected Autism Spectrum Disorders. Pediatr. Neurol. 2017, 70, 34–43. [Google Scholar] [CrossRef]
  19. Wijker, C.; van der Steen, S.; Spek, A.; Leontjevas, R.; Enders-Slegers, M.J. Social development of adults with autism spectrum disorder during dog-assisted therapy: A detailed observational analysis. Int. J. Environ. Res. Public Health 2020, 17, 5922. [Google Scholar] [CrossRef]
  20. ASHA, Definition of Communication and Appropriate Targets. Available online: https://www.asha.org/njc/definition-of-communication-and-appropriate-targets/ (accessed on 15 August 2021).
  21. McNaughton, D.; Light, J.; Beukelman, D.R.; Klein, C.; Nieder, D.; Nazareth, G. Building capacity in aac: A person-centred approach to supporting participation by people with complex communication needs. Augment. Altern. Commun. 2019, 35, 56–68. [Google Scholar] [CrossRef]
  22. Chung, Y.C.; Carter, E.W. Promoting peer interactions in inclusive classrooms for students who use speech-generating devices. Res. Pract. Pers. Sev. Disabil. 2013, 38, 94–109. [Google Scholar] [CrossRef]
  23. Escobedo, L.; Tentori, M.; Quintana, E.; Favela, J.; Garcia-Rosas, D. Using augmented reality to help children with autism stay focused. IEEE Pervasive Comput. 2014, 1, 38–46. [Google Scholar] [CrossRef]
  24. Shelton, B.E.; Hedley, N.R. Using Augmented Reality for Teaching Earth-Sun Relationships to Undergraduate Geography Students. In Proceedings of the First IEEE International Workshop Agumented Reality Toolkit, Darmstadt, Germany, 29 September 2002; Volume 8. [Google Scholar]
  25. Chen, C.H.; Chou, Y.Y.; Huang, C.Y. An augmented-reality-based concept map to support mobile learning for science. Asia-Pac. Educ. Res. 2016, 25, 567–578. [Google Scholar] [CrossRef]
  26. Berenguer, C.; Baixauli, I.; Gómez, S.; Andrés, M.D.E.P.; De Stasio, S. Exploring the impact of augmented reality in children and adolescents with autism spectrum disorder: A systematic review. Int. J. Environ. Res. Public Health 2020, 17, 6143. [Google Scholar] [CrossRef]
  27. Wang, C.P.; Tsai, C.H. Application of virtual reality to the study of festival culture in aboriginal literature. In Proceedings of the 2nd International Conference on Image and Graphics Processing, Singapore, 23–25 February 2019; pp. 136–139. [Google Scholar]
  28. Tsai, C.H.; Wang, C.P. An Analysis of the Development of Cultural Augmented Reality Education in Taiwanese Aboriginal Literature. Int. J. E-Educ. E-Bus. E-Manag. E-Learn. 2019, 9, 357–365. [Google Scholar] [CrossRef]
  29. Lim, K.C.; Selamat, A.; Alias, R.A.; Krejcar, O.; Fujita, H. Usability Measures in Mobile-Based Augmented Reality Learning Applications: A Systematic Review. Appl. Sci. 2019, 9, 2718. [Google Scholar] [CrossRef]
  30. El-Seoud, M.; Halabi, O.; Geroimenko, V. Assisting Individuals with Autism and Cognitive Disorders: An Augmented Reality based Framework. Int. J. Online Biomed. Eng. 2019, 15, 28–39. [Google Scholar] [CrossRef]
  31. Quintero, J.; Baldiris Navarro, S.M.; Rubira, R.; Cerón, J.; Velez, G. Augmented Reality in Educational Inclusion. A Systematic Review on the Last Decade. Front. Psychol. 2019, 10, 1835. [Google Scholar] [CrossRef]
  32. Baragash, R.S.; Al-Samarraie, H.; Alzahrani, A.I.; Alfarraj, O. Augmented reality in special education: A meta-analysis of single-subject design studies. Eur. J. Spec. Needs Educ. 2019, 35, 382–397. [Google Scholar] [CrossRef]
  33. Marto, A.; Almeida, H.A.; Gonçalves, A. Using Augmented Reality in Patients with Autism: A Systematic Review. In Proceedings of the ECCOMAS Thematic Conference on Computational Vision and Medical Image Processing, Porto, Portugal, 16–18 October 2019; Springer: Cham, Switzerland, 2019; pp. 454–463. [Google Scholar]
  34. Khowaja, K.; Banire, B.; Al-Thani, D.; Sqalli, M.T.; Aqle, A.; Shah, A.; Salim, S.S. Augmented reality for learning of children and adolescents with autism spectrum disorder (ASD): A systematic review. IEEE Access 2020, 8, C78779–C78807. [Google Scholar] [CrossRef]
  35. Chen, C.H.; Wang, C.P.; Su, C.C. The effectiveness of using auto organizational menu to communicate with classmates: A case study of autism spectrum disorders. In Proceedings of the 6th International Conference on Software and Computer Applications, Bangkok, Thailand, 26–28 February 2017; pp. 304–309. [Google Scholar]
  36. Vanderheiden, G.; David, K. Comparative analysis of fixed-vocabulary communication acceleration techniques. Augment. Altern. Commun. 1987, 3, 196–206. [Google Scholar] [CrossRef]
  37. Hervás, R.; Bautista, S.; Méndez, G.; Galván, P.; Gervás, P. Predictive composition of pictogram messages for users with autism. J. Ambient. Intell. Humaniz. Comput. 2020, 11, 5649–5664. [Google Scholar] [CrossRef]
  38. Garcia, L.F.; de Oliveira, L.C.; de Matos, D.M. Evaluating pictogram prediction in a location-aware augmentative and alternative communication system. Assist. Technol. 2016, 28, 83–92. [Google Scholar] [CrossRef]
  39. Pereira, J.; Franco, N.; Fidalgo, R. A semantic grammar for augmentative and alternative communication systems. In Text, Speech, and Dialogue; Sojka, P., Kopeček, I., Pala, K., Horák, A., Eds.; Springer International Publishing: Cham, Switzerland, 2020; pp. 257–264. [Google Scholar]
  40. Martínez-Santiago, F.; Díaz-Galiano, M.C.; Urena-López, L.A.; Mitkov, R. A semantic grammar for beginning communicators. Knowl. -Based Syst. 2015, 86, 158–172. [Google Scholar] [CrossRef]
  41. Goldberg, Y.; Hirst, G. Neural Network Methods in Natural Language Processing; Morgan & Claypool: San Rafael, SR, USA, 2017; pp. 104–113. [Google Scholar]
  42. Franco, N.; Silva, E.; Lima, R.; Fidalgo, R. Towards a reference architecture for augmentative and alternative communication systems. In Brazilian Symposium on Computers in Education (Simpósio Brasileiro de Infor660 mática na Educação-SBIE); 2018; Volume 29, p. 1073. Available online: https://www.researchgate.net/publication/328735940_Towards_a_Reference_Architecture_for_Augmentative_and_Alternative_Communication_Systems (accessed on 1 July 2022).
  43. Ng, C.K.; Lam, S.H.; Tsang, S.T.; Yuen, C.; Chien, C.W. The relationship between affiliate stigma in parents of children with autism spectrum disorder and their children’s activity participation. Int. J. Environ. Res. Public Health 2020, 17, 1799. [Google Scholar] [CrossRef]
  44. Pereira, J.; Pena, C.; de Melo, M.; Cartaxo, B.; Fidalgo, R.; Soares, S. Facilitators and barriers to using alternative and augmentative communication systems by aphasic: Therapists perceptions. In Proceedings of the 2019 IEEE 32nd International Symposium on Computer-Based Medical Systems (CBMS), Cordoba, Spain, 5–7 June 2019; Volume 645, pp. 349–354. [Google Scholar]
  45. Donato, C.; Spencer, E.; Arthur-Kelly, M. A critical synthesis of barriers and facilitators to the use of aac by children with autism spectrum disorder and their communication partners. Augment. Altern. Commun. 2018, 34, 242–253. [Google Scholar] [CrossRef] [PubMed]
  46. Devlin, J.; Chang, M.W.; Lee, K.; Toutanova, K. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the 685 Association for Computational Linguistics: Human Language Technologies, Minneapolis, MN, USA, 2–7 June 2019; Volume 1, (Long and Short Papers). Association for Computational Linguistics: Minneapolis, MN, USA, 2019; pp. 4171–4186. Available online: https://www.aclweb.org/anthology/N19-1423 (accessed on 1 July 2022). [CrossRef]
  47. Scarlini, B.; Pasini, T.; Navigli, R. With More Contexts Comes Better Performance: Contextualized Sense Embeddings for All-Round Word Sense Disambiguation. In Proceedings of the 2020 Conference on Empirical 700 Methods in Natural Language Processing, Association for Computational Linguistics, Online, 16–20 November 2020. [Google Scholar] [CrossRef]
  48. Ko’s Obstetrics & Gynecology. The International Standards for Cytogenomic Arrays Cortium Analysis Application Procedure. Available online: http://www.genephile.com.tw/aCGH/htm/ISCA%20array.htm (accessed on 15 August 2021).
  49. Saturno, C.E.; Ramirez, A.R.G.; Conte, M.J.; Farhat, M.; Piucco, E.C. An augmentative and alternative communication tool for children and adolescents with cerebral palsy. Behav. Inf. Technol. 2015, 34, 632–645. [Google Scholar] [CrossRef]
  50. Lin, P.C.; Peng, L.Y.; Hsiao, R.C.; Chou, W.J.; Yen, C.F. Teacher harassment victimization in adolescents with high-functioning autism spectrum disorder: Related factors and its relationships with emotional problems. Int. J. Environ. Res. Public Health 2020, 17, 4057. [Google Scholar] [CrossRef] [PubMed]
Applsci 12 08527 g001 550
Figure 1. Using the Auto Organizational Menu to request help.
Figure 1. Using the Auto Organizational Menu to request help.
Applsci 12 08527 g001
Applsci 12 08527 g002 550
Figure 2. Ten scene videos and trigger definitions.
Figure 2. Ten scene videos and trigger definitions.
Applsci 12 08527 g002
Applsci 12 08527 g003 550
Figure 3. Example of KPV materials created from AR.
Figure 3. Example of KPV materials created from AR.
Applsci 12 08527 g003
Table
Table 1. Use to Make Sentences and Rules.
Table 1. Use to Make Sentences and Rules.
Style No.Sentence and Rule
1SVN
2SVAdj
3SVAdvAdj
4SVAdjN
5SConjOVAdvAdj
6SConjOVAdjN
S = subject; V = verb; N = noun; Adj = adjective; Adv = adverb; Conj = conjunction; O = object.
Table
Table 2. Intervention and evaluation result.
Table 2. Intervention and evaluation result.
Session\ParticipantsP1P2P3AverageTD1TD2Average
Baseline33284234858183
Intervention170696969989697
Intervention272766872999999
Reversal53575756939092
Maintenance58585858939594
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Wang, C.-P.; Tsai, C.-H.; Lee, Y.-L. Requesting Help Module Interface Design on Key Partial Video with Action and Augmented Reality for Children with Autism Spectrum Disorder. Appl. Sci. 2022, 12, 8527. https://doi.org/10.3390/app12178527

AMA Style

Wang C-P, Tsai C-H, Lee Y-L. Requesting Help Module Interface Design on Key Partial Video with Action and Augmented Reality for Children with Autism Spectrum Disorder. Applied Sciences. 2022; 12(17):8527. https://doi.org/10.3390/app12178527

Chicago/Turabian Style

Wang, Chuan-Po, Cheng-Hui Tsai, and Yann-Long Lee. 2022. "Requesting Help Module Interface Design on Key Partial Video with Action and Augmented Reality for Children with Autism Spectrum Disorder" Applied Sciences 12, no. 17: 8527. https://doi.org/10.3390/app12178527

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop