Technology-Facilitated Diagnosis and Treatment of Individuals with Autism Spectrum Disorder: An Engineering Perspective

Abstract

The rapid development of computer and robotic technologies in the last decade is giving hope to perform earlier and more accurate diagnoses of the Autism Spectrum Disorder (ASD), and more effective, consistent, and cost-conscious treatment. Besides the reduced cost, the main benefit of using technology to facilitate treatment is that stimuli produced during each session of the treatment can be controlled, which not only guarantees consistency across different sessions, but also makes it possible to focus on a single phenomenon, which is difficult even for a trained professional to perform, and deliver the stimuli according to the treatment plan. In this article, we provide a comprehensive review of research on recent technology-facilitated diagnosis and treat of children and adults with ASD. Different from existing reviews on this topic, which predominantly concern clinical issues, we focus on the engineering perspective of autism studies. All technology facilitated systems used for autism studies can be modeled as human machine interactive systems where one or more participants would constitute as the human component, and a computer-based or a robotic-based system would be the machine component. Based on this model, we organize our review with the following questions: (1) What are presented to the participants in the studies and how are the content and delivery methods enabled by technologies? (2) How are the reactions/inputs collected from the participants in response to the stimuli in the studies? (3) Are the experimental procedure and programs presented to participants dynamically adjustable based on the responses from the participants, and if so, how? and (4) How are the programs assessed?

1. Introduction

Autism Spectrum Disorder (ASD) is a neurological and developmental disorder. ASD presents with a constellation of physiologic and behavioral symptoms. There are two essential domains of ASD symptoms: (1) restrictive and repetitive behaviors, for example, having a lasting and intense interest in certain topics, including sensory challenges; (2) difficulty in social communication and interaction, such as failing to make proper eye contact during a conversion with another person. Although the exact causes for ASD are not known and there is no cure, research has shown that symptoms can be improved via proper treatment, especially when detected in early childhood.

It was estimated that 1.1% of population in the United States suffer from Autism Spectrum Disorders (ASD), which amounts to more than three million people. The cost of supporting an individual with an ASD ranges from $1.4 million for those without intellectual disability to $2.3 million for those with intellectual disability during his/her lifespan (https://www.autismspeaks.org/about-us/press-releases/annual-cost-of-autism-triples). The largest cost components for children with ASD were special education services and parental productivity loss. Although the final cost is high for people with ASD, the actual damages caused by ASD are far beyond that. Both caregivers and people with ASD can be severely stressed due to the difficulty of proper mutual communication because people with ASD are often overwhelmed and anxious in social interactions. The lack of social skills for people with ASD will also significantly impact their personal lives and hamper their (future) career opportunities, especially for children. Many children with ASD have no friends and even have difficulty interacting with their parents. Not only do they often suffer from strong loneliness, but the frustration may also lead to violent behaviors, which will put both themselves and their loved ones in danger.

The rapid development of computer and robotic technologies in the last decade is giving hope to perform earlier and more accurate diagnosis of ASD, and more effective, consistent, and cost-conscious treatment. Besides the reduced cost, the main benefit of using technology to facilitate treatment is that stimuli produced during each session of the treatment can be controlled, which not only guarantees consistency across different sessions, but also makes it possible to focus on a single phenomenon, which is difficult even for a trained professional to perform, and deliver the stimuli according to the treatment plan. Computer based research typically uses virtual avatars or characters to represent therapists and other related facilitators, while robotic based treatment research typically uses humanoid robots. The computer based approach has the advantage of producing virtual reality scenes and scenarios for a treatment study, while the robotic based approach has the advantage of offering a real-human like physical embodiment.

In this article, we provide a comprehensive review of research on recent technology-facilitated diagnosis and treatment of children and adults with ASD. There are several recent reviews on similar topics. Wieckowski and White [1] provided an excellent review on technology-based interventions for children with social communication impairment. They reviewed work in the most recent seven years. They categorized the research into two major areas: (1) the reception of technology-based communication, and (2) the production of technology-based communication. For each area, the work is further divided into facial communication and non-facial communication. The research is predominately on facial communication. Non-facial communication includes verbal and gestures. The technologies that have been used include computer, mobile, virtual reality, and robotics. The discussions are focused on the clinical impact of these research works.

Aresti-Bartolome and Garcia-Zapirain reviewed the technology facilitated research work on ASD care [2]. They divided the technologies used into four categories: (1) virtual reality; (2) dedicated applications; (3) telehealth applications, and (4) robotic based. For each technology category, three research objectives are considered: (1) communication and interaction; (2) social learning and imitation skills; and (3) other miscellaneous.

There are several reviews on using a particular technology, such as virtual technology, social robotics, and serious games, to treat individuals with ASD. Scassellati et al. [3] reviewed about ten-year’s work on using socially assistive robots for autism research up to 2012. The focus was on issues related to robot design, human-robot interaction, and system evaluation. The authors specifically highlighted the need to achieve robot autonomy because the operator-controlled method has severe limitations. Diehl et al. reviewed 15 papers that reported clinical use of social robots for individuals with ASD as of 2012 [4]. This review focused on four aspects of the studies, including the response of individuals with ASD to robots, the various ways of using robots to elicit behaviors, and using robots to teach social skills, and using robots to provide feedback. The authors concluded that most of the work reviewed were exploratory and had severe limitations. Even though it is not our focus, we find the situation has not been improved in our current review. Cabibihan et al. [5] outlined robot design features, complied a list of robotic systems that have been used for ASD therapy up to 2013, and highlighted a set of activities used for ASD therapy, including imitation, eye contact, joint attention, turn-taking, emotion recognition and expression, self-initiated interactions, and triadic interactions. The article also summarized the roles that could be played by a social robot, including as a diagnostic agent, as a friendly playmate, as a behavior eliciting agent, as a social mediator, as a social actor, and as a personal therapist. Pennisi et al. [6] provided a systematic review of research for autism treatment using social robots from a clinical perspective. A major contribution of this review is that it formulated ten questions regarding the essential roles and benefits of robots for ASD treatments, and reported their findings based on the studies that they have collected up to 2015 in light of these questions. Begum et al. [7] reviewed research on using social robots for autism treatment up to 2016. They pointed out that there is little progress in making social robots as clinical useful for ASD treatment.

Parsons provided a review on autism research facilitated by virtual reality technology [8]. This review focuses on the impact of veridicality of the virtual reality technology in autism treatment under two main themes: (1) using virtual environments for learning and intervention for individuals with ASD to better prepare them to handle real world interactions; and (2) using virtual environments to study how individuals with ASD would behave under predefined social scenarios.

Zakari, Ma, and Simmons [9] presented a short review of serious games designed for children with ASD to improve their communication skills and social behavior. These games are designed to be played via touch-enabled mobile devices or via traditional computers using a keyboard and a mouse. Very recently, Grossard et al. provided a more up-to-date review on using serious games to teach individuals with ASD to improve their social interactions and emotion skills [10]. They focused on the target skills of the serious games, and the design principles of the games. They also paid attention to whether or not the games have been validated in a clinical population.

Different from the above reviews, which predominantly concern clinical issues, we focus on the engineering perspective of autism studies. All technology facilitated systems used for autism studies can be considered as a form of human machine interactive systems where one or more participants would constitute as the human component, and a computer-based or a robotic-based system would be the machine component. As shown in Figure 1, the machine can be further divided into two subcomponents: (1) one subcomponent to deliver the output of the machine, typically in terms of visual and possibly audio content; (2) the other subcomponent takes input from the human, which could be directly from the human via a keyboard/mouse or a touch-screen, or indirectly via one or more wearable sensing devices, such as those for heart rate or skin conductance, or one or more remote tracking systems, such as those for eye-tracking or for gesture/body language tracking. Depending on the sophistication level of the human machine system, there may exist a feedback loop where the output of the system is dynamically altered in response to the input to the system. The feedback loop can be pre-programmed into the system, or controlled manually by a trained clinician (typically referred to as the Wizard of Oz scheme [3]). Furthermore, even without a realtime feedback loop, the effectiveness of a program can be evaluated via the data collected by the system objectively via technology, or subjectively from a clinical professional. In some clinical-oriented studies, pre-test and post-test are administered on the social behaviors of participants to assess the effectiveness of the treatment programs using relevant clinically-proven standardized tests.

Based on the model shown in Figure 1, we organize our review with the following questions: (1) What are presented to the participants in the studies and how are the content and delivery methods enabled by technologies? (2) How are the reactions/inputs collected from the participants in response to the stimuli in the studies? (3) Are the experimental procedure and programs presented to participants dynamically adjustable based on the responses from the participants, and if so, how? and (4) How are the programs assessed?

Research Contribution

With the rapid development of new technologies, especially information technology and the unprecedented prevalence of autism spectrum disorder among the younger generation, researchers have shown tremendous interest in the field of technology-facilitated diagnosis and treatment of ASD. While it is an emerging field with promising opportunities, it is also one with challenges that are not typically faced by engineers, given that even experts in psychology and medical science are not sure about the causes and best treatment of the many different forms of ASD. Thus timely review of research in this field is crucial for researchers to navigate in this relatively new territory with good understanding of what we have learned from previous research efforts. Compared to previous reviews on similar topics, our review makes unique contribution in several aspects.

First, by utilizing a variety of literature search engines including Web of Science and Google Scholar, it breaks the boundaries among several disciplines that are highly involved in technology-facilitated diagnosis and treatment of ASD (psychology, medical science, education, and engineering) to provide a summary of the collective research efforts by researchers from each discipline.

Second, by focusing on specific technology tools and applications, which serves as the common denominator, it overcomes the barriers caused by the idiosyncratic aspect of different terminology used in each discipline and presents a cross-disciplinary synthesis of the state of art research in the field that researchers can easily relate to regardless of their background. Unlike reviews that focus on specific domains, our review would facilitate more and more efficient sharing and co-construction of knowledge among researchers regarding technology-facilitated diagnosis and treatment of ASD across disciplines.

Third, our review takes into consideration that this is a relatively new and rapidly evolving field and aims at carving out the major areas of scientific exploration, identifying the areas that have been researched heavily as well as areas that demand more attention, evaluating existing evidence supporting the effectiveness of different technologies, and pointing out potentially fruitful directions that researchers can follow in the future.

2. Inclusion of Literature

We aim to provide a semi-systematic review of the studies that employ various technologies in the diagnosis and treatment of individuals with ASD. While it is desirable to have a rigorous systematic review of the topic, where all relevant publications are identified and carefully reviewed, several factors prevented us from doing so: (1) the review of numerous publications takes many months and during this period, new important publications were published and we do not think it is appropriate to ignore these latest publications; (2) there lacks a single search tool that can index all relevant publications and we do not have the manpower to exhaustively search for all databases for relevant publications. Therefore, we decided to rely on the use of Web of Science (all databases) as the main vehicle to find available publications and subjectively identify those that are relevant. We supplemented the search using Google Scholar to find relevant publications that are missing in Web of Science. We decided to focus on literature published since 2012 and only highly cited papers published earlier are included.

We primarily used three search terms, “serious games autism”, “virtual reality autism” and “social robots autism”. These terms help us obtain publications on autism treatment based on serious games, virtual reality technology, and social robots. The result for each search is manually screened to remove duplicates, misclassified papers (such as a paper on social robots is included in the search result for virtual reality) and out of scope papers (such as those not focused on autism). We selected papers on the diagnosis of individuals with ASD and and those on helping individuals with ASD to improve their social skills and emotion regulation. We intentionally excluded papers that address other aspects of the lives of individuals with ASD, such as helping them to become more literate, to improve their locomotion skills, etc. In addition, we removed papers that do not contain sufficient details on the technology used in their studies. The search results (raw and screened) are summarized in Figure 2. It is interesting to note that the number of publications in each of the three areas are accelerating over the years.

Based on the publications selected from the filtered Web of Science search results, we further searched Google Scholar to see if the same author(s) have any follow-up studies. If so, these follow-up studies are also included in this review.

The search results include both original research and review articles. We identified ten review papers, two on serious games for children with autism, one on virtual reality based treatment for autism, five for robotic based treatment for autism, and two comprehensive reviews that encompass all technology aspects.

3. Presentation to the Participants

All autism studies use one or more carefully crafted programs to diagnose, treat, or train a participating individual with ASD. The content of such a program is presented to a participant in certain ways, which could be as lightweight as a touch-enabled mobile device, a conventional computer monitor, an immersive virtual reality environment, or via a social robot. Each presentation method has its advantages and shortcomings in terms of cost, convenience, and effectiveness.

The above classification only considers the hardware forms used to deliver the content of a serious game or program to the participants. Except for social robots, the content delivered via all other means of content delivery is virtual in that it is digitally rendered. Even for social robots, some studies used ones that are equipped with a computer monitor for virtual head/face display instead of humanoid physical entity [11]. Furthermore, many studies that employed virtual reality programs used conventional computer monitors/TVs to delivery the content. Hence, the three major categories, i.e., computer and mobile devices, virtual reality systems/devices, and social robots, have certain degree of overlap, as shown in Figure 3.

After introducing the hardware based classification, we further consider user interface design issues in this section, in particular, the primary digital forms used to render the content for the serious games and programs, including images, videos, virtual avatars, non-human objects.

3.1. Computers, Game Consoles and Mobile Devices

Most serious games designed for children with ASD are delivered via conventional computers (such as desktop or laptop computers) or mobile devices [12]. The visual content can be displayed on a laptop screen, a computer monitor, a TV screen, or a touch screen from a mobile device. An advantage of using a mobile device to deliver a game/program to a participant is the convenience [13,14,15,16,17]. A participant could play the game or engage in a program anytime he/she wants and virtually in any posture. Virtually all mobile devices and some newer computers (especially laptop computers) are equipped with touch screens, which enable users to make their choices directly on the screen instead of via a keyboard and mouse [18]. Some of the games designed to run on mobile devices [14] such as iPad, were adapted from traditional assistive tools used for ASD training, such as the cognitive affective training Kit (http://www.cat-kit.com/).

Some serious games involve avatars or a combination of avatars and pictures/videos of real people for various purposes such as the understanding emotions as well as recognition of different emotions via facial expression and body gestures [19]. Another interesting trend in serious games is development of collaborative virtual environment where users can collaborate and communicate during games [15,20,21].

In recent years, motion-based touchless serious games can be played using game consoles (selected from commercial games appropriate for children with ASD) [22,23,24], or via computer systems [25]. Different from traditional serious games, this type of games engage the full body movements of the participants. In addition to traditional learning targets such as social skills and attention skills, this type of games could enhance motor skills of children with ASD.

On the opposite end is a notion of tangible user interface where a user can directly touch physical objects, such as LEGOs. It is touted that the combination of tangible user interface and the computer-based graphical user interface will lead to more effective education/treatment for children with ASD [26].

Design guidelines for serious games for children with ASD were laid out in [25]. Even though it was introduced in the context of motion-based touchless games, they appear to be applicable for all serious games. In [27], Whyte et al. also provided a set of design principles for serious games. The guidelines include the following key points:

• Strong customizability. The game must be customizable to fit the needs and preferences of each child with autism. Unlike many diseases, autism children may have drastically different strengths and skill deficiencies.
• Increasing levels of complexity of game tasks. As a child acquires more skills, progressively more challenging tasks should be made available. This is the case for virtually all serious games. For example, Chua et al. [14] reported the development of an iPad-based game for children with ASD to learn emotion recognition. At the highest level, there are three worlds. There are six difficulty levels within each world. Each scenario contains one scenario depicted by a video of a human actor. At the end of the video, the headshot of the actor with a facial expression appropriate for the scenario is shown and the participant is asked to identify the emotion expressed by the headshot.
• Clear and easy to understand task goals. For each task, there should be a clear goal that can be easily understand by the participant.
• Multiple means of communicating game instructions, such as text, voice, and visual cues. Some children with low functioning ASD would need visual cues in particular.
• Positive reinforcement with rewards. Game score alone might not be enough to motivate children with ASD. Hence, other forms of rewards such as video or audio effect should be provided to encourage and motivate participants. In [13], a smiley face is shown at the completion of each game regardless of the score. This is referred to as reward-based intervention.
• Repeatability and predictability of game play. Unpredictability may cause anxiety to any children with ASD. The repeatability is needed for participants to learn.
• Smooth transitions. The game must be made easy to repeat and easy to transition to a higher level without noticeable delay so that children with ASD are not discouraged.
• Minimalistic graphics and sound/music. All graphical/sound elements must be included for the game goal because children with ASD maybe subject to sensor overload. Even the use of color may play an important role for improved usability for children with ASD [14].
• Dynamic stimuli. Prolong static scene should be avoided to trigger motor rigidity.

3.2. Virtual Reality Systems/Devices

Three-dimensional immersive virtual reality can be realized via sophisticated immersive virtual reality system with surrounding displays, or head-mounted virtual reality devices. Some studies did use such virtual reality systems/devices [28,29,30,31,32,33,34,35,36]. However, many studies used conventional computer monitors to display virtual reality scenes with one or more avatars [37,38,39,40,41,42,43,44,45,46,47]. Hence, there is certain degree of overlap between the virtual reality based presentation mode and the conventional computer based presentation mode. We should note that head-mounted display and virtual reality goggles may cause negative experiences such as dizziness and tiredness for some participants, as reported in [48]. Miller et al. provided an excellent review on the impacts of the level of immersion in virtual reality systems on ASD treatment [49].

In [28], a head-mounted display (eMagin Z800 3DVisor from eMagin Corporation at Bellevue, WA, USA) was used to present a virtual classroom containing multiple avatars to a participant. Different from virtual reality scenes provided by a conventional computer monitor, head-mounted display is capable of rendering the scenes in 3-dimension using stereoscopic images [50]. In addition to providing an immersive virtual reality environment, the head-mounted display can be integrated with head position monitor to detect eye gaze in the virtual visual field.

More recently, researchers started to experiment with using consumer-off-the-shelf virtual reality goggles for autism treatment. In [48], the Oculus Rift head-mounted display was used to study the willingness and acceptance of using the device, and the sense of presence and immersion the participants experienced with the device. Oculus Rift was also used in [51] to create immersive 3D scenes.

The term avatar has been used fairly liberally without a universal definition. In [25], Bartoli et al. divided the avatars used in their study into three categories: (1) articulated avatar, where the body parts are represented using lines and simple shapes, which is similar to a stick figure; (2) pointing avatar, where the body can move along a single direction; (3) real avatar, where the avatar is a silhouette of a real person. In other papers, a virtual avatar is typically referred to as a synthetic human character [52]. Carter et al. [52] further divided the avatar into two types: (1) animated avatar, which has an appearance of a 3-dimensional figure, (2) cartoon avatar, which is 2-dimensional like a cartoon character. Yet some literature calls a picture of a human face as an avatar [53]. Note that an avatar does not necessarily have to be human-like. In some programs, animal avatars were used, such as in the virtual pink dolphins project where avatars were dolphins [54,55,56].

While several different types of avatars, as well as pictures and videos of real persons have been used in many studies, few have compared their impacts to a participant regarding the effectiveness of the social skill learning. Carter et al. reported such a study [52]. In the study, two experiments were conducted. In both experiments, a therapist had a conversation with an individual with ASD. In the first experiment, three presentations of the therapist were used: (1) a realtime video feed of the therapist; (2) an animated avatar that tracks the facial motion of the therapist, and (3) a cartoon avatar that also tracks the facial motion of the therapist limited to 2-dimension. The result shows that there is no apparent variation among participants responses (verbal and nonverbal behaviors) under these three different modes of presentations. The second experiment focused on three different modes of the animated avatar. Exactly because the animated avatar is synthesized based on the therapist’s facial expressions and motions, the avatar can be altered to the way the authors’ desired. In this experiment, three conditions are tested: (1) direct facial motion tracking without any modification; (2) exaggerated facial motion; (3) dampened facial motion. The result shows that the participants exhibited more positive nonverbal responses for exaggerated facial motion, while there was no difference in verbal responses, which supports the authors’ hypothesis that an animated avatar with more exaggerated facial expressions can enhance participant engagement and comprehension.

The virtual avatars used in most studies were produced using 3D animation platforms such as Unity or Vizard. Some studies chose to intentionally use 2D avatars either for comparison or to minimize sensory overload for participants with ASD. Yet some studies chose to use animated photos of real persons as avatars, such as FaceSay [53].

For virtual avatar based serious games and programs, it is essential to produce avatars that can be used to elicit stimuli appropriate for study. Undoubtedly, facial expressions are one of the most important elements in avatar production. In [52], the two-dimensional active appearance models (AAMs) are used to track a therapist’s facial motion. In the first step, AAMs are used to build a set of models based on the appearance and shape of the person’s face. The learned models are used to produce an AAM mesh. This mesh is then used to track the facial motion of the person and to retarget to the avatar face.

In [28,37,39,40,41,57], the Vizard Virtual Reality Toolkit, a commercial product from WorldViz (http://www.worldviz.com/vizard-virtual-reality-software/) was used to produce the avatars in the study. The toolkit contains a PeopleMaker program, which users can use to custom design morphable 3D heads for avatars for different emotion expressions.

In [39], two different experimental paradigms, flashing paradigm and animated paradigm were used. The flashing paradigm involves three scenarios: (1) flashed schematic eyes, where the target event consists of eye balls slightly rotated while the non-target event consists of two eyeballs on the background screen; (2) flashed face with eye position change, where the target event consists of eyes gazing to the face’s right and the non-target event consists of a face facing straight front; (3) flashed face with eye and head position change, where the target event consists of the face facing towards the left while the non-target event consists of a face facing direct forward. The animated paradigm involves with two scenarios: (1) animated 3D body with gaze change in one avatar, where the target event consists of the avatar head turning to the right side, and the non-target event consists of the avatar head turning to the left side; (2) animated avatars with gaze change in four avatars, where the scene contains four different avatars and the target event consists of the head of the avatar at the top row turning to the right while the non-target event consists of one of the avatars turning to the right.

In [40], the Vizard Virtual Reality Toolkit is also used. Instead of using the built-in avatars, they chose avatar heads from a database from the Virtual Human Interaction Lab at Stanford University. These avatar heads were converted from 2D photos of teenagers to 3D models using the 3DMeNow software (the company that created the software has ceased to exist). The authors further customized the complexion of the avatar heads using GIMP (GNU Image Manipulation Program), which is a free open source program. The study developed an emotion model that varies the expression from angry (the negative extreme) to happy (the positive extreme). In addition to face expression variations, an avatar can make gestures based on bone-kinematics, and walk around the scene.

In [28], the Vizard Virtual Reality Toolkit is used to develop a virtual 360-degree classroom that is delivered to a participant via a head-mounted display. Due to the 360-degree virtual environment, a participant would have to turn his/her head up to 60 degrees left or right to see all avatars in the scene. Vizard is also used in [44,58] to create virtual reality scenes.

In [59], the Game Studio A6 rendering engine from Conitec, Germany was used to produce the avatars and the corresponding scenes used in their study. In this study, the participants were asked to identify the emotion of the avatar, which could be happy, sad, surprise, anger, disgust, and fear. A participant was allowed to control the virtual distance between the participant and the avatar via joystick so that he/she could position the avatar to the most comfortable distance for emotion identification. For each emotion, there are four levels of emotional intensity (ranging from vague, neutral, to clear and strong) with corresponding facial expression (at 10, 40, 70, and 100%) and body gesture (such as “clasp hands, raise shoulder, shakes head, arms crossed, look down, step back” [59]). Furthermore, the rendering engine also enabled lip-synch animation to increase the fidelity of the verbal interaction.

Another platform used by researchers for virtual avatar-based scene production is the Unity game engine from Unity Technologies. Different from the Vizard Virtual Reality Toolkit and Game Studio A6, other modeling and animation tools are needed for dynamic facial expression design and animation. In [60], the Maya tool from autodesk was used to animate the characters taken from Mixamo (http://www.mixamo.com). In this study, seven universal facial expressions were used, including “joy, surprise, contempt, sadness, fear, disgust, and angle” (page 1644) [60]. For each emotional expression, four emotion intensity levels were animated (low, medium, high, and extreme), which correspond to four different difficulty levels in facial expression recognition where the lower intention level the more difficult to recognize. Unity is also used in [61] for virtual reality production, where both static and animated avatars are used and are capable of making ten different facial motions, including happiness, anger, disgust, fear, sadness, surprise, pain, neutral, and funny face. Unlike other studies, in [61], a game controller (referred to as gamepad in the paper) was used to control the avatars, control the cursors for answering questions, and receive vibration based tactile feedback.

In addition to facial expression, the gait could also reflect the emotion of a person. Hence, some studies have focused using polygonal walking avatars to produce emotional gait stimuli. Similar to facial expressions, gait can also be morphed into different intensity levels. In [62], three levels of intensity are used, ranging from attenuated (50%), prototypical (100%), and exaggerated (150%). The emotion can be expressed via the flexion of the head and arms, as well as the torso positioning [63]. For example, the happy emotion can be expressed by tilting the head backward and leaning back the torso, and the anger emotion can be expressed by tiling the head forward and leaning forward the torso.

In [64], the design of avatar has included the consideration that individuals with ASD tend to suffer from sensory overload, which refers to a phenomenon where an individual with ASD could choose to block all external stimuli when he/she is presented with too many cues concurrently. Hence, the authors of [64] chose to use a 2D cartoon-like avatar with predictable eye blink and head tilt patterns (the avatar links once every 5 s and tilts its head to the side once every 5 s as well).

In some studies, virtual avatars are placed in specific scenes that is consistent with the story told by the avatars. For example, if a story is about an incident in a fast food restaurant, the scene could be McDonald’s. In [37,40,41], real 2D pictures of social environments were used for building the scenes. In other works, the scene is often designed using virtual 3D objects, such as [28], where a virtual classroom with multiple avatars is used in the study.

Because typically an avatar would tell a story verbally, the voice becomes an essential component in a virtual reality scene. The voices for avatars were recorded and replayed during a study session [37,41]. In some studies, lip sync, which can be facilitated by the Vizard Virtual Reality Toolkit, is used to make the story telling look more natural [40].

Yet some studies used the Second Life platform to create virtual reality scenes [46,65]. This approach could significantly reduce the development time for virtual reality scenes at the cost of flexibility.

3.3. Social Robots

Many studies have employed humanoid robots for treatment plans for children with ASD [66,67,68,69,70,71,72,73,74,75,76,77,78,79]. However, some research has used non-humanoid robots [11,80,81,82,83,84]. To employ humanoid robots in autism studies, one has to overcome two technical challenges: (1) how to control the movements of the robot, and (2) how to obtain the input from the participant. Unlike virtual avatars, where mature technologies exist to enable high fidelity animation of both facial expressions and body gestures, which are essential to facilitate closed-loop autonomous treatment programs, socially assistive robots used for autism studies so far are scripted in a very limited way (such as raising an arm etc.), presumably due to safety and cost concerns. Furthermore, the use of a social robot is meant to facilitate natural user interaction where the participant would engage in verbal conversations and/or use his/her gestures while doing so. In contrast, a virtual avatar based system can equip the participant with a keyboard and mouse, or a touch screen, which allows the participant to provide his/her input/response to a question by typing, clicking, and touching by hand.

There have been efforts towards an autonomous robotic system. Similar to a virtual reality based system, the eye gaze of the participant with respect to the robot can be detected using eye/head tracking devices, such as a remote desktop eye tracker [85] or a head-mounted eye tracking device [86]. Accurate tracking of eye gaze could enable partial autonomous operation of the treatment program via a social robot. Some studies have also enabled some form of speech recognition functionality. It is limited because for higher recognition accuracy, only scripted simple voice commands can be recognized. There is still a long way to go to support unscripted conversations between a robot and a participant. There is also work on using cameras to track facial expressions [87,88] and gestures made by a participant to understand the emotion of the participant and to allow the participant to give commands to the robots for increased degree of interaction between robots and participants [11].

Despite recent progresses made, such as motion and eye tracking [89,90,91], in social robotic research, the Wizard of Oz scheme remains the dominant approach to give an illusion that the robot can capture and understand the participant input/response and react/adapt properly while in fact the expression and motion of the robot is controlled, and the user input is captured by a therapist and/or technician.

Many different types of robots have been employed in this line of research, including both humanoid robots and non-humanoid robots. Some robots are stationary in that they do not move around even though they can move their arms, legs and heads in certain ways. Some other robots are mobile in that they can be instructed to move around or even to follow the participant [11]. Most robots are equipped with cameras (for object recognition), microphones (for voice commands or speech recognition), and speakers. Some are equipped with tactile sensors at the head and head to sense touching.

Humanoid robot Nao from Aldebaran Robotics Company is one of the most popular robots used in autism studies [85,92,93,94,95,96,97,98,99]. Nao is a child-sized robot with plastic body (58 cm tall and 4.3 kg weight). It was used in [92] for joint attention and imitation study, in [85] for joint attention study, in [95] for interventions on repetitive behaviors and affective states, in [96] for interventions on verbal communication skills, and in [97,100] for intervention on imitation and interpersonal synchrony of children with ASD. In [95], an additional mobile robot called Rovio from WowWee was used.

Another humanoid robot Zeno R-30 was used in [101] to facilitate imitation training for children with ASD. Zeno R-30 is also a child-like robot produced by Hanson RoboKind. Zeno must be controlled by a separate computer. Zeno R-30 was used for an imitation study where the robot and the participant take turns imitating each other’s movement facilitated by a Microsoft Kinect sensor, where the operator’s movement was captured and mapped to the robot and similarly the participant’s movement was captured for offline analysis.

A newer version of the Zeno robot called Robokind was used in [102,103] to engage children with ASD and emotion recognition training where the participant was asked to identify the emotion expressed by the robot. This robot has a walking body, which is a significant improvement over the Zeno robot. The robot’s skin is made of a biomimetic polymer called Frubber, which enables it to make a rich facial expressions. Some research used both Nao and Zeno robots, such as [104].

Lucy is also a humanoid robot used in an autism study [105]. Lucy is about 39 centimeters tall and 6.5 kg in weight. It has a baby-like face and it is capable of talking, recognizing faces and tracking faces. It can make limited types of facial expressions by lighting up some LEDs in the face areas, and limited number of gestures. It is also equipped with motion sensors and touch sensors, and can perform some speech recognition. In [105], Lucy was used to play games (Bingo and quizzes) with children with ASD via a tablet. Lucy was programmed to say encouraging words when the participant provided corrected answers in the quizzes (via a touch panel connected to Lucy instead of speech recognition).

Non-humanoid robots have also been used for autism research. For example, Dehkordi et al. [106] developed a parrot-like robot called RoboParrot for screening autistic children. RoboParrot is placed on a fake wooden base inside a cage mounted with a webcam, with a USB cable coming out of its tail and connecting the embedded hardware with the micro-controller board. The hardware is partially embedded inside the body of RoboParrot and partially embedded in the base. Through this system, input data can be collected via the camera on the cage, and the microphone and various sensors embedded in the robot. Specifically, there are two sensors: an infrared sensor, which is in the chest of RoboParrot and detects if someone tries to touch the chest, and a Hall Effect sensor, which is placed in the beak of RoboParrot and senses the closeness of a person’s hand over RoboParrot’s head and break.

In [11], a non-traditional robot was designed for autism study. It is equipped with a computer monitor displaying a virtual avatar face and a depth sensor called Asus Xtion PRO for tracking the gesture of the participant and for recognizing the face of the participant. This non-humanoid robot allows the participant to move it around via gestures, and engage in simple pre-scripted conversations via speed recognition and text to speech software. It is important to note that the usability of the robot was not validated via individuals with ASD.

It is interesting to note that social robots are sometimes used in conjunction with conventional computer systems. For example, in [85], two computer monitors are used (one on each side of a robot) in a program designed to improve the joint attention of individuals with ASD where objects were displayed on the monitors when the robot turns it head towards left or right. Furthermore, robotic studies are not just hardware-focused. There was also an effort to develop custom robots with multi-physiological sensing to automatically observe the emotion of a participant [107].

The need for interoperable software architecture for different brands of robots is also getting researchers’ attention. In [108], efforts were made to design a platform-agnostic robot control architecture that can be used for various treatment scenarios.

4. Input/Reactions from Participants

In response to stimuli introduced to a participant, various reactions from the participant are collected and often analyzed to different degrees either in realtime or offline. Some of these reactions are directly solicited from the participant where he/she are asked to provide explicitly manual input via a keyboard and mouse, a touch screen, or a specific action such as pointing to a particular object on the screen or touching a designated area in a social robot. Other studies use various sensing technologies, such as computer vision or physiological sensing, to capture more complete reactions produced by the participant, such as eye gaze, physiological characteristics, body language. Yet some studies depend on a trained professional to record reactions of the participant. These common means of getting the input and responses from participants are illustrated in Figure 4.

4.1. Keyboard, Mouse, Touch Screen

To collect direct input from a participant, a conventional approach is to use a menu-driven interface or to display all possible options on the screen where a participant would choose the right answer from several response options using the mouse pointer [40,41,59,60,61]. It is also common to require a participant to type a key as the answer, for example, “A” for the anger emotion, and “H” for the happy emotion. It is interesting to note that not only the correctness of the selection or input is important, but in some research, the response time, i.e., the time it takes for a participant to make the selection/input since the prompt of the question is also recorded because this delay can be scientifically significant. For example, in [60], the response time was recorded to validate their hypothesis that it takes longer time for a child with ASD to recognize a facial affect than that for a typical developing child.

With the pervasiveness of mobile devices and laptops equipped with touch screen monitors, users could also provide direct input by touching a particular area on the touch screen. This is often the required input method for serious games delivered on devices with touch screens. In [53], participants are required to perform tasks via a touch screen in three different games, including selecting the objects that the avatar is looking at, selecting the right face band aid that would correct the avatar’s face, select the correct facial expression, and change the expression of an avatar as suggested.

4.2. Facial Motion/Expression Tracking

There are relatively mature computer vision technology to facilitate facial motion tracking, such as eye gaze, blinks, and mouth motions. For example, two-dimensional active appearance models [109] have been used for facial motion tracking in realtime. In some research [110], head tracking is implemented using a Kinect sensor as an approximation of eye gaze for joint attention study. We have also seen research on using facial expressions recognition to obtain affective feedback [30,111,112,113,114,115,116]. In some research, head orientation and movement are tracked using inertial sensors embedded in smart glasses instead of cameras [117]. While the tracking itself helps tremendously in producing realistic virtual avatar faces, there is still an open research issue in how to accurately understand the captured motion with respect to nonverbal communicative behaviors of a participant with the exception of eye gaze (and to some extend facial expressions). Hence, the state of the art practice is still to record the video of the participant and then use domain experts to analyze the video regarding the participant’s nonverbal communicative behaviors [52,118,119,120].

4.3. Eye Gaze Tracking

There are two major types of eye tracking technologies. The first one is head-mounted and the other is referred to as desktop based without requiring a participant to wear any device for eye tracking on his/her body. The core technology for eye tracking is identical to head-mounted and desktop eye trackers. An infrared camera is used to detect the rotations of the eyeballs (based on the pupil position and the corneal reflection). The reason why infrared light is used for eye movement tracking is that it renders a better demarcation of the pupil and the iris than visible light. Once the eye movements are determined, the gaze can be computed based on the context.

In [37,41,121], an eye-tracker goggles from Arrington Research Inc. was used to track eye gaze in realtime. The goggles are used in conjunction with a custom designed eye tracking interface platform. The raw eye gaze data were sampled 30 times a second. Then, three features were extracted from the raw data, including the mean pupil diameter, the mean blink rate, and the average fixation duration for each region of interest.

A head-mounted device called WearCam has been used to compute eye gaze offline. This device records the eyes of the participant and the image of the field of view in front of the participant simultaneously [122]. Based on the recorded data, the direction of gaze and focus of attention can be computed. A downside of WearCam is that it requires offline analysis of the recorded video to determine gaze. Hence, this device cannot be used if eye gaze needs to be determined in realtime.

Eye gaze tracking can be made simpler if used in conjunction with head-mounted virtual reality devices, which produces a 360-degree immersive virtual 3D scene. In this case, it is sufficient to track the head movement, which is used in [28]. The head position monitor is integrated with the head-mounted display and it is used to measure the yaw (left/right), pitch (up/down), and tilt movements of the head. From these measurements, the eye gaze direction with respect to the virtual scene can be computed accurately. In [28], the social attention pattern of children with ASD is studied entirely using eye gaze as the sole input from the participants with a head-amounted eye tracker.

Because desktop based eye-trackers are fixated to the monitor, eye tracking alone is not sufficient to accurately determine the gaze direction with respect to the computer monitor. Hence, desktop eye trackers typically integrate with head tracking based on face recognition to compensate for head movement, such as the Eye-Trac 6 desktop eye tracker from Applied Science Laboratories, which is used in [123] in their study.

In [56,60], a similar desktop eye tracking device, called Tobii X120, was used in their study. The device tracks at 120 Hz and allows free head movements within the range of 30 cm by 22 cm by 30 cm (for width, height and depth) and it requires the participant to sit at a distance of 70 cm. The technical specification for Tobii X120 does not state how the head orientation is determined. The authors of [60] designed and implemented an eye tracking application to be used in conjunction with the virtual reality application. The two applications are connected via a high-speed network such that they can determine which and if the participant fixated on one of the regions of interest of the avatar, including forehead, left/right eye, nose, mouth, other face areas, and non-face regions. The same eye tracking device was used in a study with social robots [85]. However, because of the different context where the study wanted to know whether or not the participant was looking at the robot instead of a virtual avatar displayed in a computer monitor, a technician was needed to manually observe the eye gaze and constantly calibrate the system. Automated eye gaze is possible even in this context. However, it would require the participant to wear a head tracker, which was proved to be too uncomfortable for many participants (40% of participants did not want to use it) [124,125].

In [126], another desktop mounted EyeLink 1000 Remote eye tracking system was used in their study. In [127], a remote infrared camera (model D6-HS Remote from Applied Science Laboratories) was used to track the participant’s eye gaze.

Zheng et al. reported a system that is capable of automatically detecting head orientation and eye gaze using multiple cameras [128,129,130]. Based on the detected eye gaze (and engagement level), the system adapts its program content in realtime.

We should note that the trustworthiness of the collected gaze data has rarely been examined carefully with the exception of [131], where the authors studied the raw log files and determined that the quality of recognition is sub-optimal. Future research should pay more attention to the data quality issue, not only for eye gaze data, but all data collected to improve the effectiveness of treatment and to protect the safety of the participants if such data are used in a closed-loop fashion to control the flow of the treatment program.

4.4. Human Motion Tracking via Depth Sensors

Microsoft Kinect and similar depth sensing devices such as Asus Xtion PRO enabled a new way of capturing the gestures as well as facial expressions of the participant [132,133].

In [11], a participant can move the robot to the left or right via the left or right wave gestures. The OpenNI and NITE software packages were used to capture the human motion and to recognize the gestures. In [101], Microsoft Kinect sensor was used to track the participant’s movement to determine how closely the participant imitated the pre-scripted and tele-operated movement of the robot. Metrics used include the range of motion of join angles, response delay, and the distance between two time series representing the movement of the robot and that of the participant.

For studies with social robots, depth cameras are often used to track the presence and the activities of the participant for offline analysis regarding the similarity of the imitation and the original actions performed by the robot, as done in [92,101]. In [92], the pose of the participant and the libs were first identified using the depth frames, and subsequently the head region in the corresponding color image frames were cropped. From the cropped face, the pose of the head was determined using a computer vision algorithm. From the head pose data extracted from the logged data during the experiment, the joint attention behaviors of children with ASD and typical developing children are compared. The imitation score was determined by the fraction of postured successfully imitated by the participant.

Enabled by inexpensive depth sensing devices such as Microsoft Kinect, it becomes possible to engage children with ASD with motion-based touchless serious games [22,23,25,134]. Unlike other serious games, a participant engages in the game play by moving his enter body. The participant is being continuously tracked by the depth sensor and an avatar representing the participant becomes part of the game displayed. These types of games not only could be more engaging, but also may enhance the motor skills of the participants too.

Zhang et al. reported their work on using Kinect to track a participant’s gesture and assess the quality of the imitation based a set of predefined rules [135]. Feedback is provided in realtime to the participant. Ge et al. [136] used Kinect to track the engagement level of the participant. Yun et al. also used Kinect to track the activity of the participant when he/she is interacting with a social robot [137]. Kinect has also been used to track the movement patterns of the participations [29] and for self-modeling intervention for children with ASD [138].

4.5. Speech Recognition

In some studies, speech recognition is employed to enable a participant to answer questions or engage in a program via predefined voice commands. At this stage, we have yet to see work that has incorporated unscripted conversations.

In [11], a participant can interact with a robot by a number of predefined voice commands, such as move left/right, follow me, stop, and goodbye. The voice commands were captured using the Asus Xtion PRO device, which is equipped with two microphones. The Microsoft Speech Platform Software Development Kit was used to perform speech recognition. In [105], the robot Lucy is said to have speech recognition capability. However, the paper did not provide any technical detail and the use scenarios of the capability in their study.

4.6. Physiological Data Collection

It is well known that physiological characteristics reflect the emotion and attention of a person. Hence, it may be desirable to obtain such information during a session of treatment to gain better insight to the state of an individual with ASD. We have seen more autism research that employs various sensors to collected physiological signals, including Pulse Plethysmogram (PPG), Skin Temperature (ST), Electrodermal Activity (EDA), Electrocardiographic (ECG), and Electroencephalography (EEG) [39,40,51,139,140]. Typically, the classification of such signals was done offline. Hence, the classification accuracy does not impact the flow of the program and cannot cause harm to the participants. Nevertheless, in the long run, obtaining responses through EEG has the potential to develop brain-computer interfaces to improve social skills of individuals with ASD [39]. EEG data were also collected in [141,142,143,144]. In [145], Ozcan et al. reported a design for wearable companion toys in which biosensors can be embedded to measure the participant’s physiological and emotional state. In [146], physiological data, including PPG, Galvanic Skin Response (GSR), and Respiration (RSP), are collected while a participant with ASD engages in a driving task. Note that in some studies, multimodal data are collected [143]. In addition to physiological data, eye gaze and spoken conversations are also collected [143] to enable adaptive social interaction.

4.7. Video/Voice Recording

Video coding is the standard method for analysis in behavioral research [147,148]. Many autism studies that have been published in clinical journals included video coding, such as [22,38,64,102].

5. Program Customizability and Adaptability

Some programs, especially serious games, may allow users to customize their user interface, including the avatar, if it is used, game scene, as well as difficulty level. In [61], a user is asked to customize the avatar and the virtual environment of his/her choice. The user is also asked to calibrate the tactile feedback of the game controller he/she will be using in the program because the authors anticipated that different users would have different sensitivity preferences. Furthermore, different methods were provided in [61] for a user to indicate the specific emotion they recognized, including a visual non-verbal code where each emotion to be recognized was represented by a particular color/icon (happiness was represented by yellow, angle was represented by red, disgust was represented purple, fear was represented by green, sadness was represented by light blue, surprise was indicated by dark blue, pain was represented by black, neutral was represented by white, and funny face was represented by a trash can), emotional words that told the emotion as it was, and idiomatic expressions, which were short sentences containing a figurative meaning, e.g.,an idiomatic expression for fear could be “to get cold feet”. In [25], all three motion-based games were designed for customizability. A user could decide on appropriate game speed, object density, and which body parts to enable for game playing.

Different from program customization, it might be desirable for a program to dynamically adjust its flow of control, task difficulty level, or graphic content based on how a user responded to deliver a personalized program for each user [149]. Of course, this does not apply to all programs because some studies were designed to establish patterns of behaviors of individuals with ASD. In these studies, obviously there is no need to adapt the program flow and content based on the input from the participants. Jarrold et al. [28] studied social attention patterns of high functioning children with ASD in a virtual public speaking task based on eye gaze.

Many treatment programs are not adaptive in that the flow of the programs are not changed dynamically based on the input collected while being administered. For mobile games [14], a participant must provide the correct answer to each question before the participant can complete the current level, and only after completing the current level, can the participant continue on to the next level. Such games do not consider how many times a participant has failed a level.

An adaptive program requires the processing of input data from participants in real time. This can be done either automatically using technology and computer algorithms, or by a therapist manually.

5.1. Automatic Adaption of Programs

In [40], the difficulty level is adjusted based on whether or not the participant answered a question correctly via a menu-driven interface. The program has four levels of task difficulty. The difficulty level is reflected from the questions asked for each story. If the answer from the participant is deemed adequate, the next question will be elevated to a higher level unless the current level is already the highest. Mouring and Tang [150] reported an adaptive training for adolescents with ASD using neural network to monitor participant performance and a learning fuzzy inference system to determine deficits to instructional techniques.

Eye tracking is one of the few relatively mature technologies that has been used to retrieve nonverbal communication of a participant in realtime. Some studies used eye gaze information to alter the scene displayed for a participant. In [123], Courgeon et al. studied the joint attention pattern using eye tracking and virtual avatars. The virtual display blurs all the scene except a small rectangular region centered on the point of gaze of the participant. This is done so to provide visual feedback on what the participant is looking at, which enhances self-awareness of gaze.

In [28], the program included a cued condition where an avatar would fade to 70% transparency in six seconds if the participant failed to fixate his/her eyes on the particular avatar when prompted. The entire cued condition is fully automated enabled by the virtual reality program and gaze tracking via a head-mounted display and sensors. The avatar would become opaque again when the participant looked at it again.

An individualized adaptive response module constitutes a core component in the virtual reality system used in [37,41]. The module consists of two adaptive mechanisms, namely, the performance-sensitive system (PS), and the engagement-sensitive system (ES). In PS, the task difficulty is adjusted solely based on the input provided by the participant via the menu-based conversation interface. In ES, the task difficulty is adjusted based on both the input provided by the participant and the engagement level as revealed by the three features obtained from eye gaze tracking data, i.e., the mean pupil diameter, the mean blink rate, and the average fixation duration for each region of interest. For PS, the task difficulty is adjusted based on the performance of the previous task. The performance is determined by the points earned on a task. If the participant earns 70% or more the maximum score possible, then the performance is regarded as adequate. Otherwise, it is regarded as inadequate. If the performance on a task is deemed adequate, then the difficulty level of the next task is elevated to the next higher level. Otherwise, it is switched to a lower difficulty level. For ES, the rule for adjusting difficulty levels is more complicated because it must fuse both the performance from participant direct input (categorized as adequate and inadequate), and the performance from eye gaze analysis, which is categorized into “good enough” and “not good enough”.

It is encouraging to see that this line of research is gaining momentum. In addition to eye gaze, we have seen studies that employ various methods to obtain user feedback for adaptive control of a treatment/education program, including facial expression, head orientation, human gesture tracking, and physiological data collection [135,136,141,143,146,151].

5.2. Manual Control

An adaptive program requires realtime analysis of the responses of a participant during a program in one or more aspects and decides on the most appropriate next step autonomously. To a large degree, there is still a gap towards this goal. Hence, a psychologist or a trained clinician is often required to provide their subjectively evaluation based on their observation during the execution of a program. Sometimes, the professional is asked to not only evaluate the response of the participant, but also to control the flow of the program as well, which is true especially for social robotic based studies, where a social robot is controlled manually by the professional based on his/her judgment. This scheme is often referred to as Wizard of Oz scheme [3,152]. The professional evaluation can also be used to assess the effectiveness of the program.

In Carter et al.’s study of interactive avatars [38], the system output was verbal utterances by a human actor that tele-operated the avatar to talk on the screen. The avatar tried to elicit responses (verbal statements and gestures) from the participants and occasionally called on those who raised their hands (and their parents) to engage in one-on-one conversation. In [64], a carton-like avatar was controlled by an actual person. The motivation for doing so is to gain more flexibility during the interactive session because the current state-of-the-art technology is still incapable of handling spontaneous conversations.

In [85], a technician was used to monitor the eye gaze of the participant both for the purpose of calibration of the eye tracking system and for identifying when the participant looked at the robot and the object as directed by the robot. When the participant performed as instructed within 7 s, the technician also triggerred a reward for the participant by pushing a button in the system. Otherwise, a miss event was recorded and the next task was administered.

In Dehkordi et al.’s study [106], the system output consists of simple motions of the robot parrot following the commands of the operator, as well as voice from the speaker uttered by the operator. This involves the use of body motor to control the motions of the wings, the legs, and the neck, and the use of head motor to control the motions of the eyes and the beak. The operator (ideally a therapist) can use the camera on the cage to check the validity of the execution of commands sent to the robot parrot, use the speaker and microphone to verbally communicate with the ASD child, and adjust both the digital commands and verbal output based on what the child says or does.

In [95], the robots were also controlled by a trainer using a laptop running a custom program. The trainer is responsible for deciding when to greet a participant, warming up a game with body stretching activities, playing various games, and deciding when to say goodbye. The same scheme was used in [96].

In motion-based touchless serious games [22,25], a therapist was also tasked to observe the behaviors of participant and to intervene when necessary.

6. Program Evaluation

In addition to program adaptivity, objective evaluation of a program would also need to analyze the input data collected during a study. There are different types of program evaluations. For a treatment program, the evaluation focuses on its impact on the participants, which typically relies on traditional clinical-proven standard tests, but maybe assessed via technology-based methods as well. For a program aimed to diagnose autism, or a program aimed to establish patterns on social behaviors of individuals with ASD, the purpose of program evaluation is to establish the reliability of the patterns detected or diagnosis [60]. Yet some studies focus on the evaluation of effectiveness of different ways of delivering a treatment plan to individuals with ASD such as [38]. Many pilot studies simply validated a predefined set of hypotheses using their programs [28,38,41,85]. In all studies, the usability of the programs used could also be evaluated [61]. A special case for usability study is testing the acceptability of a new way of delivering content to individuals with ASD, such as [48,153], where a consumer-grade head-mounted virtual reality goggle was experimented.

6.1. Behavioral Pattern Assessment

In [60], the authors studied the patterns of facial emotion recognition and related characteristics for both children with ASD and typical developing children using a virtual avatar based program and eye tracking. They used a number of metrics, including the recognition accuracy of facial emotions, the response time and the confidence level of making the recognition. The authors found that it takes longer for children with ASD to make a recognition with lower confidence level than that of typical developing children. Furthermore, children with ASD have more variation in eye gaze patterns than those of typical developing children. However, the emotion recognition rate is rather similar between the two groups.

6.2. Evaluation of Treatment Effectiveness

In [53], emotion recognition skills of the participants were assessed using six Ekman and Friesen photographs [154] and six drawings of facial expressions, facial recognition skills were evaluated using the Benton Facial Recognition Test [155], social interaction skills in natural environments were rated by the Social Skills Rating System [156] and social skills observation. To validate that a treatment program is effective, these tests are typically administered before and after the treatment sessions and the results are compared.

In a sequence of studies [95,96,97], Srinivasan et al. presented a systematic evaluation of three different ten-week intervention programs with pretest and posttest using standardized clinical tests. In the first study [95], the repetitive maladaptive behaviors as well as the affective states of participants were coded by clinicians and then compared using statistical analysis. In the second study [96], a standardized test of joint attention was used to determine the verbal communication skills of the participants before and after the ten-week intervention program. In the third study [97], a standardized test of motor performance called BOT-2 was used in the pretest and posttest to access the effectiveness of the intervention program on imitation, interpersonal synchrony, and motor performance of children with ASD. In all these studies, the authors observed that there were no advantages of using social robots compared with rhythm based training.

In [14], a clinician was asked to evaluate the following aspects of the game: (1) how well the game was able to attract the attention of the participant; (2) how well the participant was able to carry out tasks on their own; (3) the degree of effort the participant had made to find the right answer; (4) how well the participant showed signs of anticipation; (5) how well the participant was able to wait patiently to transition from one level to another; and (6) the degree of self-esteem had shown by the participant during the game with respect to the confidence displayed when making successful attempts and despite failures.

In [61], the effectiveness of the treatment program using JeStiMulE was evaluated by comparing the results of the facial emotion recognition accuracy obtained before the four-week long treatment program and after using both the avatars employed in JeStiMulE and the pictures of real-life characters. They observed statistically significant improvement of the treatment plan.

To assess the effectiveness of using motion-based touchless serious games to improve autistic children’s learning, two variables, namely selective attention ( i.e., how well a participant could focus on an important stimulus in the presence of distractions) and sustained attention ( i.e., how well a participant could hold his/her attention to complete a task), were measured and compared using the Bell Test at the beginning, during, and seven days after the end of the treatment [22]. Data for the two variables were extracted from the recorded videos taken during the game plays. The researchers noticed an increase of the two variables for all children. In a follow-up study [25], assessment was made at the beginning and the end of treatment period. In addition to select attention and sustained attention, three additional variables were used, including visual perception ( i.e., how well a participant could process visual information), motor coordination ( i.e., how well a participant could move body parts coherently), and visuo-motor integration ( i.e., how well a participant could control his/her body movement when prompted visually).

In some studies, both parameters collected while a participant was playing a serious game (or a sequence of games) and standardized tests were used to assess the effectiveness of a treatment plan. The data were collected at the beginning and at the end of the six-month treatment plan. In [13], the game-associated parameters used for assessment include the game identifier and level of difficulty, when the game was played, the game score, and the reaction time for a participant to complete a task. The clinical measures include the Autism Diagnosis Observation Schedule, the Vineland Adaptive Behavior Scale II, Weschsler scales, the Child Behavior Checklist, and the Social Communication Questionnaire, and the Parenting Stress Index. The researchers found that computer-based serious games benefited children with ASD. In particular, they found that 4 out of 6 imitation games significantly improved the participants’ imitation scores, and 3 out of 4 joint attention games shortened the time it took for participants to complete the task over multiple sessions.

In [19], the Social Responsiveness Scale, 2nd Edition (SRS-2) standardized test was used in pretest and posttest to determine the effectiveness of a serious game called emotiplay.

6.3. Usability Assessment

In [61], the usability of the program called JeStiMulE was evaluated in terms of its adaptability, effectiveness and efficiency with a group of individuals with ASD. For adaptability, the authors observed that all participants managed to use the gamepad to control the avatar, 91% of participants understood the association between facial emotion and the corresponding non-verbal code, and different participants chose their own favorite ways of indicating the facial emotion that they have identified. The effectiveness is determined by the percentage of participants who managed to complete different modules in JeStiMulE. 73% of participants were able to complete all modules. The efficiency is determined by how long the participants were able to complete each module among those who have completed all modules. On average, it takes 49 minutes for a participant to complete each module.

In [48], the acceptability and immersiveness of the consumer-grade head-mounted goggle, Oculus Rift, among individuals with ASD were examined via a two-phase study. The acceptability of the device is determined by the fraction of participants who managed to complete the study. Out of 29 participants, 25 completed the phase I study. 23 participants proceeded on to complete the second phase of the study. For the immsersiveness of the device, the authors chose to use the Independent Television Commission-Sense of Presence Inventory (ITC-SoPI) to evaluate the subjective experiences by participants who have completed the study. ITC-SoPI evaluates user experiences in the following four categories: (1) spatial presence, which is about how a participant feels he/she is really present in the physical environment; (2) user engagement, which is about how involved a participant is in a virtual reality program and how intense it is; (3) ecological validity, which is about how natural a participant feels about the experience; and (4) negative effects, which are about whether or not a participant feels any dizziness, headache, etc. This study also briefly analyzed participant behavior while using the device by first video taping the sessions and then analyzing the recorded video qualitatively manually.

7. Discussion

As we have seen in the reviewed literature, there are two primary means of eliciting proper reactions from individuals with ASD, one using virtual reality technologies, and the other using social robotic technologies. The former typically relies on one or more avatars to serve as an agent to communicate with a participant, while the latter uses a robot. Virtual reality has two advantages over robotics: (1) both the avatar and the virtual scenes can be changed dynamically that fit the treatment scenarios, while a robot cannot easily change its appearance and the treatment environments cannot be changed easily; (2) the virtual reality based systems can be much cheaper than robots. Hence, unless the cost of robots is reduced drastically, virtually reality based systems could be more attractive to home-based uses.

That said, robotic based solutions provide physical embodiment, which is not possible for virtual reality based systems. This advantage may lead to better generalization of skills learned in treatment sessions to the real-world scenarios, although this speculation has yet to be proven in clinical trials.

For virtual reality based systems, even though it is desirable to render 3D views to a participant, which could potentially increase the treatment effectiveness and transferability to real-world skills, the use of head-mounted displays or virtual reality goggles might not be suitable for all individuals with ASD because some are more prone to cyber sickness. This concern is also applicable for wearable devices that take physiological data from a participant.

Another observation is that the current autism studies have predominately focused on treatment programs that target a specific set of social skills. Although temporary improvements in social skills for individuals with ASD have been observed, it is doubtful such improvements can last and can be generalized. More importantly, such training programs do not address the psychological and emotion issues for individuals (especial for children and adolescents) with ASD. What is urgently needed for an individual with ASD is a life coach who he/she trusts, who shares the same interest with him/her with, who knows his/her emotions, and who is never upset no matter how badly he/she behaves. A recently granted patent describes a system that partially fulfill this goal [157]. This system provides an avatar-based mobile interface for an autism child to bound with the avatar character and to learn various lessons on social and emotional regulation skills. The system relies on the use of one or more therapist or parent to interact with an autism child. Essentially, the system presents an illusion that the child with ASD is interacting with an intelligent avatar who he/she feels comfortable interacting with while in reality he/she is communicating with an actual human being who he/she otherwise would be reluctant to communicate with. To help the participating parent or therapist, the system also provides carefully crafted prompts to guide him/her on how to properly interact with the child with ASD. We are currently also working on a system that aims to serve as a virtual life coach for children with ASD [158].

Several publications have reported so called closed-loop autonomous systems based on social robots or virtual avatars [135]. Unfortunately, such systems have achieved autonomy only for very specific scenarios. For example, in [135], the system is capable of assessing a very small set of imitation gestures made by a participant in realtime using predefined rules. This is far from the degree of autonomy needed to train children with ASD towards improved practical social skills. In [159], an ambitious design towards a virtual avatar that can autonomously respond to the participant’s emotional reactions and action tendencies was presented. Unfortunately, the authors conceded that it is exceedingly difficult to model the participant accurately based on the limited information collected, including the frequency of touch (on the screen) and eye tracking. In the end, they supplemented their system with a Wizard of Oz control panel for a therapist to intervene when needed.

A much more viable approach is proposed by Esteban et al. where the design goal should be a supervised autonomous system [160]. According to this design, autonomy can be achieved for specific tasks, such as to determine where a social robot (or a virtual avatar) looks at next. With supervised autonomy, a therapist is needed to provide high level guidance and if necessary, to override the decision made autonomously. To accomplish this goal, the system would need to track several primary and secondary variables. The primary variables include the performance of the participant who performs the designated tasks. The secondary variables include: (1) the level of social engagement reflected by eye-contact and verbal utterances; (2) the emotional level (positive or negative); (3) the behavioral level (stereotypical, adaptive, and maladaptive); (4) the cognitive level (rational or irrational). In addition to tracking the participant, the system proposed by Esteban et al. also tracks the behavior of the social robot for maximum safety of the participant [160]. In this system, three regular cameras and three Kinect sensors are used (one of the Kinect sensor is used to self-monitor the social robot).

With the supervised autonomous system design, we believe machine learning and artificial intelligence [161,162,163,164,165], in conjunction with multimodal sensing, will play much more important roles in the next generation autism diagnosis and treatment programs. Considering that good therapists are always in very high demand for the autism population, we envisage that a therapist could operate multiple supervised autonomous systems remotely at different sites, which could benefits more children with ASD while reducing the cost of autism care.

8. Conclusions

In this article, we reviewed the autism research facilitated by various technologies from an engineering perspective. We primarily focused on research results produced in the last five years (2012–2017). As can be seen, while significant progress has been made, there are major challenges to overcome in future research. First, other than manual input via keyboard/mouse/touch-screen, the only mature technology for automatically capturing participant responses is eye gaze. To assess how well a participant responded to an intervention program, current research predominately relied on video tapping and offline analysis of the recorded videos. Even though there have been efforts to use various motion/facial tracking and physiological tracking devices to capture the emotional state of the participant, such efforts are still in preliminary stages. Second, few autism treatment programs are customizable and automatically adaptive. This is obviously a limitation for autism studies because individuals with ASD are known to be highly different in their deficiencies, skill levels and preferences. To overcome this limitation, some studies relied on the use of therapists to operate the virtual avatar or social robot in realtime. However, this approach is not only not conducive for reducing the cost of autism treatment using technology, but also has negative impact on the repeatability of experiments. Furthermore, the quality of the program control also heavily depends on the training and expertise of the therapist. Hence, the solution is not scalable. The resolving of the first issue will open the door for designing and implementing fully autonomous virtual avatars and social robots that help individuals with ASD.

Finally, we provide our observation on how technological limitations might be affecting ASD therapies. First, there lacks a highly reliable and accurate sensing platform that can capture and recognize the affective state of the participant. As pointed in [160], this is the prerequisite to achieving autonomous control of a therapy program that is personalized for each individual and is capable of dynamically adapting the program content based on the affective state of the individual. Many current studies failed to consider the affective state of the participants completely. Even for those that do, the technologies employed are ad-hoc, are often without establishing the ground-truth, and are not processed in realtime.

Second, the therapy program content is typically designed one-size-for-all. It is well known that children with ASD typically have intense interests in certain topics. The therapy programs could be much more effective to an individual if the program content aligns well with the individual’s intense interests. Getting to know which participant to have what intense interests would go beyond technology. Ideally, for each child with ASD, an individualized educational placement (IEP) team would be needed to provide intimate knowledge about the child so that the content of the therapy program can be customized for the individual [158].

Third, there lack longitudinal studies on the long-term effectiveness of technology-based therapies. While it is accepted that children with ASD are generally more receptive to technology, for example, they show less anxiety when interacting with virtual avatars or social robots, it is unclear such technologies can truly help children with ASD to gain social skills and emotional regulation skills in practice in the long-run when they have to directly interact with real human beings day-in and day-out.