1. Introduction
The use of social robots in the treatment of children with Autism Spectrum Disorder (ASD) has been a topic of research for years because it is one of the most promising applications for these robots [
1,
2]. Moreover, the therapies with robots may deliver improvements that are quite similar to or even better than therapies provided by human therapists [
3,
4].
The main argument for the use of robots in behavioral therapies for children with ASD is the ability of these robots to stimulate engagement, and thus motivate children to take part in therapy sessions, as reported in [
1,
5]. Additionally, the children with ASD would eagerly focus their attention on the given task, which contributes to the effectiveness of the treatment for these children [
1,
5,
6]. However, most research on engagement and attention in interactions with robots reports results of interactions from single sessions to up to eight sessions [
5,
7]. We consider these amounts of interactions with the robot as short interactions since the treatment provided by human trainers, such as Applied Behavioral Analysis (ABA) based Pivotal Response Treatment, (PRT) can take much longer.
Several studies have investigated whether the attention and the engagement in long-term child–robot interaction are sustained [
8,
9]. Both studies [
8,
9] found a decrease in attention or engagement over time. Another research proposed strategies to prolong the engagement of children with ASD by conducting therapies through games and developing game goals that could be achieved only through several follow-up interactions [
9]. In this research, the children had to do a different task each time, but did not have a choice for the game, and were not personalized to the individual child. Yet another study personalizes the interaction by adapting the robot behavior to the child’s emotional state and making the robot remember the names and performance of the children on tasks [
9], i.e., continuous adaptation. Differently, in the current study the focus was on the personalization and appropriateness of the games between a robot and children with ASD. The designed game interactions followed the individual development of each child and involved the therapy-specific PRT motivational techniques, such as respecting the child’s choice for a game, providing a clear opportunity for a child to respond, personalizing the scenario by including learning opportunities for both maintained (easy) and new (difficult) tasks for each child, and providing direct and natural reinforcement to the child, which is further elaborated in [
10].
The current study is a part of a clinical study with 73 children with ASD in a three-armed Randomized Controlled Trial (RCT) in which Pivotal Response Treatment (PRT) (N25) and robot-assisted PRT (PRT+ robot) (N25) were compared with treatment as usual (N23) with children between the ages of three and eight over 6 months. The main targeted areas within this study were the social development of the children before the treatment, at twenty weeks onset of the therapy, and after thirty weeks (follow-up) of the treatment [
4]. A special effort was invested in the design of the games to comply with the individual development of each child and to involve the therapy-specific PRT motivational techniques [
10]. The main experiment, reported in [
4,
11] found that children, who were in the robot-assisted PRT group outperformed the other two groups in terms of social responsiveness as observed by a blinded psychiatrist. The collected data provided the opportunity to look into the child–robot interactions throughout 20 sessions, conducted over six months. Specifically, we analyzed the change in the attention and the engagement of these children over time. Therefore, the analysis presented in this paper aims to determine whether the interest of the children in the robot condition decreases over time, and not to determine the impact of the robot on children’s behavior which was already established following clinically validated measurement instruments [
11].
We hypothesize that there will be a decrease in both the attention and the engagement of the child during interactions with the robot throughout the therapy. Even though a robot has the potential to capture the attention and the engagement of a child with ASD [
1], we expect that a novelty effect is present, i.e., the initial increase in interest when the new technology is introduced, and will fade away after some time [
12]. We base this hypothesis on previous research that suggests that there is a decrease in attention [
12] and engagement [
8] in child–robot interactions over time.
The outcomes of this research are expected to provide guidelines for the further improvement of robot-assisted therapy for children with ASD. Particularly, how robots can be used in long-term interactions without becoming boring to the children and requiring an unrealistic amount of content creation.
The paper is organized as follows:
Section 2 provides brief background information on autism and treatment and defines attention and engagement. The methodology will be discussed in
Section 3. This section involves details about the experimental design and the used observation scheme.
Section 4 will show the results of the experiments. In conclusion,
Section 5 will discuss the findings of the work, followed by the conclusion from the conducted research.
3. Methodology
3.1. Participants
A total of twenty-five children took part in the robot-assisted PRT, as explained in more detail in [
11]. This group was part of a larger randomized clinical trial aiming to establish the effectiveness of PRT for children with ASD. Inclusion criteria were: (1) a clinical diagnosis of ASD, (2) meeting criteria for ASD based on DSM-IV, (3) aged 3–8 years, (4) a total intelligence quotient (TIQ) of above 70 measured using either the Wechsler Intelligence Scale for Children (WISC-III) [
33], Wechsler Preschool and Primary Scale of Intelligence (WPSSI-III) [
34] or Mullen Scales of Early Learning (MSEL) [
35], (5) ability to speak with one-word sentences at minimum, and (6) at least one of the parents speaks Dutch to the child. Six out of these twenty-five children were selected for this analysis based on (7) the availability of the recorded data at the test moments during the therapy (twelve out of twenty-five children met this criterion) and, the (8) distribution of gender to be the same as in the distribution among the original sample (original distribution was 20 boys and 4 girls). The selected participants for this study were five boys and one girl with a mean age of 5.17 (SD = 1.47) and a mean TIQ of 94.20 (SD = 11.8).
3.2. Procedures
In the RCT, the participants’ parents received verbal and extensive written information on the outline and aims of the study and both parents signed an informed consent form prior to inclusion. Baseline measures were performed if these had not been used as part of the diagnostic procedure. Thereafter, participants were randomly assigned to the robot-assisted PRT, PRT, or treatment as usual by the principal investigator of the study who was not familiar with the participants. Before each intervention, parents received psycho-education on ASD if they had not received such in the past. This was conducted either individually or within a group. Robot-assisted PRT consisted of 20 sessions of therapy, once a week, by a certified PRT therapist trained up until level III for this treatment. The study received ethical approval by the Dutch Research Ethical Committee (CMO Arnhem-Nijmegen, NL50509.091.14) and all procedures were in accordance with the 1964 Helsinki declaration and its later amendments.
3.3. Setting
The study took place at Karakter, an academic center for child and adolescent psychiatry. The robot-assisted PRT was provided in regular treatment rooms. The robot was placed on a rectangular table and controlled by the therapist, as shown in
Figure 1. All sessions were recorded using a video camera that was placed in front of or beside the child. Parents were also involved in the sessions, but not always visible in the videos due to the positioning of the camera.
3.4. Design and Materials
3.4.1. Game Flow
The gameplay interactions of children with ASD during robot-assisted PRT lasted between 15 to 20 min per session. The NAO robot, a small humanoid robot distributed by SoftBank Robotics [
36] was used in this experiment, and it was controlled with TiViPE software [
37] to ensure that the robot behaviors could be programmed and fine-tuned by the researchers from Karakter Child and Adolescent Psychiatry University Centre, and thus increase their clinical validity. Additionally, the TiViPE software allows the therapist to provide real-time prompts for the targeted behavior using an interface that is connected to the ongoing (preprogrammed) scenario.
The robot followed game-based interactive scenarios that used teleoperation in many instances to redirect the interaction or give an answer to unexpected questions. In the games, different PRT elements were incorporated: the child selecting the game, providing clear learning opportunities for the child to respond to, interspersing skill maintenance tasks with new tasks, direct and natural reinforcement, and the reinforcement of attempts. Most of these techniques are used to increase the motivation of the child in the therapy, enhance their effort into showing target behavior, and couple the child’s behavior to its consequences [
21]. The child could choose different kinds of games: card games, building with LEGO, or making a puzzle, with three variations of each game, and seven levels of each variation. For the card games, the child could choose between different themed quartet-games (sea animals, fairy-tale characters, and cards with Frog and Friends). The older children (aged 5 or above) could choose between three different LEGO games (a plane, a car, or a house). Younger children (aged three to five) had a choice between three different LEGO DUPLO buildings (a plane, ducks, or a house). For the puzzle game, the child could select one of three puzzles (animals, a magnetic boat, or trucks). The level of the game that was used in the therapy session was selected by the therapist based on the child’s target behavior. These behaviors are assessed using two- or three-word sentences, wh-questions (what? where? who? which?), multiple cues, asking for an object/activity, asking for help, protesting, and (early) conversational skills.
3.4.2. The Robot Behavior
The behavior of the NAO robot consists of actions and speech that occur simultaneously. The robot speech was used to: (1) communicate with the child about the game or other non-game-related subjects, and (2) elicit targeted behavior of the child (e.g., by saying: “I know how to play this game” expecting that the child will propose to play the game together), (3) reward the child after performing/attempting the target behavior (e.g., when the child asks a question, the robot directly responds), (4) keep the child focused on the tasks, and (5) prompt the child.
There were four different levels of prompting: the waiting prompt (waiting for the child to respond), the open question prompts (e.g., “What could you ask me?”), the fill-in prompt (e.g.,: “you can ask me: what is…”), and the tell-prompt (e.g., “you can ask me: could you open the block?”). Parts of the prompts were preprogrammed and could be adjusted by the therapist to match the target behavior of the child, and the others were provided during the interaction.
The movements of the robot came simultaneously with the speech. These were both direct and natural reinforcements that were used to reward the (attempted) use of the target behavior. The direct reinforcements consisted of cheering by raising the arms in the air when the child did something good. The natural reinforcements provided by the robot consisted of opening the box with pieces of the game that the child needed to finish it.
3.4.3. Building the Coding Scheme
To analyze the attention and the engagement of the participants, we looked into different coding schemes for attention and engagement, but could not find a scheme with a specific focus on attention and engagement of young children with ASD with robots. We, therefore, needed to create a new coding scheme. We started with a set of behaviors and modifiers from the engagement coding scheme developed by Perugia et al. [
30]. This coding scheme was selected because it was created for people who (1) cannot self-report and (2) are sited on a table when interacting with games or robots. However, the behavior of children with ASD and people with dementia may be very different so we needed to develop a coding scheme for children with ASD interacting with a robot and a game. The new coding scheme was developed in two phases: orientation and verification. During the orientation phase, six videos were watched (three in the PRT session without the robot and three in the robot-assisted PRT-sessions). Of these sessions, only the behavioral cues that were mentioned in the literature (gaze, facial expression, and gesture) were noted and included [
9,
17,
25,
27,
29]. A useful addition was the valence (the affect) of the behavior as identified as an important modifier by [
29]. During the verification phase, the draft version of the coding scheme was evaluated in multiple discussions with expert clinicians to refine and make the coding scheme specific for the participant group of children with ASD.
After the alterations, the components of the coding scheme were more specifically defined. The direction of behavior was determined by the object the child would look at or reach towards. The affective component for the gaze was linked to the facial expression of the participant. The positive facial expression denoted when the child was smiling and laughing. Negative facial expressions were denoted when the facial expression was identified as confusion, sadness, fear, anger, or boredom. The affective component for the gestures was linked to the specific behaviors that are interpreted positively in the behavior of a child with ASD. A positive gesture would be sharing game items with the parent or therapist, handshaking while playing, and asking to share something while pointing at it or reaching out, e.g., asking the parent to help them while reaching out to a bag with game components. If the same behavior occurred outside of play, this would be regarded as a neutral behavior. Negative gestures were categorized as throwing the game pieces, grabbing without asking, or refusing to touch and play with the game. A summary of the final observation scheme that was used for the analysis is shown in
Table 1.
3.4.4. Video Material
The video material depicted the game-mediated interactions between a child and an NAO robot in the presence of a therapist and a parent. The starting point of the observation was the moment when the robot greeted the child by saying “hello” and invited the child to shake its hand. The endpoint of the observation was the moment when the robot said goodbye to the child and invited the child to shake its hand again. When no handshake occurred at the end of the interaction, the endpoint of the observation was determined by the moment the robot was switched off by the child or the therapist. The analyzed videos were: early sessions (sessions 2 and 4), middle sessions (sessions 10 and 11), and sessions at the end of the treatment (sessions 19 and 20). Session 2 and 4 were taken as the starting point since sessions 1 and 3 were without the robot-assisted element. As explained in [
38], the first session, the PRT therapist observes the parent and child interacting. In session 2, the robot interacts with the child. In the 3rd session, the parents were asked how they think about implementing PRT principles and the PRT therapist informs the parents about her observations. Within sessions 4, 5, 7, 8, 10, 11, 13, 14, 16, 17 and 19, 20 the robot training takes place.
3.5. Data Analysis
The video material was scored using the coding scheme presented in
Table 1. To score the video material the software of Observer XT 14.2 was used. To analyze the statistical data SPSS version 20.0 was used. The dependent variables were calculated in the following way.
The duration of attention is the percentage of time that the child interacts with the robot during the play session. The components of the observation scheme that were selected as measures for the attention of the child in interaction with the robot are: gaze towards the robot, gaze towards the game, reaching out towards the robot, and reaching out towards the game. Elements of both gaze and reach were considered to be equal parts of the behavior. These components of the observation scheme were selected because the child is in gameplay interaction with the robot. The measure of attention is comprised of the sum of the percentages of gaze and arm/hand behavior regarding the robot and game divided into 2.
The duration of engagement is the percentage of time that the child interacts with the robot during the play session in therapy with positive affective expressions or gestures. The components of the observation scheme selected as measures for engagement of the child in interaction with the robot are: gaze towards the robot with a positive facial expression, gaze towards the game with positive facial expression, reaching towards the robot with positive gestures, and reaching towards the game with positive gestures.
The percentages of the independent variables were calculated per session. To establish the percentage at different moments in the therapy, an average of sessions 2 and 4, 10 and 11 and 19 and 20, was taken, which would represent the attention and the engagement at the start, middle, and end of the therapy. A one-way repeated-measure ANOVA was computed to see whether the attention and/or engagement towards the robot would significantly differ between the stages of treatment within the subject. A post hoc analysis test using the Bonferroni correction was made to compare the different periods of time. In the final step of the statistical analysis, we looked at the individual trends of the individual components (game and robot) separately and their changes over time.
Additionally, a statistical analysis was conducted regarding the change in the attention and the engagement towards the others in the therapy (parent and therapist). This was conducted in a similar way for the dependent variables (attention and engagement towards the robot and the game).
4. Results
4.1. Attention towards the Robot
Children’s attention towards the robot was measured at three different stages of treatment.
Figure 2 displays the overall average change in the attention towards the robot (AR) and the attention towards others (AO) (attention to both parent and therapist) at the different stages of treatment. Normality checks were carried out on the residuals which were approximately normally distributed with an
Fmax of 3.05. Mauchly’s test indicated that the assumption of sphericity was not violated. A repeated-measures ANOVA showed that the mean attention of the child towards the robot did not significantly differ between the different stages of the treatment
F (2, 10) = 0.19,
p = 0.83, partial
η2 = 0.04. This indicates that there is no change in attention to the robot over time.
In follow-up analyses, we focused on the attention of the child towards the robot and attention towards the game as two separate variables. These were measured at the same stages of treatment as the original measurement. For both variables, the assumptions for conducting an ANOVA were met. A repeated-measures ANOVA showed that the mean attention towards the robot only did not significantly differ between the different stages of the treatment F (2, 10) = 3.58, p = 0.07, partial η2 = 0.42. This indicates that there is no change over time in the attention towards the robot as a stand-alone element. The repeated-measures ANOVA showed that the mean attention towards only the game did not significantly differ between the different stages of the treatment F (2, 10) = 0.38, p = 0.7, partial η2 = 0.07. This indicates that there is no change over time in the attention to the game on its own.
4.2. Engagement with the Robot
Children’s engagement with the robot was measured at three different stages of treatment.
Figure 3 displays the overall average change of engagement with the robot (ER) and the engagement towards others (EO) (engagement with both parent and therapist) at the different stages of treatment. Normality checks were carried out on the residuals which were approximately normally distributed. Boxplots and Shapiro–Wilk statistics indicated that the assumption was supported:
Fmax was 1.37. Mauchly’s test indicated that the assumption of sphericity was not violated. A repeated-measures ANOVA showed that the mean engagement with the robot did not significantly differ between the time points
F(2, 10) = 0.34,
p = 0.72, partial
η2 = 0.06. This indicates no change in engagement with the robot over time.
In follow-up analyses, the focus was on the engagement of the participant with the robot and engagement with the game as two separate variables. These were measured at the same stages of treatment as the original measurement. For both variables, the assumptions for conducting an ANOVA were met. A repeated-measures ANOVA showed that the mean engagement towards only the robot did not significantly differ between the different stages of the treatment F (2, 10) = 0.84, p = 0.46, partial η2 = 0.14. This indicates that there is no change over time in the engagement with the robot as a stand-alone element. A repeated-measures ANOVA showed that the mean engagement towards only the game did not significantly differ between the different stages of the treatment F (2, 10) = 0.37, p = 0.70, partial η2 = 0.07. This indicates that there is no change over time in the engagement in the game without the robot.
4.3. Attention towards Others in the Therapy
Additional analyses were conducted to investigate whether the attention towards others (parents and therapist) had changed at three different stages of treatment. In
Figure 4, the average overall change in the attention towards the others (AO), the attention towards parents (AP), and the attention towards the therapist (AT) are displayed at different stages in treatment. Normality checks were carried out on the residuals which were approximately normally distributed. Boxplots and Shapiro–Wilk statistics indicated that the assumption was supported:
Fmax was 1.486. Mauchly’s test indicated that the assumption of sphericity was not violated. A repeated-measures ANOVA showed that the mean attention towards the others in the therapy significantly differed between the different stages of the treatment
F (2, 10) = 4.96,
p = 0.03, partial
η2 = 0.5. Post hoc tests using the Bonferroni correction revealed no significant changes at different moments measured in time. This indicates that there is an increase in the attention of the child towards others in the therapy.
In follow-up analyses, we focused on the attention of the participant towards the parent and the attention towards the therapist as two separate variables. These were measured at the same measurement stages as the original measurement. For both variables, the assumptions for conducting an ANOVA were met. A repeated-measures ANOVA showed that the mean attention towards the only parents did not significantly differ between the different stages of the treatment F (2, 10) = 3.71, p = 0.07, partial η2 = 0.43. Besides, the within-subjects contrasts showed a linear effect of time of F (1, 5) = 27.86, p = 0.00, partial η2 = 0.85. This indicates that there is a change over time in the attention of the parent with a linear increase in time duration. A repeated-measures ANOVA with a Huynh–Felt correction showed that the mean attention towards the therapist, without the parent, did not significantly differ between the different stages of the treatment F (1.12, 5.57) = 0.60, p = 0.49, partial η2 = 0.11. This indicates that there is no change over time in the attention of the therapist.
4.4. Engagement with Others in the Therapy
Additional analyses were conducted to investigate if the engagement with others (parent and therapist) changed at three different stages of treatment.
Figure 5 shows the average overall change of the engagement with others (EO), the engagement with the parent (EP), and the engagement with the therapist (ET) displayed at the different stages of treatment. Normality checks were carried out on the residuals which were approximately normally distributed. Boxplots and Shapiro–Wilk statistics indicated that the assumption was supported:
Fmax was 10.957. Mauchly’s test indicated that the assumption of sphericity was not violated. A repeated-measures ANOVA showed that the mean attention towards the others in the therapy significantly differed between the different stages of the treatment
F (2, 10) = 4.48,
p = 0.04, partial
η2 = 0.47. Moreover, the within-subjects contrasts showed a linear effect of time of
F (1, 5) = 9.11,
p = 0.03, partial
η2 = 0.65. Post hoc tests using the Bonferroni correction revealed no significant decrease at the different moments measured in time. This indicates that there is a change in engagement over time towards other humans with a linear increase over time.
In follow-up analyses, we focused on the engagement of the participant with the parent and the engagement with the therapist as two separate variables. These were measured at the same stages of the treatment as the original measurement. For both variables, the assumptions for conducting an ANOVA are met. A repeated-measures ANOVA showed that the mean engagement with the only parent did not significantly differ between the different stages of the treatment F (2, 6) = 6.65, p = 0.03, partial η2 = 0.43. Besides the within-subjects contrasts showing a linear effect of time of F (1, 3) = 11.25, p = 0.04, partial η2 = 0.79, post hoc tests using the Bonferroni correction revealed no significant decrease at the different moments measured in time. This indicates that there is a change over time in the engagement towards the parent with a linear increase over time. A repeated-measures ANOVA showed that the mean engagement with only the therapist did not significantly differ between the different stages of the treatment F (2, 10) = 1.68, p = 0.24, partial η2 = 0.25. This indicates that there is no change over time in the engagement with the therapist on its own.
5. Discussion
This research aimed to find out whether the attention and engagement of children with ASD will change with the time during longitudinal interactions of 20 sessions of robot-assisted therapy provided within a period of 6 months. We hypothesized that there would be a decrease in the attention and engagement of the child with the robot throughout the repeating sessions of therapy.
Our results rejected the hypothesis and showed that the children remained attentive to the game and the robot throughout the overall treatment. These results contradict the findings reported in related research by Coninx et al. [
8] and Ahmad et al. [
9], which found that typically developing children’s attention and engagement with the robot will decrease over time. The most probable explanation is the high variety and personalization of these games from the perspective of game design and from the clinical perspective. In designing these games, we included multiple elements and objects that especially appeal to the thinking patterns and the perception of children with ASD. Examples of this are the choice of typical games that children with ASD play, uncovering the game opportunities gradually by a robot (i.e., by further opening a sliding box with the missing puzzle piece), etc. From the perspective of clinical practice, we adapted the difficulty of the game to the developmental level of each child and used specific motivational techniques for the children with ASD. These techniques increase the clearness of the interactions, which could have also contributed to the sustained engagement of this particular user group. An alternative explanation for the constant level of attention might be the possible hyper-focus of the children with ASD on interesting objects [
26], since most children with ASD are believed to enjoy the interaction with robots, such as NAO, that have simple facial features and interact clearly through moving body parts and speech.
Other studies with typically developing children have used alternative methods for sustaining engagement, focusing on adapting to the child’s emotional state and ensuring the robot remembers the names and performance of the children on tasks [
9]. These are complementary approaches that should be used in the further development of the robot therapies. The same study found that game adaptation did not result in sustaining long-term social engagement, which differs from our results. The long-term effect of a game that continued during several sessions with children with ASD (who build different parts of a LEGO city at each session) [
10] also kept the engagement of these children high for 4–6 sessions with a robot. The current study succeeded in keeping the engagement high by creating more game variability and personalization for 20 therapy sessions.
There are several limitations to this study. The first limitation is the lack of visibility of children’s behavior at some moments due to the fixed camera. To prevent inaccurate scoring, poorly visible behaviors were scored as not visible or as no movement. The second limitation of the study was the limited flexibility of the behavior of the robot: as the therapy progressed, it seemed to be too familiar and not tuned to the progress of the children in the course of the therapy, since often the children responded faster than the robot could respond. Most of the games were preprogrammed by the researcher with a social science background using a specially created end-user interface of the TiViPE robot controller [
37,
38]. This is an achievement by itself, but also restricts the interaction fluency since the end-user interface allows less flexible interactions than the complete interface. A third limitation is the small sample size. This makes it hard to draw firm conclusions on the effect of the robot on children with ASD in general. The fourth limitation of this study is the scoring of the data by a single observer. To increase the reliability of this research, the results should be confirmed by a second rater. A major recommendation for the improvement of this line of research would be to further enhance the variety of the games, since some children expressed their desire for new games that follow the progress of the child.