Multi-Session Influence of Two Modalities of Feedback and Their Order of Presentation on MI-BCI User Training

Abstract

By performing motor-imagery tasks, for example, imagining hand movements, Motor-Imagery based Brain-Computer Interfaces (MI-BCIs) users can control digital technologies, for example, neuroprosthesis, using their brain activity only. MI-BCI users need to train, usually using a unimodal visual feedback, to produce brain activity patterns that are recognizable by the system. The literature indicates that multimodal vibrotactile and visual feedback is more effective than unimodal visual feedback, at least for short term training. However, the multi-session influence of such multimodal feedback on MI-BCI user training remained unknown, so did the influence of the order of presentation of the feedback modalities. In our experiment, 16 participants trained to control a MI-BCI during five sessions with a realistic visual feedback and five others with both a realistic visual feedback and a vibrotactile one. training benefits from a multimodal feedback, in terms of performances and self-reported mindfulness. There is also a significant influence of the order presentation of the modality. Participants who started training with a visual feedback had higher performances than those who started training with a multimodal feedback. We recommend taking into account the order of presentation for future experiments assessing the influence of several modalities of feedback.

1. Introduction

Motor-Imagery based Brain-Computer Interfaces (MI-BCIs) enable their users to send commands to a digital tool, for example, a wheelchair or a video-game, by performing motor-imagery tasks only, for example, imagining hand movements, while their brain activity is recorded [1]. MI-BCIs are not only useful to control applications. They can also be used for neurofeedback training [2]. For instance, they can be used in the context of post-stroke motor rehabilitation to provide sensory feedback when the system detects that the modulation of the brain activity has exceeded a certain threshold. The aim is to promote neurological modifications and thereby motor recovery [3].

Despite their very promising applications, BCIs are not much developed outside research laboratories yet. One of the main reasons is that the average BCI performance, that is, rate of correctly recognized MI tasks, is too low. For example, when the system has to decide which task the user is performing between two motor imagery tasks, for example, imagining a right versus a left hand movement, on average the system is mistaken once every four predictions [4].

1.1. Role and Limitations of Feedback in BCI User Training

In order to improve BCIs, some research is led to better train users to produce brain activity that is recognizable by the machine [5]. Indeed, for the users to control a BCI they must first learn to understand and control the feedback that the BCI provides them with. However, the adequacy of the feedback provided during the training has been questioned both by the theoretical literature [5] and by experimental results [6].

The typical visual feedback, consisting of an extending bar or a moving cursor, does not take into account recommendations from the educational psychology field on feedback, for example, providing multimodal feedback [7]. The literature indicates that taking these recommendations into account can improve BCI performances and user experience [8,9]. Also, several experimental studies have demonstrated that the typical visual feedback can be improved. For instance, using a congruent realistic visual stimulus representing the body part that the participant is imagining enables better and less variable BCI performances as well as an increase of the sense of agency, the imagined kinaesthetic sensations and the sense of embodiment [10,11]. Previous results indicate that the sense of embodiment correlates with MI-BCI performances and the ability to modulate the sensorimotor rhythm [10,12]. The inadequacy of the training, and more particularly of the feedback, is probably part of the reasons why MI-BCIs remain insufficiently reliable [5].

1.2. Improving the Feedback through Its Modality

One of the main characteristics of the feedback is its modality of presentation, which represents how the information that it conveys is presented [13]. Feedback is provided through external sources or displays, for example, visual, auditory or haptic displays. Currently, visual stimuli are the most common type of feedback, most probably because vision is the sense on which daily life perception relies the most. The optimisation of the feedback through the adaptation of its modality has been a subject of investigation in several fields such as motor training or rehabilitation [14,15]. For instance, in motor skills related studies, the complexity of the motor task to learn, as well as the skills of the learner, have a main influence on the type of feedback modality to favour [14].

Somatosensory feedback, through the use of vibrotactile stimuli, functional electrical stimuli (FES), orthosis/exoskeleton or vibrations on the muscles and tendons, was used for BCI user training [13]. Initial research on somatosensory feedback focused on tactile feedback through the use of vibrotactile motors. Compared to simple visual feedback, continuous vibrotactile haptic feedback provided on the neck, the neck and forearms or on the palm of the hands, does seem to be just as efficient in terms of neurophysiological response and BCI performances [16,17,18]. Comments from participants indicate that tactile feedback feels more natural [16]. Even though the performances post training with visual and vibrotactile feedback seem similar, haptic feedback could interfere with the motor imagery task, especially when providing negative feedback during misclassification [16]. The real benefit from tactile feedback compared to visual feedback seems to arise when the visual attention or cognitive load is high [16,17,19].

In everyday life, the brain relies on information arising from multiple senses which often complement and confirm each other. This redundancy increases the degree of confidence associated with the perception [20]. Compared to a visual feedback alone, a multimodal feedback composed of both visual and somatosensory stimuli (provided using an orthosis or vibrotactile stimuli), was found to increase the characteristic neurophysiological response to a motor preparation or execution as well as the MI-BCI performances of neurotypical participants [21,22,23].

The multi-session influence of vibrotactile feedback remains unknown. To our knowledge, all previous studies using vibrotactile feedback only assessed its impact during a single user training session [16,17,18,19,22,23,24]. Such multi-session influence could be impacted by the fact that negative vibrotactile feedback, that is, vibrations indicating to the user that there is a mismatch between the instructed task and the recognized task by the system, could interfere with the motor imagery task [16]. Furthermore, a multi-session use of vibrotactile feedback could lead to a decrease in the perceived intensity of the feedback [19]. Finally, as found in motor skill related studies, the skills of the learner, which evolve in time, might influence the modality of feedback to favour [14]. Previous studies in BCI do report that the expertise of BCI users influences the characteristics of the user training and feedback to favor [25,26]. For instance, Bonnet et al. found that, when playing a MI-BCI video game, a 2 players condition improved the performances of the best-performing participants compared to a single-user condition [25]. Also, Gargiulo et al. found that compared to a unimodal visual feedback, a multimodal visual and audio feedback increased the performances of only naïve BCI participants [26].

Related to this evolution of skills in time, an influence of the order of presentation of the feedback on BCI user training outcomes was already explicitly hypothesised in [19], but not formally evaluated. Several experiments testing the influence of two modalities of feedback have randomized their order of presentation [18,21,24], thus implicitly hypothesizing that the order of presentation of the modalities of feedback could influence their results. However, to our knowledge the influence of the order of presentation remains unknown and was never formally investigated.

1.3. Influence of the Participants’ Profile on BCI User Training

An association between the participants’ profile and BCI performances is well established [6,27,28]. For instance, participants with low mental rotation scores [29], tensed and/or non-autonomous (both measured using the 16PF5 questionnaire [30]) were shown to have lower BCI performances than the others [31].

Such influence could be, at least in part, due to a differential impact of feedback on learning outcomes, that is, BCI performances and user experience, depending on the participants’ profile [13]. For instance, we found that a social presence and emotional feedback provided using a learning companion had a differential impact on the participants’ performances and reported efficiency/effectiveness felt during BCI training, that is, a measure of the user experience, depending on their autonomy [32]. Also, the influence of the participants’ tension on MI-BCI performances seems to be modulated by the gender of the experimenter, which can be considered as a complex type of social presence and emotional feedback. Tensed and non tensed participants had better performances when training with men experimenters and women experimenters, respectively [33].

Furthermore, it can be assumed that the characteristics of the learners, particularly in terms of perceptual and processing resources, could influence the modality that should be favored to provide feedback [13,34]. The ability to imagine movements encompasses two components. One is the visual component, when people visually picture themselves or someone else performing a movement. The other is the kinaesthetic component, when people associate somatosensory sensations to their representation of the movement [35]. Previous results on neurotypical participants regarding the impact of visual and kinaesthetic abilities on motor imagery based BCI training are not conclusive. Two validated questionnaires are available to assess visual and kinaesthetic imagery abilities. The Kinaesthetic and Visual Imagery Questionnaire (KVIQ) [36] and the Motor Imagery Questionnaire Revised-Second Edition (MIQ-RS) [37]. Vuckovic et al. found that offline BCI performances when classifying right versus left hand motor imagery tasks were strongly correlated to the kinaesthetic imagery score of the KVIQ [38]. Also, the representation of subjective behaviours of the MIQ-RS was found to be a predictor of MI-BCI performances with abstract visual feedback [39]. Recently, higher kinaesthetic imagery abilities of participants were associated with a higher similarity between neurophysiological response occurring during executed movement and kinaesthetic imagination of the same movement [40]. However, Rimbert et al. did not find any correlation of the MIQ-RS scores with MI-BCI performances when classifying resting state versus right hand motor imagery, using realistic visual feedback during the training [41]. We hypothesise that the modality of feedback might benefit participants differently depending on their visual and kinaesthetic motor imagery abilities. Indeed, if participants rely on visual or kinaesthetic motor imagery, then providing them respectively with visual or tactile feedback might improve or impede their performance of the task by soliciting similar cognitive resources.

1.4. Research Hypotheses

In the previous Sections, we have highlighted the presence of several gaps in the literature. Previous studies using vibrotactile feedback assessed its influence during only a single training session [16,17,18,19,22,23,24]. Thus, the multi-session influence of vibrotactile feedback on BCI user training outcomes, that is, BCI performances and user experience, remains unknown. Also, an influence of the order of presentation of different modalities of feedback was hypothesised and taken into account in previous studies [18,19,21,24]. However, the influence of the order of presentation of the modalities of feedback on BCI user training outcomes, that is, BCI performances and user experience, has never been assessed. Finally, the participants profile, in particular the autonomy, tension, mental rotation abilities and initial visual and kinaesthetic imagery abilities, has been related to BCI performances and the type of feedback to provide [6,13,27,28] (Note, however, that these associations between users’ profile and BCI performances are not always consistently found across experiments [42,43,44]). However, the literature provides very little information on the type of modality of feedback to provide depending on the profile of the participants. Based on the literature, and focusing on the comparison of a unimodal realistic visual and a multimodal realistic visual and vibrotactile feedback, we formulated the following hypotheses, that we test in this paper:

• (H1—MI-BCI performances) MI-BCI performances undergo a multi-session influence of the modalities of feedback, possibly modulated by their order of presentation.
• (H2—User experience) User experience undergo a multi-session influence of the modalities of feedback, possibly modulated by their order of presentation.
• (H3—Participants’ profile) These effects are modulated by participants’ profile, that is, autonomy, tension, mental rotation abilities or initial visual and kinaesthetic imagery abilities.

The remainder of this article is organised as follows. In Section 2 we present the details of the experimental protocol that we used. Then, in Section 3 and in Section 4, we respectively report and discuss the results from our experiment. Finally, in Section 5, we offer a conclusion on the matter as well as ideas and recommendations for future research.

2. Materials & Methods

The neurotypical participants included in this experiment took part in 10 sessions each, 5 for each modality of feedback, that is, multimodal vibrotactile and visual feedback together and unimodal visual feedback alone. A within participant comparison for the influence of the modalities of feedback was chosen. Thus, the order of presentation of the modalities of feedback was randomized across participants (see Figure 1).

2.1. Participants

Sixteen MI-BCI naive participants were included in this study (8 women; age 18–27, M = 22.31, SD = 2.33). None of them had motor impairment nor any history of neurological or psychiatric disorder. Participants randomly started training with either a unimodal visual or a multimodal visual and vibrotactile feedback (see Figure 1).

Our study was conducted in accordance with the relevant guidelines for ethical research according to the Declaration of Helsinki. Participants gave informed consent before participating in the study. The study has been reviewed and approved by Inria’s ethics committee, the COERLE (approval number: 2019-04).

2.2. Tasks

To control the BCI, participants were asked to perform one of the following three tasks: (1) rest (2) imagine moving either their right or left hand and (3) execute a movement of the opposite hand to the one they imagine moving. Indeed, as we considered a potential use of such modalities of feedback for motor rehabilitation post-stroke in the future, we also considered the motor execution task which is recommended for such an application with patients who can move [45]. However, in this paper we focus on the effect of the feedback on motor imagery BCI performances, as neurotypical people do not have intrinsic feedback when performing motor imagery tasks, compared to when they perform motor execution tasks. Moreover, by definition a BCI cannot be based on real movements. Thus, we report motor imagery tasks BCI performances only in this paper. To stay focused, the results related to executed movements are thus not reported, though they can be found in [13]. Participants were instructed to kinaesthetically imagine an opening and closing movement of their hand. In other words, participants were asked to focus on the sensations that they would feel when imagining the movement. Feedback was only provided for the motor-related tasks. The feedback represented how well the BCI system recognized the modification occurring in the brain activity of the participants when they performed motor-related tasks compared to their brain activity when they were in a resting state. It lasted 5 s and was updated at 16 Hz, according to the last 1 s of EEG signals.

2.3. Feedback

A realistic visual feedback representing arms was displayed on a screen that was placed over the arms of the participants to give the impression of embodiment (see Figure 2). Thus, the participants could see a virtual representation of arms but could not see their own arms. This choice was made based on recommendations to provide a feedback that is congruent with the MI strategy adopted by the participants [10,11,12] (see Section 1.1 for further detail). Participants were asked to place their hands on the table in front of them in a supine position (palms facing upwards) below the screen. Virtual hands performed opening and closing movements depending on the classifier output. The more confident the system was in its recognition of the task, the faster the hand was opening and closing. Only positive feedback was displayed, that is, the feedback was provided only when there was a match between the instructed task and the task recognized by the BCI system. Indeed, the use of negative feedback has a negative impact on the sense of embodiment [46]. Also, a negative vibrotactile feedback was reported to interfere with a motor imagery task [16].

In addition to the visual feedback, a tactile feedback was provided during the 5 first sessions for half of the participants and during the last 5 sessions for the other half. This tactile feedback consisted in vibrations on the wrist provided using vibrotactile motors contained in gloves worn by the participant. The system of vibrotactile motors embedded in gloves was used in a previous MI-BCI experiment aiming at comparing, for a single session, a visual and an equivalent tactile feedback in a high cognitive load situation [19]. The intensity of the vibration depended on the output of the classifier. The better the classifier recognized the task performed by the participant, the stronger the vibration got. The minimal and maximal vibration frequencies were adjusted at the beginning of the first session. Participants were presented with the lowest, that is, 50 Hz, and highest, that is, 200 Hz, intensities of vibrations and asked if they felt the vibrations and if feeling them repetitively during 6 s would be painful. None of the participants asked to change these default intensities. There were five thresholds separating uniformly six different intensities of vibration. The discriminability between two successive intensities of vibrations was tested as well. Participants were asked to recognize the highest intensity of vibration between each pair of consecutive intensities presented successively in a random order. Each of our participants distinguished consecutive vibrations for at least three of the five thresholds. The intensity of vibration and the discriminability were tested independently for each hand.

Participants were asked to perform or imagine performing the movements as fast as the maximum speed of the feedback, that is, one opening and closing movement per second. A demonstration of the maximum speed was made at the beginning of the session. We chose to give these instructions to maximize the similarity between the realistic feedback presented on the screen and the motor imagery task that the participant would perform to promote the sense of agency, that is, feeling of control over the feedback.

2.4. Session Organisation

Participants took part in 10 MI-BCI sessions, each lasting between 1.5 and 2 h, spread over a month with 2 to 3 sessions per week and no more than one session per day. The sessions were organized as follows. First, depending on the session, participants were asked to complete one validated psychometric questionnaire (see Section 2.6) assessing some aspects of their personality and/or cognitive profile (∼10 to 20 min). Then, the EEG headset was installed as well as the gloves containing vibrotactile motors (∼10 to 20 min). Two baselines were recorded. One to assess the brain activity of the participants while being at rest with eyes opened (∼3 min) and one to assess the brain activity of the participants while they only perceived sham vibrotactile and/or visual feedback without performing any motor-related task (∼6 min). Next, participants performed 7 runs, each lasting 5.33 min (∼45 min containing 5 min of break). The first three runs of each session were used to either calibrate the BCI system (for session 1 and 6) or to recalibrate it if necessary (all other sessions, see Section 2.7 for details). Finally, the cap was uninstalled, one or two questionnaires were filled-in depending on the session and a quick debriefing was made.

2.5. Trials Organisation

During each run, participants had to perform a total of 20 trials, 10 trials of motor execution and 10 trials of motor imagery. Tasks were presented in a random order and each trial lasted 15 s. Each trial included both a resting task and a motor task, that the BCI aimed at discriminating online. A trial unfolded as described in the following sentences (see Figure 3). At t = 0 s, a cross was displayed in the center of the screen. At t = 1 s, a “beep” announced the coming instruction and half a second later, at t = 1.5 s, the participants were asked to rest for 2.5 s. Then, at t = 4.5 s an arrow pointing left or right indicated which task, that is, left or right hand movement execution or imagery, the participants had to perform repeatedly until the end of the trial. Finally, from t = 5.75 s, either visual feedback only or both visual and tactile feedback was continuously provided until the end of the trial. A gap lasting between 3.5 s and 4.5 s separated each trial.

2.6. Questionnaires

Throughout the experiment, participants were asked to fill or carry out the following questionnaires or tests:

• Beginning of the experiment - 5th edition of the 16 Personality Factors (16PF5) [30] - To assess the personality and the cognitive profile of the participants including their autonomy and tension.
• Every session - NeXT questionnaire [47]-To assess participants’ states and user experience. This questionnaire provides five dimensions of user-state and/or user experience. Three of them are assessed pre and post training and evaluate the mood, mindfulness and motivational states of the user. Two of them assess the user experience post-training through the cognitive load, that is, amount of cognitive resources required to control the MI-BCI system, and the agency, that is, the feeling of control of the participant over the feedback provided by the MI-BCI. The evolution of the participant’s states also provides an information regarding the user experience.
• 1st, 6th and last sessions - Kinesthetic and Visual Imagery Questionnaire (KVIQ) [36]-To determine participants’ ability to visualize and feel an imagined movement.
• 2nd session-Mental Rotation Test [29]-To determine participants’ ability to mentally visualize a 3D object rotating in space.

2.7. EEG Recordings & Signal Processing

The electroencephalographic (EEG) data were recorded with 11 active electrodes, using a g.USBAmp EEG amplifier (g.tec, 4521 Schiedlberg, Austria). The electrodes were placed on the scalp of the participant over the sensorimotor area (at locations FC3, FC4, C5, C3, C1, Cz, C2, C4, C6, CP3, CP4 in the 10–20 system). They were referenced to the left earlobe and grounded to AFz. The data was sampled at 256 Hz, and processed online using OpenViBE 2.1.0 [48].

We used a participant-specific classifier to compare the data acquired during the resting task to the data acquired during the motor imagery (MI) task. We used the following pipeline to classify the data. First, two Laplacian spatial filters were computed over C3 and C4 [49]. Then, EEG signals were band-pass filtered in 8–10 Hz, 10–12 Hz, 12–16 Hz, 16–20 Hz and 20–24 Hz using Butterworth filters of order 5. The band power of the 5 frequency filtered EEG signals were then computed by squaring the signals, averaging them over the last 1 s of EEG signal (we used an overlapping sliding window with a length of 1 s and an overlap of 15/16 s) and log-transforming them. This resulted in 10 different band-power features that were fed to a shrinkage Linear Discriminant Analysis (LDA).

For the first session with a new feedback modality, that is, session 1 or session 6, the LDA classifier was initially calibrated on the data from the first three runs of that session. Since these three runs were the first ones with a new feedback modality, no classifier was available yet to provide real-time feedback. Rather than providing no feedback for these first three runs, we provided sham feedback, that is, the virtual hands moved similarly to what could happen if there was a classifier but randomly, as in [19]. Participants were aware that this feedback was fake during these three runs. This sham feedback enabled us to build a more robust classifier, by collecting EEG data including subjects’ reaction to the feedback, that is, EEG signals similar to those that will be collected in subsequent runs. The calibration data from the first three runs of that session were also used to compute the median and mean absolute deviation of the MI LDA classifier outputs (distance to the separating hyperplane) to normalize the classifier output. The MI classifier normalised outputs were then computed online by subtracting the median value to the LDA classifier outputs (distance to the separating hyperplane) and dividing it all by twice the value of the mean absolute deviation. This was done to reduce possible bias due to EEG non-stationary. Indeed, a similar approach was shown to improve BCI performance in [50]. This classifier with normalized output was then applied on the subsequent runs of that session to provide online feedback.

For each of the subsequent sessions with the same feedback modality, that is, sessions 2 to 5 or 7 to 10, the LDA classifier used was trained and centered (i.e., its output was normalized as described above) on the data from all the runs of the previous session. This enabled us to use a classifier trained on more data (7 runs versus only 3 for the classifier used in session 1 and 6)—and thus likely to be more robust—as well as to provide immediately real feedback from the very first run of each session. However, due to between-session non-stationarity [51], there was still a risk that the classifier built on data from the previous session may not work that well on the data from the current session. Therefore, after the third run of each session (except session 1 or 6), we estimated the average MI performance over the three first runs to assess the relevance of the current classifier. Here, the MI classification accuracy, or performances, corresponded to the percentage of epochs (1 s long time windows) from the feedback and resting state periods that were correctly classified. The performances of the current classifier were considered as low if they were below the statistical chance level of 62.5% for 30 test trials [52] and/or below the minimum performances across runs of the last session. If the performances of the classifier were low, then the median and mean absolute deviation used to normalize the data of the classifier were changed for the ones computed on the data from the first three runs of the day, to try to reduce some of the non-stationarity. The performances obtained during the first three runs were then recalculated with the new centering data. If the new performances were not low anymore, then the centered classifier was kept for the last four runs of the session. If the new performances were still low, then the classifier was retrained on the data of all the available runs from the current session, that is, the first three runs. If the cross validation accuracy on the first three runs minus 5% (a margin made to compensate for the optimistic estimations of cross validation accuracy [53]) was not considered as low, then the new classifier was kept. Otherwise, the classifier allowing the best performances over the first three runs was kept. To summarize, for each session (except the first one with a given feedback modality), the classifier used was trained on the data from the previous session. If that classifier performed poorly on the first three runs of the current session, it was either recentered or retrained on the data from these three runs, depending on which solution led to the highest performances (see Figure 4).

2.8. Variables, Factors & Statistical Analyses

As presented in the introduction, our experiment aimed to test three different hypotheses. In the following Subsections we present the variables, factors and statistical analyses used to test each of these hypotheses. The statistical analyses mostly consist of ANOVAs, that are considered as robust against the normality assumption. To the best of our knowledge, no other non parametric test enabled to perform the analysis that we were interested in performing. Spearman or Pearson correlations were also obtained depending on the distribution of the data collected (assessed using Shapiro-Wilk tests).

2.8.1. H1—MI-BCI Performances

Our first hypothesis (H1) was that MI-BCI performances undergo a multi-session influence of the modalities of feedback, possibly modulated by their order of presentation. The measure of performance used for the analysis was the online MI classification accuracy (see Section 2.7) averaged over the runs 4 to 7 of the different sessions, that is, when the classifier was considered optimized for the participant’s session. The MI performance, or classification accuracy, corresponds to the percentage of recognition of the motor imagery task (classification of rest vs. MI).

We thus tested the influence of the modality of feedback, that is, “Modality” and its order of presentation, that is, “Order” on the evolution of MI classification accuracy over the sessions, that is, “Session”. To do so, we performed a 3-way repeated measures mixed ANOVA with “Modality” (2 modalities of feedback), “Order” (2 order of presentation randomized across participants) and “Session” (5 sessions per modality) as independent variables and the repeated measures of MI classification accuracy as dependent variable.

2.8.2. H2—User Experience

Second, we assessed the potential multi-session influence of the modalities of feedback and its order of presentation on the user experience. The user experience is assessed using the two percentages of cognitive load and agency provided by the NeXT questionnaire of Hakoun et al. [47,54]. It is also assessed through the evolution of mood, mindfulness and motivation (in percent) of the participants between the beginning and end of the training. This evolution is evaluated by subtracting the measure post training to the measure pre training for each session. The higher the percentage, the more participants reported increased levels of positive emotions and calm, mindfulness, motivation, cognitive load and sense of agency.

These five measures of user experience were then used in five 3-way mixed ANOVAs with “Modality” (2 modalities of feedback), “Order” (2 order of presentation randomized across participants) and “Session” (5 sessions per modality) as independent variables and the difference of mood, mindfulness or motivation between the end and the beginning of the sessions or the measures of agency or cognitive load post training as dependent variables.

2.8.3. H3—Participants’ Profile

Finally, we wanted to know if characteristics of the participants’ profile influenced the type of modality to favour and could provide first elements of comprehension regarding the potential difference in MI-BCI performances (H3). We focused on traits and states that were shown to have an influence on MI-BCI performances in previous studies, that is, mental rotation scores (MRS) [29], tension and autonomy traits (both measured using the 16PF5 questionnaire [30]) [31]. We also assessed how well our participants could kinaesthetically and visually imagine the side of the body for which they had to perform the motor imagery task [36]. The Kinaesthetic and Visual Imagery Questionnaire (KVIQ) provided us with two scores, one for the visual imagery abilities and one for the kinaesthetic imagery abilities.

There were differences in initial visual imagery abilities between the participant groups formed by the modality they started with. Thus, we analysed the potential influence of participants’ initial visual imagery abilities in a specific analysis whose results are presented in Appendix A.

We also assessed whether the autonomy, tension, mental rotation abilities or initial kinaesthetic imagery abilities of our participants had impacted their mean MI classification accuracy by performing analyses of correlation.

3. Results

Before it all, to make sure that our results would not be biased, we checked for the presence of outliers among our participants. A performance was considered as an outlier if it was superior (or inferior) to the mean performances of all the participants by more (or less) than two standard deviation. We did not find any outlier for both modalities of feedback.

We also verified if there were significant differences of mental rotation scores (MRS), tension, autonomy or initial kinaesthetic or visual imagery abilities in the groups depending on the modality of feedback that they started training with, that is, “Order”. To check if the groups were comparable, we ran an ANOVA with “Order” as independent variables and either mental rotation scores (MRS), tension, autonomy, initial kinaesthetic or visual imagery abilities as dependent variable. Results indicate that groups were comparable in terms of MRS [F(1, 14) = 1.74, p = .21, ${\eta }^2$ = .13], tension [F(1, 15) = .7, p = .8, ${\eta }^2$ < ${10}^{-2}$], autonomy [F(1, 15) = .61, p = .45, ${\eta }^2$ = .05] and initial kinaesthetic imagery abilities [F(1, 15) = .57, p = .46, ${\eta }^2$ = .05].

However, initial visual imagery abilities were not comparable between the groups depending on the first modality of feedback they were training with [F(1, 15) = 6.94, p = .02, ${\eta }^2$ = .37]. Participants that first trained with visual feedback had lower initial visual imagery abilities (M = 3.38, SD = .3) than participants that started training with both visual and tactile feedback (M = 4.5; SD = .3). Previous results from the literature found that initial visual imagery abilities influence BCI performances [38,39]. Thus, we controlled for the potential influence of these variables. The results of this analysis are presented in Appendix A and did not reveal any potential bias from the difference found in participants’ initial visual imagery abilities that started with different modalities of feedback.

In the following sections, we report the results of the analyses presented in Section 2.8 that we performed to test each of our hypotheses.

3.1. H1—MI-BCI Performances

We started by testing the H1 hypothesis, that is, MI-BCI performances undergo a multi-session influence of the modalities of feedback, possibly modulated by their order of presentation. As stated in Section 2.8.1, we performed a 3-way repeated measures ANOVA with “Modality”, “Order” and “Session” as independent variables and the repeated measures of MI classification accuracy as dependent variable.

We found single significant effects of “Modality” [F(1, 14) = 8.47, p = .01, ${\eta }^2$ = .38], “Session” [F(2.22, 31.09) = 3.75, p = .03, ${\eta }^2$ = .2] (specificity corrected using the Greenhouse-Geisser method (epsilon = .56)) and “Order” [F(1, 14) = 7.02, p = .02, ${\eta }^2$ = .33]. No interaction was found between “Modality*Order” [F(1, 14) = .83, p = .38, ${\eta }^2$ = .06], “Session*Order” [F(2.22, 31.09) = 1.31, p = .29, ${\eta }^2$ = .09] (specificity corrected using the Greenhouse-Geisser method (epsilon = .56)), “Modality*Session” [F(2.45, 34.26) = 2.36, p = .1, ${\eta }^2$ = .14] (specificity corrected using the Greenhouse-Geisser method (epsilon = .61)), nor “Modality*Session*Order” [F(2.45, 34.26) = .62, p = .57, ${\eta }^2$ = .04] (specificity corrected using the Greenhouse-Geisser method (epsilon = .61)).

The significant impact of the session is related to the stable or decreasing performances of the participants until the third or fourth session which then tend to increase again in the remaining session(s). The mean MI classification accuracy of participants was significantly higher when they trained with multimodal feedback (M = 69.61, SD = 2.34) compared to when they trained with visual feedback only (M = 66.66, SD = 2.18). Finally, the mean MI classification accuracy over both feedback modalities of participants that started training with the visual feedback only (M = 72.95, SD = 2.57) was significantly higher than the mean MI classification accuracy of the participants that started training with the visual and tactile feedback (M = 63.32, SD = 2.57). The results can be seen in Figure 5.

3.2. H2—User Experience

Then, we tested the H2 hypothesis, that is, user experience undergo a multi-session influence of the modalities of feedback, possibly modulated by their order of presentation. As stated in Section 2.8.2, we analysed the multi-session influence of the modalities of feedback on the five indicators of the user experience, that is, cognitive load, sense of agency, mood, mindfulness and motivation.

We performed five 3-way mixed ANOVAs with “Modality”, “Order” and “Session” as independent variables and the difference of mood, mindfulness or motivation between the end and the beginning of the sessions or the measures of agency or cognitive load post training as dependent variables.

We found a significant influence of “Modality” on mindfulness [F(1, 14) = 5.85, p = .03, ${\eta }^2$ = .3] as well as a significant influence of “Modality*Session” on sense of agency [F(4, 56) = 4.32, p < ${10}^{-2}$, ${\eta }^2$ = .24], mood [F(4, 56) = 3.77, $p<{10}^{-2}$, ${\eta }^2$ = .21] and mindfulness [F(4, 56) = 2.97, p = .03, ${\eta }^2$ = .18]. Finally, we found a significant influence of “Session*Order” on the motivation [F(4, 56) = 2.71, p = .04, ${\eta }^2$ = .16]. All the detailed results from these analyses can be found in Appendix B. The significant results obtained are presented in the paragraphs below.

The agency was increasing over the three first sessions, decreased drastically on the fourth and increased on the fifth with the multimodal feedback. With the visual feedback only, it was decreasing over the three first sessions, and increased over the rest of the sessions. The agency seemed higher for the participants with a multimodal feedback during the second and third sessions. However, it seemed higher for the visual feedback than the multimodal one on the fourth session (see Figure 6).

The evolution over the sessions of difference in positive and calm emotions between the beginning and end of the sessions seems to be negative with a visual and tactile feedback and positive with a visual feedback only. During the first two sessions, when participants were training with visual and tactile feedback, they felt more positive emotions at the end of the session compared to the beginning of the session. It was the opposite when participants trained with visual feedback. During the last two sessions, whether participants trained with visual or multimodal feedback, they felt less positive emotions at the end of the training compared to the beginning of the training. The decrease in positive emotions seemed greater when participants were training with a multimodal feedback compared to when they were training with a visual feedback only (see Figure 7).

Mindfulness decreased significantly more over the session when participants were training with a visual feedback (M = −9.38, SD = 2.86) than when they were training with a multimodal feedback (M = −2.88, SD = 2.35). This is particularly visible during the first two sessions. During these sessions, the mindfulness increases between the beginning and the end of the session when the participants were training with multimodal feedback but greatly decreases when they were training with visual feedback. Overall, the mindfulness seems to decrease over the sessions when participants were training with a multimodal feedback whereas a large increase is visible between the third and the last sessions when participants were training with a visual feedback (see Figure 8).

Regardless of the modality of feedback that they were training with, and apart from the fourth session, participants’ motivation increased more when they started training with a visual feedback than when they started training with a multimodal feedback. Overall, the difference of motivation between the end and the beginning of the session seems to increase over the sessions (see Figure 9).

3.3. H3—Participants’ Profile

As presented in the introduction, previous studies have shown that feedback influence on learning is modulated by participants’ traits and states [13]. In these analyses, we focused on the autonomy, tension, mental rotation abilities and initial kinaesthetic imagery abilities of our participants.

We assessed if the autonomy, tension, mental rotation abilities or initial kinaesthetic imagery abilities of our participants had impacted their mean MI classification accuracy by performing analyses of correlation. There was no correlation between the mean MI classification accuracy over the sessions and the autonomy [Spearman correlation, r = .16, p = .55], tension [Pearson correlation, r = −.41, p = .12], mental rotation abilities [Pearson correlation, r = .03, p = .92] and initial kinaesthetic imagery abilities [Pearson correlation, r = −.26, p = .33].

3.4. Summary of the Results

All the significant results found in our statistical analyses are summarized in

Table 1. Summary of the significant results per hypothesis.

Hypothesis	Analyses	Significant Results
H1—MI-BCI performances	3-way repeated measures mixed ANOVA with “Modality”, “Order” and “Session” as independent variables and the repeated measures of MI classification accuracy as dependent variable	“Modality” [F(1, 14) = 8.47, p = .01, ${\eta }^2$ = .38] “Session” [F(2.22, 31.09) = 3.75, p = .03, ${\eta }^2$ = .2] “Order” [F(1, 14) = 7.02, p = .02, ${\eta }^2$ = .33]
H2—User experience	3-way repeated measures mixed ANOVAs with “Modality”, “Order” and “Session” as independent variables and one of the indicators of the user experience, that is, cognitive load, sense of agency, mood, mindfulness and motivation, as dependent variables	Influence of “Modality” on mindfulness [F(1, 14) = 5.85, p = .03, ${\eta }^2$ = .3] Influence of “Modality*Session” on sense of agency [F(4, 56) = 4.32, p < ${10}^{-2}$, ${\eta }^2$ = .24], mood [F(4, 56) = 3.77, $p<{10}^{-2}$, ${\eta }^2$ = .21] and mindfulness [F(4, 56) = 2.97, p = .03, ${\eta }^2$ = .18] Influence of “Session*Order” on motivation [F(4, 56) = 2.71, p = .04, ${\eta }^2$ = .16]
H3—Participants’ profile	Correlation analyses	-

below.

4. Discussion

In the following Subsections, we discuss the results obtained for our hypothesis. As the results regarding the MI-BCI performances and the user experience can provide insight on one another, they are discussed together in the following subsection.

4.1. H1—MI-BCI Performances & H2—User Experience

In terms of performance, our results showed some clear between-session variability, as well as between-subject variability. This is in line with the well documented between- and within-subject variability in MI-BCI [51]. Here, our participants seemed to have decreasing MI performances until the third or fourth session and then increasing ones during the remaining session(s). This could be related to the results found on the agency and mindfulness for which the lowest values are also found at the third and fourth sessions (see Figure 6 and Figure 8).

Interestingly, a significant and strong impact of the order of presentation was found on the MI performances. Participants that started training with visual feedback only had better performances than participants that started training with both visual and vibrotactile feedback. The order of presentation of the modalities of feedback was also found to influence the evolution of the motivation across session. Integrating information arising from two modalities of feedback while performing the task could be particularly challenging for a novice learner. Starting with both modalities at once could indeed be overwhelming and undermine the participants’ motivation. Thus, it might be better adapted to let participants learn how to process a unimodal feedback before transitioning to a more complex multimodal one. This result is in accordance with previous results from the literature indicating that the expertise of participants should be taken into account to adapt the user training [25,26].

Also, the mean MI classification accuracy was higher when participants were training with a visual and vibrotactile feedback than when they were training with a visual feedback only. This result is in accordance with the literature indicating that a multimodal feedback, with somatosensory and visual stimuli, has a better influence on BCI training than a unimodal visual one over one session [21,22]. Similar results were obtained by Barsotti et al. who found that proprioceptive stimuli, based on vibration patterns, in addition to a realistic visual feedback led to higher classification accuracy and more stable Event-Related Desynchronization (ERDs), that is, decrease in EEG activity following a motor imagery or execution task, than a realistic visual feedback alone over one session [23].

Our results do not indicate an influence of the session on the positive and significant difference of performances between a multimodal feedback composed of vibrotactile and visual stimuli and a unimodal visual feedback alone. The visual and tactile feedback lead to higher average classification accuracies in all five sessions except one (the fourth one). This suggests a robust positive effect of a vibrotactile and visual feedback compared to a visual one across sessions. There does not seem to be any desensitization to the vibrotactile feedback. The positive influence of a multimodal feedback could be caused by the redundancy and congruence of the information provided on different modalities of feedback [10,20].

This redundancy and congruence of the information provided by our multimodal feedback could have resulted from an increase in the sense of agency, that is, the subjective feeling of being able to control one’s own action (body agency), and through it, external events (external agency) [55]. Indeed, we found that the evolution of participants’ agency depended on the modality of feedback. It seems that overall participants felt more in control of the visual and tactile feedback than of the visual feedback alone during the first sessions. This is inverted for the fourth session and nonexistent for the last.

The difference of mindfulness pre and post sessions also provides an interesting insight that might explain why a multimodal visual and vibrotactile feedback seemed more effective than a unimodal visual one. The training have a negative impact on the reported state of mindfulness. The decrease of mindfulness is stronger when participants were training with a visual feedback only. Our results suggest that the feedback has a differential impact on the mindfulness depending on its modality of presentation. As mindfulness was associated with better MI-BCI performances, one hypothesis could be that this decrease in mindfulness could have had a detrimental impact on the performances [56,57].

The multi-session beneficial influence of a visual and vibrotactile feedback compared to a visual one tends to be modulated by the evolution of the difference of mood and mindfulness reported pre and post session. During the first sessions, the visual and vibrotactile feedback tended to have a better influence on the mood and mindfulness reported by our participants than the visual feedback alone. However, these tendencies seem to be reversed during the last sessions.

4.2. H3—Participants’ Profile

We did not find a correlation of the MI-BCI performances with the autonomy, tension and mental rotation abilities. The absence of correlation between the performances and the rotation abilities is in accordance with the results from Leeuwis et al. [44].

Finally, no influence of the initial visual and kinaesthetic imagery was found on the MI performances. This result is in accordance with the ones of Rimbert et al. [41], who found that the kinaesthetic and visual abilities might not be predictors of performances when classifying rest vs. hand movement imagery task. Both Rimbert et al. and us have used a realistic visual feedback. Previous experiments indicating an impact of visual and kinaesthetic imagery abilities on MI-BCI performances used either no feedback or an abstract feedback and their participants performed right versus left hand MI tasks [38,39]. Further studies taking into account the classified tasks and the modality of feedback are required to investigate the influence of initial visual and kinaesthetic imagery on BCI performances. An analysis of the strategies that the participants use to perform the motor imagery tasks might also provide more insight into these results.

4.3. Limitations

Our work presents some limitations, in particular, the number of participants included in the study. Even though our study is the result of a big investment in time and effort (16 participants × 10 sessions ×∼2 h experiment), and the study of vibrotactile BCI feedback with the largest number of sessions to date, future experiments with a greater number of participants are necessary to confirm our results. Another limitation could arise from the number of training sessions per modality of feedback. While such a number of sessions is relatively high compared to previous BCI studies comparing feedback modalities, as our participants trained with both modalities of feedback, they only trained during five sessions with each modality. A higher number of sessions per modality might provide more insight on the multi-session impact of both modalities. The number of participants included and the number of sessions per modalities are related issues. Both of them arise from a compromise between the relevance of the results and the time needed to perform the experiments.

Other limitations emerge from the combination of instructions and feedback that participants were provided with. Using a realistic feedback and providing the participants with the instruction to imagine a similar movement as the one performed by the virtual hand during feedback aimed at eliciting a greater sense of agency from the participants [58]. However, the temporal and spatial asynchrony occurring when the virtual arms were not perfectly aligned with the participants’ arms and when the task was not perfectly recognized by the MI-BCI system might have decreased the sense of agency of our participants [59]. Low performers had the larger discrepancy between their motor imagery and the visual feedback they received. This might have particularly impeded their learning.

Finally, all the participants were asked to perform both motor execution and imagination tasks. Even though the participants did not perform these tasks with the same hand, and used a different BCI classifier for each hand, the performance of motor execution tasks might have influenced the results that we obtained with the motor imagery BCI. Indeed, motor execution and imagery tasks do activate similar patterns of neurons and some are shared between opposite limbs of the body [60]. The performance of motor execution tasks might have primed the activation of the sensorimotor cortex and influenced its activation during motor imagery tasks [60]. However, all subjects performed the motor tasks, and did so in all conditions. Thus, if there were any influence of motor tasks, this influence was the same for all conditions, thus making a confounding effects of motor tasks unlikely.

5. Conclusions and Prospect

In this experiment, the BCI participants trained with both a realistic visual feedback only and a vibrotactile feedback on their wrist in addition to this same visual feedback. The order of presentation of the modalities of feedback was balanced over our participants. Our goal was to assess the impact of the modality of feedback and its order of presentation over the evolution across multiple sessions of the BCI performances and the user experience. We also wanted to assess if characteristics of the profile of participants, particularly the visual and kinaesthetic imagery abilities, modulated the influence of the feedback modality.

We found that using a vibrotactile feedback in addition to a realistic visual feedback has a beneficial influence on MI-BCI performances. This result is in accordance with previous ones, obtained from neurotypical participants, demonstrating that a multimodal feedback seems to be preferable to a unimodal one for performances obtained over a single session of user training [21,22,23]. The results from our experiment indicate that this beneficial impact seems to remain true for multi-session training, which had not been tested before. They also indicate that the multimodal feedback can include tactile stimuli instead of proprioceptive stimuli and still remain more efficient than a visual feedback only. To our knowledge, only Shu et al. used vibrotactile feedback for post-stroke motor rehabilitation [61]. It might represent an acceptable and less expensive alternative to feedback based on FES or orthesis.

Interestingly, our results regarding the order of presentation and the user experience tend to modulate the differences found between our multimodal and unimodal feedback. The order of presentation of the modalities of feedback was found to have an influence on MI classification performances and on the evolution of the users’ motivation. People that started training with a visual feedback had higher MI-BCI performances and were less demotivated after a training session compared to people that started training with both a visual and vibrotactile feedback. We hypothesize that the difference of skills between the use of the first and second modality of feedback could explain their differential influence on the performances and motivation. The skill level was found to influence the modality of feedback to favour for motor skill user training [14]. Furthermore, the results obtained for the user experience, that is, the evolution of mood and mindfulness over the sessions, seem to progressively favour the visual feedback alone over the sessions.

In summary, for future BCI studies, we would recommend the use of a visual feedback for naive users and then of a multimodal feedback, once the learners have acquired some skills to interpret the feedback. The use of a vibrotactile feedback seems to be an acceptable and less expensive alternative to a proprioceptive feedback. Future studies are necessary to assess the potential differential impact of a proprioceptive feedback compared to a vibrotactile one.