3. Study Design
Patients who had balance disorders and were attending a VRh program were the potential participants in this study. The diagnoses of these patients include stroke, multiple sclerosis, meningioma, subdural hematoma, cervical myelopathy, Guillain-Barré syndrome, Parkinson disease, brain tumors, and vestibular pathology.
The inclusion criteria were: (1) signed written informed consent; (2) age >17; (3) balance disorders evidenced by clinical balance scales; (4) normal cognitive functions; (5) ability to follow instructions (mini-mental state examination [41
] >24). The exclusion criteria were: (1) dementia; (2) hemispatial neglect; (3) visual deficit; (4) severe hearing impairment; (5) unsolved acute trauma injury.
3.2. Study Interventions
In the study, the USEQ is used to test satisfaction of the ABAR system [42
]. The ABAR system is a VRh system that is specifically designed to recover balance (Figure 1
ABAR integrates the Nintendo® Wii Balance Board (WBB) (Nintendo, Kyoto, Japan) for the interaction of the patient. WBB is a low-cost, widely-available device that allows the center-of-pressure of the patient to be obtained.
Five different games can be selected in ABAR to recover balance, in both sitting and the standing positions.
3.3. Study Procedures
The study was conducted in a specialized rehabilitation facility of a hospital under clinical supervision. Each patient completed the USEQ after the first session with ABAR. Each session with the system lasted 30 min; each session mixed periods of playing and resting, according to the specialist’s indications. A member of our team was with the patients while they were answering the questionnaire.
3.4. Outcome Measures
The primary outcome measures were provided by the questionnaire. The scores for the questions of the USEQ allowed us to carry out the statistical analysis as described in Section 4
Secondary measures were obtained when the patients completed the USEQ. At this stage, we had informal conversations about the USEQ with patients after the completion of the questionnaire. Although the informal conversations with patients are a subjective source of information, they provided us with responses to questions that are related to perceived questionnaire length or perceived questionnaire difficulty.
3.5. Data Analysis
Data analysis was carried out with SPSS for Windows, version 15 (SPSS Inc., Chicago, IL, USA) on a standard PC. To test the internal consistency reliability, we used Cronbach’s alpha [43
]. Cronbach’s alpha is a coefficient of internal consistency that is commonly used to estimate the reliability of a test.
For sampling adequacy, the Kaiser-Meyer-Olkin (KMO) index and Bartlett’s test of sphericity were calculated. While the KMO index ranges from 0 to 1, adequate sample size is accepted for a value over 0.5. For factor analysis to be considered suitable, Bartlett’s test of sphericity must be less than 0.05.
To identify the factor structure of the USEQ, we carried out an exploratory factor analysis (analysis of principal components), retaining components with eigenvalues greater than 1; in addition, we carried out a scree plot inspection. For the correlations between the items and the factor, we used unrotated factor loadings above 0.3.
5. Discussion and Conclusions
Despite the fact that virtual rehabilitation is an emerging field that shows great potential, with many studies in recent years, there are no specific usability or satisfaction questionnaires with validated internal consistency for virtual rehabilitation systems. The USEQ is a questionnaire that is designed to evaluate satisfaction, which is part of usability, in virtual rehabilitation systems. This study has addressed the factor structure and internal consistency of the USEQ. Analysis of item-total correlation (Table 3
) suggested that all items correlated well with the overall scale because all of the values were above 0.3. Therefore, the results suggest that the USEQ is a reliable questionnaire.
With regard to the Cronbach’s alpha evaluation, Cronbach explains in [43
] how this coefficient should be interpreted. Cronbach indicates that alpha values greater than or equal to 0.7, in general indicate good internal consistency. On the other hand, if Cronbach’s alpha is too high, it may suggest redundancies because some items are testing the same question but in a different way. Streiner [45
] recommends a maximum alpha value of 0.9.
In general, although no accurate ranges exist to classify the Cronbach alpha coefficient, an alpha coefficient ranging between 0.7 and 0.9 is considered to be acceptable. The Cronbach alpha coefficient for the USEQ was 0.716; therefore, this indicates adequate internal consistency.
The increase in Cronbach’s alpha when an item is deleted indicates that the item could probably be removed from the scale. For the USEQ, the results of the study showed that the Cronbach alpha values if the item is deleted were minor for all six items (Table 3
). Therefore, we kept all of the questions of the USEQ.
Regardless of the fact that an analysis of the principal components showed two factors with an eigenvalue greater than 1, the features of the second factor (Table 4
) and the scree plot inspection (Figure 2
) suggest that only the first factor (which includes all six questions) can be considered to be appropriate. Therefore, a one-factor solution was considered to be appropriate, which accounted for 42.869% of the total variance. As shown in Table 4
, all items had a correlation greater than 0.4 with the factor, which implies that they are probably meaningful. Thus, the factor under consideration represents ‘user satisfaction’ because all of the items were designed to measure user satisfaction with the system, and it has a Cronbach alpha coefficient of 0.716.
Other similar studies that evaluate the internal consistency of tests with good results show comparable values.
], the authors evaluated HARUS (Handheld Augmented Reality Usability Scale), a questionnaire composed of 16 statements where users rate their agreement by using a seven-point Likert scale. HARUS has a two-factor structure. Statements 1 to 8 are measures of manipulability, while statements 9 to 16 are measures of comprehensibility. The authors confirm that the separate manipulability and comprehensibility scales have good internal consistency because they obtained alpha values between 0.71 and 0.83 in all their experiments.
Fackrell et al. [47
] evaluated the validity and reliability of the Hyperacusis Questionnaire. The authors also specify α > 0.7 as reliability criteria for the scales evaluated.
Although the Cronbach alpha coefficient in this study is acceptable (0.716), it is just above the lower value of the range of values considered to be adequate (0.7 ≤ α ≤ 0.9). Several factors related to the sample may have influenced this result. We hypothesize that the size of the sample and/or the age of the patients (mostly elderly) could have influenced the result. As we suggest later, further studies with different samples can allow us to check how these factors influence the results.
In the study, the patients observed that the USEQ has a convenient number of questions. They also considered the questions in the USEQ to be clear. This perception of the questionnaire by the patients is especially interesting considering the age of the patients enrolled in the study (most were elderly) and their level of studies (67.50% had only primary studies or less). With younger patients, and/or with patients with a higher level of studies, it is expected that the perception of the questionnaire will be even more favorable.
With regard to the results for the evaluated system (ABAR), the USEQ score was 25.80. Since the scale ranges from 6 (lower satisfaction) to 30 (higher user satisfaction), the score suggests that the satisfaction perceived by patients was very high. With regard to the characteristics of the sample, it is necessary to pay attention to some points. The sample includes a heterogeneous population from the point of view of their disabilities. This is due to the VRh system used for the evaluation: ABAR is a VRh system designed to help in the recovery of balance, and there are many different disabilities that cause balance disorders. Moreover, these disabilities are most common among the elderly and, consequently, the patients of the sample tend to be seniors. Future studies that evaluate the USEQ with a younger population and/or with a homogeneous population with respect to their disabilities would be interesting, in order to compare their results to the results of this study.
With regard to the sample size, the main drawback of the study is that it is a bit limited to perform factor analytic studies. However, but it was impossible to add more patients that fulfilled the inclusion–exclusion criteria. In the future, further studies with larger samples will allow the results of this study to be compared.
Based on the results obtained, we do not suggest any changes in the items of the USEQ. From an objective point of view, this is for especially two reasons. First, the results confirmed a one-factor solution and adequate internal consistency. Second, the results support that all the items are necessary: first, because the results of the study showed that the Cronbach alpha values, if an item was deleted, were minor for all six items, and, second, because the Cronbach alpha coefficient is below 0.9 (a high value of alpha may suggest redundancies and indicate that the length of the questionnaire should be shortened). From a subjective point of view, the positive perception of the USEQ by the patients support our suggestion.
In summary, the USEQ presented in this study is an easy-to-understand questionnaire that has an appropriate number of questions that are correlated with each other. The USEQ is a reliable and useful tool for properly evaluating the satisfaction of the user (which is part of usability).