Next Article in Journal
Ultrasensitive Magnetic Field Sensing Based on Refractive-Index-Matched Coupling
Next Article in Special Issue
SNORAP: A Device for the Correction of Impaired Sleep Health by Using Tactile Stimulation for Individuals with Mild and Moderate Sleep Disordered Breathing
Previous Article in Journal
GNSS-R Altimetry Performance Analysis for the GEROS Experiment on Board the International Space Station
Previous Article in Special Issue
A Hybrid FPGA-Based System for EEG- and EMG-Based Online Movement Prediction
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Sensors
Volume 17
Issue 7
10.3390/s17071589
sensors-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

USEQ: A Short Questionnaire for Satisfaction Evaluation of Virtual Rehabilitation Systems

by
José-Antonio Gil-Gómez
1,*,
Pilar Manzano-Hernández
2,
Sergio Albiol-Pérez
3,
Carmen Aula-Valero
2,
Hermenegildo Gil-Gómez
1 and
José-Antonio Lozano-Quilis
1
1
Instituto Universitario de Automática e Informática Industrial, Universitat Politècnica de València, Camino de Vera s/n, 46022 Valencia, Spain
2
Hospital S. José, Av. Zaragoza 16, 44001 Teruel, Spain
3
Aragón Health Research Institute (IIS Aragón), Universidad de Zaragoza, Ciudad Escolar, 44003 Teruel, Spain
*
Author to whom correspondence should be addressed.
Sensors 2017, 17(7), 1589; https://doi.org/10.3390/s17071589
Submission received: 28 April 2017 / Revised: 3 July 2017 / Accepted: 4 July 2017 / Published: 7 July 2017
(This article belongs to the Special Issue Sensors and Analytics for Precision Medicine)
Browse Figures
Versions Notes

Abstract

:
New emerging technologies have proven their efficacy in aiding people in their rehabilitation. The tests that are usually used to evaluate usability (in general) or user satisfaction (in particular) of this technology are not specifically focused on virtual rehabilitation and patients. The objective of this contribution is to present and evaluate the USEQ (User Satisfaction Evaluation Questionnaire). The USEQ is a questionnaire that is designed to properly evaluate the satisfaction of the user (which constitutes part of usability) in virtual rehabilitation systems. Forty patients with balance disorders completed the USEQ after their first session with ABAR (Active Balance Rehabilitation), which is a virtual rehabilitation system that is designed for the rehabilitation of balance disorders. Internal consistency analysis and exploratory factor analysis were carried out to identify the factor structure of the USEQ. The six items of USEQ were significantly associated with each other, and the Cronbach alpha coefficient for the questionnaire was 0.716. In an analysis of the principal components, a one-factor solution was considered to be appropriate. The findings of the study suggest that the USEQ is a reliable questionnaire with adequate internal consistency. With regard to patient perception, the patients found the USEQ to be an easy-to-understand questionnaire with a convenient number of questions.

1. Introduction

1.1. Usability

Usability is an important quality attribute of a user’s experience when interacting with a system or tool, and it is also an important attribute in helping users to achieve the suggested goals [1]. With regard to HCI (Human–Computer Interface) and usability, Bevan states in [2] that standards related to usability can be categorized as being primarily concerned with the use of the product (effectiveness, efficiency, and satisfaction in a specific context of use).
The categorization of Bevan is coherent with the ISO 9241-11 standard [3,4,5], which describes a widely accepted definition of usability. This standard indicates the rules that are needed in terms of ergonomics, hardware, software, and environments in order to obtain good usability for a product or system. Section 8.1 describes the term usability as “the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use”.

1.2. Usability in Virtual Rehabilitation

One of the promising and emerging fields within rehabilitation therapies for different pathologies is virtual rehabilitation (VRh) [6,7,8,9]. VRh systems are designed to assist clinical specialists and patients in the rehabilitation process [10]. The use of ground-breaking technologies together with the emergence of entertaining and playful virtual environments (VE) have demonstrated promising results in the rehabilitation process [11,12,13,14], improving the adherence to treatments [12]. However, these systems should be tested regarding important aspects such as usability.
Currently, there are different questionnaires that are designed to evaluate usability in general-purpose systems. The best-known usability questionnaire is the system usability scale (SUS) [15,16], which measures the feeling of usability of the users when using computer systems. It is composed of 10 questions with a five-point Likert attitude scale (from strongly disagree to strongly agree). This questionnaire has been used in different domains such as: security software [17], mobile phones [18,19], PDA [20], Social Network sites [21,22], wiki sites [23], serious games [24], or robotics [25]. Even though the SUS questionnaire is not specifically designed for VRh systems, it has also been used for rehabilitation purposes due to the lack of questionnaires that focus on VRh systems. Meldrum et al. [26] tested balance in patients with vestibular and other neurological diseases using VRh and quantified the usability of the Nintendo Wii Fit Plus®. Duvinage et al. [27] assessed the usability of a P300 system (using Brain–Computer interfaces) for lower-limb rehabilitation purposes. One considerable advantage of the SUS questionnaire is the reasonable number of questions that are to be answered at the end of the first session. However, the concepts of this questionnaire are too generic (computers, PDAs, Websites, etc.). The main drawback of the SUS questionnaire is that it does not include questions to obtain responses about specific items related to Virtual Rehabilitation.
Another well-known usability questionnaire is VRUSE [28]. Fitzgerald et al. [29] assessed the usability of the E-Yoga system using VRUSE, with the goal of improving postural control and biomechanical alignment of the subjects in a rehabilitation process. The VRUSE evaluates a wide range of concepts: functionality, user input, system output (display), user guidance and help, consistency, flexibility, simulation fidelity, error correction/handling and robustness, sense of immersion/presence, and overall system usability. The main drawback of this test is the large number of questions that the patients are required to answer [28]: the complete questionnaire has 100 questions. This drawback is especially important if the patients involved in a rehabilitation process have neurological and/or cognitive disorders. Other simplified usability questionnaires for VRh with reasonable outcomes are described in [30,31,32,33], but the drawback of these questionnaires is that the internal consistency has not yet been validated.
Kizony et al. [34] published the Short Feedback Questionnaire (SFQ), which is a questionnaire that is related to Witmer and Singer’s Presence Questionnaire [35]. It is composed of eight questions with a five-point Likert attitude scale, and it has been used in virtual reality environments [36,37,38]. The SFQ questionnaire evaluates the user’s sense of presence, perceived difficulty of the task, and any discomfort that users may have felt during the experience. This questionnaire does not focus on VRh systems.
To our knowledge, there are no validated questionnaires for testing usability or satisfaction of virtual rehabilitation systems. A questionnaire for this purpose must have a reasonable number of questions and internal consistency reliability.
Following the definitions of usability in [2,3,4,5], usability can be divided into three components: efficiency, effectiveness, and satisfaction. Focusing on VRh, efficiency and effectiveness can usually be measured through a clinical trial. With a classical clinical trial, we can compare an experimental group (using a VRh system) with a control group (following a traditional rehabilitation program) by evaluating efficacy and comparing the recovery level of the two groups. With regard to effectiveness, we can measure, for instance, the number of sessions that each group needs to reach a certain level. However, the third component of usability, satisfaction, cannot be evaluated in the same way as efficiency and effectiveness: a reliable and consistent questionnaire (with an adequate number of questions) is necessary to measure the satisfaction of the users.
The aim of the present study is to introduce the USEQ, a user satisfaction questionnaire that is specifically designed to evaluate satisfaction with virtual rehabilitation systems, and to validate their reliability by analyzing their internal consistency.

2. USEQ: The User Satisfaction Evaluation Questionnaire

2.1. SEQ: The Suitability Evaluation Questionnaire

In [39], the SEQ was introduced as a 14-question questionnaire that is designed to test items such as satisfaction, acceptance, and security of use in virtual rehabilitation systems. The SEQ was designed by a multidisciplinary team of clinical and technical experts. Factors such as the length of the questionnaire, the type of questions to be asked and what to ask were taken into account in the design of the questionnaire. For the length of the questionnaire, the clinical experts that collaborated in the design of the SEQ estimated that a maximum of 15 questions would be an acceptable length for patients.
For the type of questions, the designers of the SEQ considered 13 questions with a five-point Likert Scale, plus an open-ended question offering patients the possibility to add comments if necessary. The SEQ has a five-point Likert Scale questions (instead of other options such as seven-point Likert Scale questions) because the authors considered five options of answers to be good enough, and, also, it is coherent with the main usability questionnaires that are currently being used: SUS [15], VRUSE [28], and SFQ [34] also use five-point Likert Scale questions.
For what to ask about, the designers of the SEQ composed the questions taking into account the usability questionnaires available and their own experience, both in the technical and in the clinical field.
A previous study evaluating the suitability of virtual rehabilitation for the elderly was carried out using the SEQ [40]. The SEQ was used to evaluate the ABAR (Active Balance Rehabilitation) system, the VRh system that is used in this study. The study presented in [40] allowed the evaluation of the perceived length and difficulty of the SEQ. In [40], the patients completed the questionnaire without any problems. None of the patients considered the questionnaire to be too long. The main drawback of SEQ is that it is composed of different dimensions; therefore, it is not possible to evaluate their internal consistency.

2.2. USEQ: Questions

The USEQ questionnaire is composed of the set of questions in the SEQ that evaluate satisfaction. The USEQ has six questions with a five-point Likert Scale. The questions and their scores are shown in Table 1.
The total score of the USEQ questionnaire ranges from 6 (poor satisfaction) to 30 (excellent satisfaction). To calculate this total score, we consider all of the questions to be positive, except for Q5, which is considered to be a negative question. The numerical value of the positive questions is used to calculate the score (for instance, if the patient selects 4 in Q1, then 4 is added to the total score). The negative question subtracts the numerical value of the response from 6 and then adds this result to the total score (for instance, if the patient selects 2 in Q5, then 4 is added to the total score).

3. Study Design

3.1. Subjects

Patients who had balance disorders and were attending a VRh program were the potential participants in this study. The diagnoses of these patients include stroke, multiple sclerosis, meningioma, subdural hematoma, cervical myelopathy, Guillain-Barré syndrome, Parkinson disease, brain tumors, and vestibular pathology.
The inclusion criteria were: (1) signed written informed consent; (2) age >17; (3) balance disorders evidenced by clinical balance scales; (4) normal cognitive functions; (5) ability to follow instructions (mini-mental state examination [41] >24). The exclusion criteria were: (1) dementia; (2) hemispatial neglect; (3) visual deficit; (4) severe hearing impairment; (5) unsolved acute trauma injury.

3.2. Study Interventions

In the study, the USEQ is used to test satisfaction of the ABAR system [42]. The ABAR system is a VRh system that is specifically designed to recover balance (Figure 1).
ABAR integrates the Nintendo® Wii Balance Board (WBB) (Nintendo, Kyoto, Japan) for the interaction of the patient. WBB is a low-cost, widely-available device that allows the center-of-pressure of the patient to be obtained.
Five different games can be selected in ABAR to recover balance, in both sitting and the standing positions.

3.3. Study Procedures

The study was conducted in a specialized rehabilitation facility of a hospital under clinical supervision. Each patient completed the USEQ after the first session with ABAR. Each session with the system lasted 30 min; each session mixed periods of playing and resting, according to the specialist’s indications. A member of our team was with the patients while they were answering the questionnaire.

3.4. Outcome Measures

The primary outcome measures were provided by the questionnaire. The scores for the questions of the USEQ allowed us to carry out the statistical analysis as described in Section 4.
Secondary measures were obtained when the patients completed the USEQ. At this stage, we had informal conversations about the USEQ with patients after the completion of the questionnaire. Although the informal conversations with patients are a subjective source of information, they provided us with responses to questions that are related to perceived questionnaire length or perceived questionnaire difficulty.

3.5. Data Analysis

Data analysis was carried out with SPSS for Windows, version 15 (SPSS Inc., Chicago, IL, USA) on a standard PC. To test the internal consistency reliability, we used Cronbach’s alpha [43]. Cronbach’s alpha is a coefficient of internal consistency that is commonly used to estimate the reliability of a test.
For sampling adequacy, the Kaiser-Meyer-Olkin (KMO) index and Bartlett’s test of sphericity were calculated. While the KMO index ranges from 0 to 1, adequate sample size is accepted for a value over 0.5. For factor analysis to be considered suitable, Bartlett’s test of sphericity must be less than 0.05.
To identify the factor structure of the USEQ, we carried out an exploratory factor analysis (analysis of principal components), retaining components with eigenvalues greater than 1; in addition, we carried out a scree plot inspection. For the correlations between the items and the factor, we used unrotated factor loadings above 0.3.

4. Results

4.1. Sample Characteristics

In the study period, 198 patients who had balance problems and were attending a rehabilitation program in the clinical facilities were the potential participants in this study. Of these, 108 patients had unsolved acute trauma injuries, 36 patients had cognitive impairment, 5 patients had visual deficit or severe hearing impairment, and 9 patients refused to participate in the study. A final sample of 40 patients fulfilled the inclusion–exclusion criteria, and were included in the study.
Table 2 shows information summarizing the characteristics of the patients.
In the final sample, 19 patients were male and 21 patients were female. Most of the patients were elderly: 80% of the patients were older than 65 years old and the average age was 74.35 (SD 14.59) years old. Based on the post-injury time [44], 10 patients were post-acute (0–5 months post-injury), 11 patients were acute (6–23 months post-injury), and 19 patients were chronic (24 months or more post-injury). Most of the patients came from an urban background (75%), and 25% came from a rural background. With regard to the level of studies, 85% of the patients had completed primary (60%) or secondary (25%) studies whereas only three patients had completed higher studies and only three patients had not completed any studies.

4.2. USEQ Scores

The results corresponding to the USEQ question evaluation are presented in Table 3.
The mean USEQ score was 25.80 (SD 3.589). The scale mean if the item is deleted was measured for all the items, ranging from 21.10 to 22.00.

4.3. Item-Total Correlation and Cronbach’s Alpha

To carry out the item analysis for selecting items for inclusion in the scale, we used the corrected item-total correlation. In this way, we avoid the problem of performing the correlation of an item with the total of the scale, considering that this total includes the value of the item whereby that correlation would be skewed. The corrected item-total correlation values ranged from 0.321 to 0.666 (Table 3).
The Cronbach’s alpha value for the complete scale was 0.716. Cronbach’s alpha if the item is deleted was calculated for all six items, and none of the values were above Cronbach’s alpha for complete scale (Table 3).

4.4. Factor Structure

The KMO index of sampling adequacy was 0.60. Bartlett’s test of sphericity was significant (p < 0.001). The analysis of principal components indicated two components with an eigenvalue greater than 1, which accounted for 65.777% of the total variance. The first of the components had all six items (Table 4), and it explains 42.869% of the variance; the items of the first component had a correlation with the factor that ranged from 0.468 to 0.816. The second component had only four items, only two of which had factor loadings greater than 0.5 (Table 4).
The scree plot (Figure 2) did not reveal a clear point of inflexion, but the sharpest angle is placed in the second component.

4.5. Informal Conversations

The informal interview after the completion of the questionnaire is a subjective source of data, but it is always very interesting to know the opinions of patients. The patients considered that the questionnaire was short and they were not distracted when answering the questions. The patients also perceived the USEQ to be an easy-to-understand questionnaire.

5. Discussion and Conclusions

Despite the fact that virtual rehabilitation is an emerging field that shows great potential, with many studies in recent years, there are no specific usability or satisfaction questionnaires with validated internal consistency for virtual rehabilitation systems. The USEQ is a questionnaire that is designed to evaluate satisfaction, which is part of usability, in virtual rehabilitation systems. This study has addressed the factor structure and internal consistency of the USEQ. Analysis of item-total correlation (Table 3) suggested that all items correlated well with the overall scale because all of the values were above 0.3. Therefore, the results suggest that the USEQ is a reliable questionnaire.
With regard to the Cronbach’s alpha evaluation, Cronbach explains in [43] how this coefficient should be interpreted. Cronbach indicates that alpha values greater than or equal to 0.7, in general indicate good internal consistency. On the other hand, if Cronbach’s alpha is too high, it may suggest redundancies because some items are testing the same question but in a different way. Streiner [45] recommends a maximum alpha value of 0.9.
In general, although no accurate ranges exist to classify the Cronbach alpha coefficient, an alpha coefficient ranging between 0.7 and 0.9 is considered to be acceptable. The Cronbach alpha coefficient for the USEQ was 0.716; therefore, this indicates adequate internal consistency.
The increase in Cronbach’s alpha when an item is deleted indicates that the item could probably be removed from the scale. For the USEQ, the results of the study showed that the Cronbach alpha values if the item is deleted were minor for all six items (Table 3). Therefore, we kept all of the questions of the USEQ.
Regardless of the fact that an analysis of the principal components showed two factors with an eigenvalue greater than 1, the features of the second factor (Table 4) and the scree plot inspection (Figure 2) suggest that only the first factor (which includes all six questions) can be considered to be appropriate. Therefore, a one-factor solution was considered to be appropriate, which accounted for 42.869% of the total variance. As shown in Table 4, all items had a correlation greater than 0.4 with the factor, which implies that they are probably meaningful. Thus, the factor under consideration represents ‘user satisfaction’ because all of the items were designed to measure user satisfaction with the system, and it has a Cronbach alpha coefficient of 0.716.
Other similar studies that evaluate the internal consistency of tests with good results show comparable values.
In [46], the authors evaluated HARUS (Handheld Augmented Reality Usability Scale), a questionnaire composed of 16 statements where users rate their agreement by using a seven-point Likert scale. HARUS has a two-factor structure. Statements 1 to 8 are measures of manipulability, while statements 9 to 16 are measures of comprehensibility. The authors confirm that the separate manipulability and comprehensibility scales have good internal consistency because they obtained alpha values between 0.71 and 0.83 in all their experiments.
Fackrell et al. [47] evaluated the validity and reliability of the Hyperacusis Questionnaire. The authors also specify α > 0.7 as reliability criteria for the scales evaluated.
Although the Cronbach alpha coefficient in this study is acceptable (0.716), it is just above the lower value of the range of values considered to be adequate (0.7 ≤ α ≤ 0.9). Several factors related to the sample may have influenced this result. We hypothesize that the size of the sample and/or the age of the patients (mostly elderly) could have influenced the result. As we suggest later, further studies with different samples can allow us to check how these factors influence the results.
In the study, the patients observed that the USEQ has a convenient number of questions. They also considered the questions in the USEQ to be clear. This perception of the questionnaire by the patients is especially interesting considering the age of the patients enrolled in the study (most were elderly) and their level of studies (67.50% had only primary studies or less). With younger patients, and/or with patients with a higher level of studies, it is expected that the perception of the questionnaire will be even more favorable.
With regard to the results for the evaluated system (ABAR), the USEQ score was 25.80. Since the scale ranges from 6 (lower satisfaction) to 30 (higher user satisfaction), the score suggests that the satisfaction perceived by patients was very high. With regard to the characteristics of the sample, it is necessary to pay attention to some points. The sample includes a heterogeneous population from the point of view of their disabilities. This is due to the VRh system used for the evaluation: ABAR is a VRh system designed to help in the recovery of balance, and there are many different disabilities that cause balance disorders. Moreover, these disabilities are most common among the elderly and, consequently, the patients of the sample tend to be seniors. Future studies that evaluate the USEQ with a younger population and/or with a homogeneous population with respect to their disabilities would be interesting, in order to compare their results to the results of this study.
With regard to the sample size, the main drawback of the study is that it is a bit limited to perform factor analytic studies. However, but it was impossible to add more patients that fulfilled the inclusion–exclusion criteria. In the future, further studies with larger samples will allow the results of this study to be compared.
Based on the results obtained, we do not suggest any changes in the items of the USEQ. From an objective point of view, this is for especially two reasons. First, the results confirmed a one-factor solution and adequate internal consistency. Second, the results support that all the items are necessary: first, because the results of the study showed that the Cronbach alpha values, if an item was deleted, were minor for all six items, and, second, because the Cronbach alpha coefficient is below 0.9 (a high value of alpha may suggest redundancies and indicate that the length of the questionnaire should be shortened). From a subjective point of view, the positive perception of the USEQ by the patients support our suggestion.
In summary, the USEQ presented in this study is an easy-to-understand questionnaire that has an appropriate number of questions that are correlated with each other. The USEQ is a reliable and useful tool for properly evaluating the satisfaction of the user (which is part of usability).

Acknowledgments

The authors wish to thank the patients and the clinical specialists from San José Hospital (Teruel, Spain) for their collaboration, especially M. Forcano, MD.

Author Contributions

J.G., H.G. and J.L. designed the questionnaire. S.A., P.M., C.A. and J.G. designed the study. P.M. or C.A. were with the patients while they were answering the questionnaire and acquired the subjective responses. S.A., J.G., P.M. and C.A. designed the ABAR System. J.G., S.A., H.G. and J.L. contributed to the analysis and interpretation of the results.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Nielsen, J. Usability Engineering; Morgan Kaufmann Publishers Inc.: San Francisco, CA, USA, 2008. [Google Scholar]
  2. Bevan, N. International standards for HCI and usability. Int. J. Hum. Comput. Stud. 2001, 55, 533–552. [Google Scholar] [CrossRef]
  3. Abran, A.; Khelifi, A.; Suryn, W.; Seffah, A. Usability meanings and interpretations in ISO standards. Softw. Qual. J. 2003, 11, 325–338. [Google Scholar] [CrossRef]
  4. ISO 9241–11. Ergonomic Requirements for Office Work with Visual Display Terminals (VDTs) Part 11: Guidance on Usability. Available online: http://www.iso.org/iso/catalogue_detail.htm?csnumber=16883 (accessed on 25 April 2017).
  5. Jokela, T.; Iivari, N.; Matero, J.; Karukka, M. The standard of user-centered design and the standard definition of usability: analyzing ISO 13407 against ISO 9241-11. In Proceedings of the Latin American Conference on Human-Computer Interaction, Rio de Janeiro, Brazil, 17–20 August 2003; ACM New York: New York, NY, USA, 2003; pp. 53–60. [Google Scholar]
  6. Holden, M.K. Virtual environments for motor rehabilitation: Review. Cyberpsychol. Behav. 2005, 8, 187–211. [Google Scholar] [CrossRef] [PubMed]
  7. Sveistrup, H. Motor rehabilitation using virtual reality. J. Neuroeng. Rehabil. 2004, 1, 1–10. [Google Scholar] [CrossRef] [PubMed]
  8. Lange, B.; Koenig, S.; Chang, C.Y.; McConnell, E.; Suma, E.; Bolas, M.; Rizzo, A. Designing informed game-based rehabilitation tasks leveraging advances in virtual reality. Disabil. Rehabil. 2012, 34, 1863–1870. [Google Scholar] [CrossRef] [PubMed]
  9. Lewis, G.N.; Rosie, J.A. Virtual reality games for movement rehabilitation in neurological conditions: How do we meet the needs and expectations of the users? Disabil. Rehabil. 2012, 34, 1880–1886. [Google Scholar] [CrossRef] [PubMed]
  10. Singh, D.K.; Mohd Nordin, N.A.; Abd Aziz, N.A.; Lim, B.K.; Soh, L.C. Effects of substituting a portion of standard physiotherapy time with virtual reality games among community-dwelling stroke survivors. BMC Neurol. 2013, 13, 199. [Google Scholar] [CrossRef] [PubMed]
  11. Green, D.; Wilson, P.H. Use of virtual reality in rehabilitation of movement in children with hemiplegia-a multiple case study evaluation. Disabil. Rehabil. 2012, 34, 593–604. [Google Scholar] [CrossRef] [PubMed]
  12. Meldrum, D.; Herdman, S.; Moloney, R.; Murray, D.; Duffy, D.; Malone, K.; French, H.; Hone, S.; Conroy, R.; McConn-Walsh, R. Effectiveness of conventional versus virtual reality based vestibular rehabilitation in the treatment of dizziness, gait and balance impairment in adults with unilateral peripheral vestibular loss: a randomized controlled trial. BMC Ear Nose Throat Disord. 2012, 12, 3. [Google Scholar] [CrossRef] [PubMed]
  13. Gil-Gómez, J.A.; Lloréns, R.; Alcañiz, M.; Colomer, C. Effectiveness of a Wii balance board-based system (eBaViR) for balance rehabilitation: A pilot randomized clinical trial in patients with acquired brain injury. J. NeuroEng. Rehabil. 2011, 8, 1–10. [Google Scholar] [CrossRef] [PubMed]
  14. Albiol-Pérez, S.; Gil-Gómez, J.A.; Lloréns, R.; Alcañiz, M.; Font, C.C. The role of virtual motor rehabilitation: A quantitative analysis between acute and chronic patients with acquired brain injury. IEEE J. Biomed. Health Inform. 2014, 18, 391–398. [Google Scholar] [CrossRef] [PubMed]
  15. Brooke, J. SUS: A “quick and dirty” usability scale. In Usability Evaluation in Industry; Jordan, P.W., Thomas, B., Weerdmeester, B.A., McClelland, A.L., Eds.; Taylor and Francis: London, UK, 1996; pp. 189–194. [Google Scholar]
  16. Brooke, J. SUS: A Retrospective. J. Usabil. Stud. 2013, 8, 29–40. [Google Scholar]
  17. DeWitt, A.J.; Kuljis, J. Aligning usability and security: A usability study of Polaris. In Proceedings of the 2nd Symposium on Usable Privacy and Security, Pittsburgh, PA, USA, 12–14 July 2006; ACM: New York, NY, USA, 2006; pp. 1–7. [Google Scholar]
  18. Jokela, T.; Koivumaa, J.; Pirkola, J.; Salminen, P.; Kantola, N. Methods for quantitative usability requirements: a case study on the development of the user interface of a mobile phone. Pers. Ubiquitous Comput. 2006, 10, 345–355. [Google Scholar] [CrossRef]
  19. Vermeulen, J.; Neyens, J.C.; Spreeuwenberg, M.D.; van Rossum, E.; Sipers, W.; Habets, H.; Hewson, D.J.; de Witte, L.P. User-centered development and testing of a monitoring system that provides feedback regarding physical functioning to elderly people. Patient Preference Adher. 2013, 7, 843–854. [Google Scholar] [CrossRef] [PubMed]
  20. Költringer, T.; Grechenig, T. Comparing the immediate usability of graffiti 2 and virtual keyboard. Proceedings of CHI ’04 Extended Abstracts on Human Factors in Computing Systems, Vienna, Austria, 24–29 April 2004; ACM: New York, NY, USA, 2004; pp. 1175–1178. [Google Scholar]
  21. Tsung-han, T.; Fong-gong, W.; Yu-Hsiu, H. The Survey of Usability Evaluation in Social Network Sites’ Reply Mechanism. In Proceedings of the International Conference on Universal Access in Human-Computer Interaction, Las Vegas, NV, USA, 21–26 July 2013; Springer: Berlin, Germany, 2013; pp. 718–725. [Google Scholar]
  22. Duffy, S.A.; Fowler, K.E.; Flanagan, P.S.; Ronis, D.L.; Ewing, L.A.; Waltje, A.H. The development of the tobacco tactics website. JMIR Res. Protoc. 2013, 2, e22. [Google Scholar] [CrossRef] [PubMed]
  23. Mathew, D.; McKibbon, K.A.; Lokker, C.; Colquhoun, H. Engaging With a Wiki Related to Knowledge Translation: A Survey of WhatisKT Wiki Users. J. Med. Internet. Res. 2014, 16, e21. [Google Scholar] [CrossRef] [PubMed]
  24. Tolentino, G.P.; Battaglini, C.; Pereira, A.C.V.; de Oliveria, R.J.; de Paula, M.G.M. Usability of Serious Games for Health. In Proceedings of the 3rd International Conference on Games and Virtual Worlds for Serious Applications, Athens, Greece, 4–6 May 2011; IEEE Computer Society: Los Alamitos, CA, USA, 2011; pp. 172–175. [Google Scholar]
  25. Mattos, L.S.; Deshpande, N.; Barresi, G.; Guastini, L.; Peretti, G. A novel computerized surgeon-machine interface for robot-assisted laser phonomicrosurgery. Laryngoscope 2013, 124, 1887–1894. [Google Scholar] [CrossRef] [PubMed]
  26. Meldrum, D.; Glennon, A.; Herdman, S.; Murray, D.; McConn-Walsh, R. Virtual reality rehabilitation of balance: assessment of the usability of the Nintendo Wii(®) Fit Plus. Disabil. Rehabil. Assist. Technol. 2012, 7, 205–210. [Google Scholar] [CrossRef] [PubMed]
  27. Duvinage, M.; Castermans, T.; Petieau, M.; Seetharaman, K.; Hoellinger, T.; Cheron, G.; Dutoit, T. A subjective assessment of a P300 BCI system for lower-limb rehabilitation purposes. In Proceedings of the 34th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, San Diego, CA, USA, 28 August–1 September 2012; IEEE Service Center: Piscataway, NJ, USA, 2012; pp. 3845–3849. [Google Scholar]
  28. Kalawsky, R.S. VRUSE–A computerised diagnostic tool: For usability evaluation of virtual/synthetic environment systems. Appl. Ergon. 1999, 30, 11–25. [Google Scholar] [CrossRef]
  29. Fitzgerald, D.; Kelly, D.; Ward, T.; Markham, C.; Caulfield, B. Usability Evaluation of E-Motion: A Virtual Rehabilitation System Designed to Demonstrate, Instruct and Monitor a Therapeutic Exercise Programme. In Proceedings of the Virtual Rehabilitation Conference, Vancouver, BC, Canada, 25–27 August 2008; IEEE Service Center: Piscataway, NJ, USA, 2008; pp. 144–149. [Google Scholar]
  30. Jeon, W.; Clamann, M.; Zhu, B.; Gil, G.H.; Kaber, D. Usability evaluation of a virtual reality system for motor skill training. In Advances in Usability Evaluation Part II; Rebelo, F., Soares, M.M., Eds.; CRC Press: Boca Raton, FL, USA, 2012; pp. 86–95. [Google Scholar]
  31. Cameirão, M.S.; Badia, S.B.; Oller, E.D.; Verschure, P.F. Neurorehabilitation using the virtual reality based Rehabilitation Gaming System: methodology, design, psychometrics, usability and validation. J. Neuroeng. Rehabil. 2010, 7, 1–14. [Google Scholar] [CrossRef] [PubMed]
  32. Regenbrecht, H.; Hoermann, S.; McGregor, G.; Dixon, B.; Franz, E.; Ott, C.; Hale, L.; Schubert, T.; Hoermann, J. Visual manipulations for motor rehabilitation. Comput. Graph. 2012, 36, 819–834. [Google Scholar] [CrossRef]
  33. Shin, J.H.; Ryu, H.; Jang, S.H. A task-specific interactive game-based virtual reality rehabilitation system for patients with stroke: A usability test and two clinical experiments. J. Neuroeng. Rehabil. 2014, 11, 1–10. [Google Scholar] [CrossRef] [PubMed]
  34. Kizony, R.; Katz, N.; Weiss, P.L. Adapting an immersive virtual reality system for rehabilitation. J. Vis. Comput. Animat. 2003, 14, 261–268. [Google Scholar] [CrossRef]
  35. Witmer, B.G.; Singer, M.J. Measuring Presence in Virtual Environments: A Presence Questionnaire. Presence Teleoper. Virtual Environ. 1998, 7, 225–240. [Google Scholar] [CrossRef]
  36. Kizony, R.; Katz, N.; Rand, D.; Weiss, P.L. Short Feedback Questionnaire (SFQ) to enhance client-centered participation in virtual environments. CyberPsychol. Behav. 2006, 9, 687–688. [Google Scholar]
  37. Kizony, R.; Raz, L.; Katz, N.; Weingarden, H.; Weiss, P.L. Using a video projected VR system for patients with spinal cord injury. J. Rehabil. Res. Dev. 2005, 42, 595–608. [Google Scholar] [CrossRef] [PubMed]
  38. Iman, B.; Miller, W.C.; McLaren, L.; Chapman, P.; Finlayson, H. Feasibility of the Nintendo WiiFit™ for improving walking in individuals with a lower limb amputation. SAGE Open Med. 2013, 1, 1–11. [Google Scholar]
  39. Gil-Gómez, J.A.; Manzano-Hernández, P.; Albiol-Pérez, S.; Aula-Valero, C.; Gil-Gómez, H.; Lozano-Quilis, J.A. SEQ: Suitability evaluation questionnaire for virtual rehabilitation systems. Application in a virtual rehabilitation system for balance rehabilitation. In Proceedings of the 7th International Conference on Pervasive Computing Technologies for Healthcare, Venice, Italy, 5–8 May 2013; pp. 335–338. [Google Scholar]
  40. Muñoz Tomás, M.T.; Gil-Gómez, J.A.; Gil-Gómez, H.; Lozano-Quilis, J.A.; Albiol-Pérez, S.; Forcano García, M. Suitability of virtual rehabilitation for the elderly: A study of a virtual rehabilitation system using the SEQ. Eur. Geriatr. Med. 2013, 4, S109. [Google Scholar] [CrossRef]
  41. Folstein, M.F.; Folstein, S.; Mchugh, P.R. Mini-Mental State: A practical method for grading the cognitive state of patients for the clinicians. J. Psychiatr. Res. 1975, 12, 189–198. [Google Scholar] [CrossRef]
  42. Forcano García, M.; Albiol-Pérez, S.; Aula Valero, M.C.; Gil-Gómez, J.A.; Solsona Hernández, S.; Manzano Hernández, P. Balance virtual rehabilitation in the elderly: The use of the “ABAR” system. Eur. Geriatr. Med. 2013, 4, S109. [Google Scholar] [CrossRef]
  43. Cronbach, L.J. Coefficient alpha and the internal structure of tests. Psychometrika 1951, 16, 297–334. [Google Scholar] [CrossRef]
  44. Babikian, T.; Asarnow, R. Neurocognitive outcomes and recovery after pediatric TBI: Meta-analytic review of the literature. Neuropsychology 2009, 23, 283–296. [Google Scholar] [CrossRef] [PubMed]
  45. Streiner, D.L. Starting at the beginning: an introduction to coefficient alpha and internal consistency. J. Personal. Assess. 2003, 80, 99–103. [Google Scholar] [CrossRef] [PubMed]
  46. Santos, M.E.C.; Taketomi, T.; Sandor, C.; Polvi, J.; Yamamoto, G.; Kato, H. A usability scale for handheld augmented reality. In Proceedings of the 20th ACM Symposium on Virtual Reality Software and Technology, Edinburgh, UK, 11–13 November 2014; ACM: New York, NY, USA, 2014; pp. 167–176. [Google Scholar]
  47. Fackrell, K.; Fearnley, C.; Hoare, D.J.; Sereda, M. Hyperacusis Questionnaire as a Tool for Measuring Hypersensitivity to Sound in a Tinnitus Research Population. BioMed Res. Int. 2015, 2015, 1–12. [Google Scholar] [CrossRef] [PubMed]
Sensors 17 01589 g001 550
Figure 1. Patient interacting with the ABAR system.
Figure 1. Patient interacting with the ABAR system.
Sensors 17 01589 g001
Sensors 17 01589 g002 550
Figure 2. Scree plot showing eigenvalues of components after analysis of the principal components. Only two components had an eigenvalue greater than 1. The scree plot did not present a clear point of inflexion, although a closer inspection indicated a slight inflexion in the second component.
Figure 2. Scree plot showing eigenvalues of components after analysis of the principal components. Only two components had an eigenvalue greater than 1. The scree plot did not present a clear point of inflexion, although a closer inspection indicated a slight inflexion in the second component.
Sensors 17 01589 g002
Table
Table 1. The USEQ (User Satisfaction Evaluation Questionnaire).
Table 1. The USEQ (User Satisfaction Evaluation Questionnaire).
QuestionResponse
Not at All–Very Much
Q1. Did you enjoy your experience with the system?1 2 3 4 5
Q2. Were you successful using the system?1 2 3 4 5
Q3. Were you able to control the system?1 2 3 4 5
Q4. Is the information provided by the system clear?1 2 3 4 5
Q5. Did you feel discomfort during your experience with the system?1 2 3 4 5
Q6. Do you think that this system will be helpful for your rehabilitation?1 2 3 4 5
Table
Table 2. Characteristics of the participants.
Table 2. Characteristics of the participants.
MeanSD
Age (years)74.2514.59
 Duration of the balance disorders (years)3.395.51
n%
Sex
 Male1947.50
 Female2152.50
Chronicity
 Postacute (0–5 months post-injury)1025.00
 Acute (6–23 months post-injury)1127.50
 Chronic (24 months or more post-injury)1947.50
Background domicile
 Urban3075.00
 Rural1025.00
Level of studies
 No studies37.50
 Primary studies2460.00
 Secondary studies1025.00
 Higher studies37.50
Table
Table 3. Score on questions of the USEQ. Corrected Item-Total Correlation. Cronbach’s Alpha if item is deleted.
Table 3. Score on questions of the USEQ. Corrected Item-Total Correlation. Cronbach’s Alpha if item is deleted.
QuestionMeanSDScale Mean if Item DeletedCorrected Item-Total CorrelationCronbach’s Alpha if Item Deleted
Q14.430.87421.380.3260.712
Q24.231.09721.580.6660.597
Q33.801.22422.000.5720.637
Q44.650.66221.150.4040.694
6-Q54.700.60821.100.4960.678
Q64.000.96121.800.3210.716
Table
Table 4. Component matrix. This table contains the correlations between the items and the factors with an eigenvalue greater than 1. The percentage of variance explained by each factor is also shown.
Table 4. Component matrix. This table contains the correlations between the items and the factors with an eigenvalue greater than 1. The percentage of variance explained by each factor is also shown.
Component
12
Questions
Q20.816
Q30.780
Q50.6690.647
Q40.582−0.414
Q60.5410.672
Q10.468−0.490
Variance explained (%)42.869%22.908%

Share and Cite

MDPI and ACS Style

Gil-Gómez, J.-A.; Manzano-Hernández, P.; Albiol-Pérez, S.; Aula-Valero, C.; Gil-Gómez, H.; Lozano-Quilis, J.-A. USEQ: A Short Questionnaire for Satisfaction Evaluation of Virtual Rehabilitation Systems. Sensors 2017, 17, 1589. https://doi.org/10.3390/s17071589

AMA Style

Gil-Gómez J-A, Manzano-Hernández P, Albiol-Pérez S, Aula-Valero C, Gil-Gómez H, Lozano-Quilis J-A. USEQ: A Short Questionnaire for Satisfaction Evaluation of Virtual Rehabilitation Systems. Sensors. 2017; 17(7):1589. https://doi.org/10.3390/s17071589

Chicago/Turabian Style

Gil-Gómez, José-Antonio, Pilar Manzano-Hernández, Sergio Albiol-Pérez, Carmen Aula-Valero, Hermenegildo Gil-Gómez, and José-Antonio Lozano-Quilis. 2017. "USEQ: A Short Questionnaire for Satisfaction Evaluation of Virtual Rehabilitation Systems" Sensors 17, no. 7: 1589. https://doi.org/10.3390/s17071589

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop