Next Article in Journal
Key Factors for Project Crowdfunding Success: An Empirical Study
Next Article in Special Issue
Social Trust and Open Innovation in an Informal Economy: The Emergence of Shenzhen Mobile Phone Industry
Previous Article in Journal
Fostering Resident Pro-Environmental Behavior: The Roles of Destination Image and Confucian Culture
Previous Article in Special Issue
How Does Technology Startups Increase Innovative Performance? The Study of Technology Startups on Innovation Focusing on Employment Change in Korea

Sustainability 2020, 12(2), 598;

Factors Influencing Usability of Rehabilitation Robotic Devices for Lower Limbs
Industrial Design Department, Kyungil University, 50 Gamasil-gil, Hayang-eup, Gyeongsan-si, Gyeongsangbuk-do 38428, Korea
Received: 29 December 2019 / Accepted: 10 January 2020 / Published: 14 January 2020


In recent years, there has been a sharp increase in the number of elderly people in South Korea; this has led to rising costs and concerns on the quality of physical therapy treatment involving rehabilitation robotic devices. Therefore, the government has asked academia to expand its research scope to evaluate the usability of these devices. Hence, this study aimed to identify the major factors influencing the usability of a rehabilitation robotic device for lower limbs and the reasons for involving several diverse user groups for a more comprehensive evaluation. To measure usability as perceived by three primary user groups of user experience (UX) professionals, rehab professionals, and lay people, this study collected 196 survey. The results of an EFA showed that among three general quality factors and five device specific factors, visual pertinence, use confidence, and safety were the critical factors influencing usability, and the results of ANOVA offered that there was discrepancy in the influential factors, namely visual pertinence, transferring, and holding the body. These findings indicate that the necessity of employing a posture-centered approach and multiple user groups in assessing the usability of rehabilitation devices. Given these findings, it is suggested that the industry and design community should move toward implementing a more explorative perspective to enable a more human-centered and posture-concerned approach to provide better usability to the diverse users of medical devices.
rehabilitation robotic device; diverse user groups; medical device; lower-limb rehabilitation; usability assessment; human-centered approach

1. Introduction

1.1. Increasing Demands for Rehabilitation Robotic Devices

After the invention of MIT-Manus, the first robotic device for rehabilitation in 1992 [1], over the last decade, the application of robotic technologies to rehabilitation has progressed from conception to reality. However, in South Korea, robotics-based rehabilitation has emerged only very recently. Across countries, there are some common factors that make robot-assisted therapy acceptable in medical settings, such as demographic factors, financial situations, and healthcare service issues.
The first basic factor is the increase in the number of the elderly requiring rehabilitation treatment and the resulting financial burden. The global population is aging, and especially South Korea’s population is aging at the fastest pace [2]. Koreans aged 65 and older accounted for 14.3 percent of the total domestic population of approximately 50 million [3]. This sharp increase in the elderly population has increased the demand for rehabilitative medical treatment, which is conventionally human resource intensive [4] and thus costly. Moreover, there are service quality issues. There are very few physiotherapists available [5]. Among the advanced countries, the percentage of physiotherapists for every 100 people in South Korea is 0.54, lower than 0.62 of the US and 0.89 of Europe [6]; this is also a critical impediment for providing equitable healthcare services by service providers. In addition, conventional manual therapy is significantly affected by the experience of therapists, which makes it difficult to fulfil the requirements of high-intensity and repetitive training [7]; therefore, effective treatment cannot be guaranteed [8]. Utilizing robots in rehabilitation medicine is expected not only to release physicians from the heavy burden of training sessions but also to help patients precisely, quantitatively, and scientifically perform training exercises [9]. Hence, demographic changes, costs incurred by patients, and service quality are the main parameters accelerating the development of rehabilitation robotic devices comprising some automatic systems, which allow several types of users from multiple dynamic settings, such as hospitals, wherein emergency situations occur, to utilize these devices to aid in their everyday activities.
Considering this high demand for robotic systems in rehabilitation treatment, the global rehabilitation robot market is expected to grow from $40 million USD in 2014, at a compound annual growth Rate (CAGR) of 86.1 percent (24.27%), to $1.8 billion USD by 2020 [10]. In comparison, the domestic market in South Korea was worth $7.6 million in 2014 but is projected to grow to $65.2 million USD, increasing at a CAGR of 43.1 percent, by 2020 [11]. Despite these promising growth prospects, there is some skepticism regarding the use of rehabilitation robots. Though MIT-Manus for upper limbs, Lokomat for lower limbs, and others claim to be successful in the global market, there are limited commercially available devices due to various reasons such as complexity of usage and difficulties in reproducing prototypes, lack of clinical trials showing evidence of their effectiveness and acceptance of the system in clinical practice, or price- and time-related issues [12]. Similarly, although the South Korean government designated the robotics industry as a strategic sector in 2003 to foster growth and has strengthened investment and policy supports, the rehabilitation robotics market is still at its initial stage [13].
Owing to difficulties in compiling statistical data at such an early stage, to analyze the rehabilitation robotics industry and market, it is worth referring to associated fields like the medical device industry and robotics industry of South Korea. As of 2014, the technical standard of rehabilitation medical devices in South Korea is 77.0 percent of and is 3.1 years behind the most advanced countries like the US and European countries [14]. The medical device market of South Korea mainly comprises small- and medium-sized enterprises (SMEs), which account for 75.8 percent and 19.5 percent of the market, respectively [15]. On the other hand, despite the high utilization of robots in industry and with the industrial robot market of South Korea ranked among the top five in the world [16], the robot industry of South Korea is mostly composed of SMEs (92%) as well [17]. These facts indicate that the rehabilitation robot industry of South Korea needs consistent support financially and through favorable policies to develop the competitive edge required to establish itself in the global market.
Because of the practical needs of society and the healthcare industry, rehabilitation robotics has become an attractive research field that has drawn considerable attention in the last decade [7]; however, research progress in this field remains slow. In the past decade, most studies have focused on demonstrating the therapeutic effectiveness, efficiency, or advantages of robot-assisted therapy rather than conventional labor-intensive training procedures; therefore, research in this field should focus on overcoming these challenges to make robotics more user-friendly [1] and create better interaction between robots and their diverse users [18]. These unmet needs emphasize the importance of using a human-centered approach for developing rehabilitation robots. Nevertheless, the continuous advancements in rehabilitation robotics like other everyday devices have caused people to think that these devices are technology-driven products [19] while ignoring the fact that these devices are to be used by a person.
To provide various users with a better experience when utilizing robotic devices, it is important to understand the factors influencing the perceptions of users regarding these devices; it is necessary to make them realize that these devices are usable from the pre-use to the post-use stage. Hence, to boost the development of rehabilitation robotic devices with enhanced usability, this paper mainly focuses on assessing the factors influencing usability on the basis of a usability questionnaire on robotic devices and then comparing the ratings provided by possible users for the factors influencing the usability of these devices.

1.2. Usability and Users of Medical Devices

In product development, user-centered approaches engaging users, and assessing and fulfilling their needs are used and these approaches have proved their worth [20]. Likewise, in developing medical devices, user involvement has become a touchstone for determining the success or failure of a device [21]. It is because no one but the users themselves can better judge and identify the barriers in using these devices; further, sharing their experience presents a valid rationale to identify possible risks [22]. A more crucial point is that misunderstanding or not considering the needs of users in developing a consumer product causes under-utilization or rejection of the product [23]; however, in medical devices, it raises the possibilities of errors leading to preventable patient injuries and more severe incidents in some cases [21,24,25,26,27]. In other words, the occurrence of the errors indicates failure in understanding the usage contexts of the device and presuming situations where errors are likely to occur rather than actually analyzing the error possibilities; thus, the device is not in line with the mental models of users or the thought processes of users about how the device works in the real world.
Who is the user of medical devices including rehabilitation robots and who should be concerned during a development process? In the real world, some devices are operated only by clinicians, and some are operated by trained caregivers. As the diversity of medical devices widens, their users, having traditionally been regarded as just healthcare professionals and patients [26], can also further diversify. In practice, users of medical devices range from well-trained people, such as medical doctors or nurses, to untrained people, such as laypersons or patients [28,29,30]. These facts indicate that the users are heterogeneous with different points of view.
Through analyzing research on medical devices with user involvement, a medical device user can be defined as anyone who uses a medical device for treatment and/or for taking care of oneself or someone else; users necessarily fall into either the primary or secondary class based on the usage purpose; they could be using the devices for therapeutic purposes or for testing, calibration, and research [31]. In addition, under these user classes, there are the various divisions of user groups and types depending on why the device is used, where the device is used, what stage the device is in its lifecycle, and what type of device is used.
This flexible user definition with several subclasses seems inevitable owing to two explicit trends in the healthcare industry. One is the increased use of devices by laypersons [32] who, as a group, have higher diversity and thus should be included in any user testing [33]. The other is the involvement of diverse users [34], because user engagement is extremely important at each stage of the development process. These trends explain the necessity of having flexible definitions for medical device users and such definitions and classifications help integrate and understand the perspectives of diverse users by analyzing the different opinions they have on using medical devices.

1.3. Usability Assessment of Medical Devices by SMEs

The International Electro-Technical Commission (IEC) requires that medical device manufacturers implement a usability engineering process to analyze, specify, design, verify, and validate the usability of their equipment [35]. In addition, the US Food and Drug Administration (FDA) requires proof of human factors/usability procedures in the development or (re)design of the medical devices for sale in the US [32]. In the medical device industry, usability is no longer a luxury. Though the requirements of the IEC and FDA do not provide specific questionnaires immediately applicable to individual usability tests, these requirements specified help in raising awareness on the usability of medical devices in the industry.
However, when considering the medical device industry of South Korea, predominantly comprising SMEs, the requirements of agencies/regulatory bodies for examining and demonstrating the user-centered design of devices places an enormous burden on the industry. This has caused the SMEs of South Korea to adopt discount research techniques. From a practical perspective, usability tests of medical devices frequently involve moderately large sets of users ranging from 50 to 150 participants [36]; however, the medical device enterprises of South Korea conduct usability tests of medical devices with approximately 20 people referring to the best practices of the industry [37]. Such a small number of participants appears to be the reason for situations where there are doubts concerning the validity of the test results and user diversity. Usability practitioners familiar with handy research techniques [38] may find that these techniques are not sufficient to address human safety requirements [36], which is one of the main issues associated with medical devices. Medical device development during the last decade has placed increased importance on user issues related to device design, human errors, and patient safety [19,24,39,40,41]. These factors necessitate the importance of utilizing referential tools universally and immediately for usability tests of various medical devices, which in turn can help reduce the burden of SMEs.
In many cases, usability research is characterized as a highly qualitative approach, but questionnaires are often utilized to identify remaining problems and evaluate success or failure in achieving the target usability values [42]. Though there are well-known quantitative approaches—such as standardized questionnaires to evaluate usability including the Questionnaire for User Interface Satisfaction (QUIS), Software Usability Measurement Inventory (SUMI), Computer System Usability Questionnaire (CSUQ), and System Usability Scale (SUS) [32]—these evaluation methods appear insufficient for medical devices because the methods mainly focus on assessing two-dimensional screens in computer systems. Table 1 summarizes these assessment tools.
Despite the absence of standardized questionnaires to evaluate the usability of medical devices, relatively less effort has apparently been made to establish a tool for evaluating rehabilitation devices. This is evident from the small number of studies regarding usability questionnaires in the healthcare area. Let us consider journal articles recently published between 2014 and 2019 (as of 30 August) in English and Korean from PubMed in the US and Research Information Sharing Service (RISS) of South Korea, respectively. When searched following a scoping review approach using the keywords “usability,” “questionnaire,” and “design,” combined with “medical device,” a total of 38 and eight studies were found, respectively. Out of the 38 studies listed on PubMed in English, 14 focused on screen-based equipment like mobile phones or information systems, 12 on small personal equipment like pen injectors and inhalation devices, 11 on intangible services, and one on orthodontic appliances for the upper limbs. Further, among the eight studies from the RISS, four studies were on screen-based devices, two focused on small personal equipment, one on various device comparisons, and another on service evaluation. Thus, considering this scarcity of studies about usability questionnaire development regarding medical devices for lower limbs, it is worth trying to establish a questionnaire for evaluating the usability of these devices.

2. Materials and Methods

2.1. Materials and Subjects

To create a base for standardized questionnaires to evaluate the usability of medical devices for lower limbs, focusing on visual design, this study tested a working prototype of a rehabilitation robotic device for pre-gait training. Considering the increasing interest in robotic medical devices resulting from the increasing number of older people and soaring costs of rehabilitative therapy, it is appropriate to develop such questionnaires.
The device with a semi-automatic system helps service-provider users like medical staff to reduce their workload and makes patient users perform weight shifting, sitting, and gait training. More specifically, the use procedure for the device considered in this study is as follows: Once a patient user sits on the device, it automatically adjusts the chest board and seat of the device according to the height of the patient user. When performing the main exercise, it sets a task of weight balancing, automatically moves the central pole back and forth, and moves the foot pad up and down depending on the exercise program mode. Additionally, it forces the patients to shift their weights between the two legs and the left and right sides of the pelvis when sitting on the seat and bend their knees in turn while leaning over the chest board (Figure 1).
This robotic device is intended to be used under the supervision of a clinician (physiatrist, orthopedist, and physiotherapist); however, its semi-automatic systems allow a guardian to operate it. Hence, the primary users can be categorized into two groups: clinicians and laypersons. Laypersons refer to people (potentially without medical training) such as the patients, who use the device, and their guardians, who may operate the device for them. However, in this study, it was considered better to seek others representing real patient users having impaired motor functions regarding ethics to prevent participants from experiencing unexpected adverse incidents while testing the device. Consequently, to obtain the opinions of patient users who are the ultimate recipients, it is cogent to replace them with user experience (UX) designers who are educated and trained in various industry practices to convey users’ views. Thus, it was decided that the participants acting as primary users were rehabilitation medical treatment professionals (hereafter referred to as rehab professionals), lay people representing patients’ guardians, and UX designer professionals (hereafter referred to as UX professionals) conveying the patients’ views.

2.2. Questionnaire Development

To identify influential factors with statements before developing a questionnaire which measures perceived usability, the following process was adopted. First, preliminary questions were collected through a literature review on electronic medical devices [43] and the SUS [44]. Second, with the help of four UX professionals skilled in evaluating user experiences using a low-fidelity working model of the device, the questions were reviewed and revised. This allowed for verifying the validity of the content in the preliminary questions. Third, a pilot test using the questionnaire was conducted with five people to find any potential errors in wording, terminology, and conciseness, and to ensure the appropriateness of the questionnaire. Based on the findings from the above three steps, the final version of the questionnaire was developed.
The questionnaire comprised 56 questions divided into three parts, focusing on demographic information, overall quality parameters applicable to different devices, and device specifics based on device components. The demographic information part included four questions regarding prior experience in using medical devices, age, gender, and professional experience. The prior experience question simply asked whether each participant had used any rehabilitation robotic devices similar to the robotic device in this study. The part regarding the general quality parameters applicable for evaluating the usability of most devices contained statements of effectiveness, efficiency, and satisfaction based on the definition of usability. Though there are numerous measures for evaluating products with high usability such as learnability, memorability, satisfaction, accessibility, and others [45], the International Organization for Standardization (ISO 9241-11) [46] states that measures of usability need to consider the user’s ability to complete tasks using the system (effectiveness), resource expenditure required for task completion (efficiency), and satisfaction. In addition, attractiveness was considered as a measure to evaluate aesthetic usability [47]. The device-specific questions included items on the attributes of each part possibly influencing on its usability (e.g., comfort, durability, and visual appropriateness) based on its operational procedure. Table 2 presents the structure of the developed questionnaire. Each question was rated on a five-point Likert scale from 1 (strongly disagree) to 5 (strongly agree).

2.3. Survey Administration

The survey to collect evaluation data of perceived usability for the robotic device was administered digitally through a computer to show video clips and pictures related to the device’s specific components or usage procedures. Prior to answering the survey questions, each participant was offered an outline of the tasks to better understand the survey. Then, they were asked to watch a how-to-use video about the device. Despite concerns about confidentiality and privacy, video-based surveys are promising methods in human-factor engineering research [48]. While a subject’s own experience can offer subject-rich and direct information to evaluate products and video recordings may hide some challenges faced by a user [49], video data have proved to be a powerful tool for making informed decisions regarding design changes [50] and also allow raters to detect synchronized actions and events in the video footage. After watching the video, the users could again review the video to answer the survey questions. Furthermore, they could change their answers whenever they wanted. In the final session of the survey, when component-specific questions were given, photos and videos showing each of the parts were presented to the participants. To assist participants in managing unexpected problems during the survey, the research team or its representative was present. All the participants were informed about the incentive for participating in the survey, a gift voucher worth KRW 10,000, which is equivalent to $10 USD.

3. Results

3.1. Characteristics of Respondents

The survey to determine the visually perceived ease of use by the user groups in using the rehabilitation robotic device was conducted with 196 participants. The participants consisted of 51 UX design professionals, 51 rehab professionals, and 94 ordinary people. They were contacted through convenience sampling in South Korea. To reduce the cost, initial participants of the group, gathered from the authors’ personal network, were asked to spread the survey through their personal networks. The UX design professional group consisted of UX designers who were working on researching and analyzing home appliances, automobiles, and mobile phones. The length of their careers varied from 1–13 years. The rehab professional group comprised physical therapists and physiatrists with experiences ranging from 1–20 years. In terms of age, the UX design professionals ranged from 20–40 years, the rehab professional group was aged from 20–50 years, and the lay people group comprised people aged 20–60 years (Table 3). Introduced after the early 2000s, UX design and rehabilitation medicine are relatively new fields in South Korea; therefore, the participants in the professional groups were young compared to those in the lay people group, which included people aged over 60 who lived independently. Of the participants, 42.35% were females, and 57.65% were males. The first contact with the participants was initiated via text messages, emails, or meetings in person; subsequently, the goal and process of the survey was explained to the participants. All participants indicated that they had not used rehabilitation robots similar to the robotic device tested in this study.

3.2. Identifying Factors Influencing Usability

To uncover the relational structure of large variables and their effective factors, Table 4 presents the Exploratory Factor Analysis (EFA) results.

3.3. Differences in Usability Evaluation between User Groups

To identify differences in the extent of influence in terms of the withdrawn factors between the three primary user groups, the UX professionals, rehab professionals, and lay people, one-way analysis of variance (ANOVA) tests with Scheffé post hoc comparisons at 0.05 were conducted. With a large sample (>30) and the results of Levene’s Test showing equality of variances, the ANOVA test results revealed that there was discrepancy in the influential factors, namely visual pertinence, transferring, and holding the body (Table 5). More specifically, the mean scores of the lay people statistically did not match with those of the UX design professionals in terms of visual pertinence (UX professionals M = 3.93, rehab professionals M = 3.66, and lay people M = 3.48, when F = 5.62, p = 0.004), transferring (UX professionals M = 3.38, rehab professionals M = 3.28, and lay people M = 3.03, when F = 3.11, p = 0.047) and holding the body (UX professionals M = 3.73, rehab professionals M = 3.62, and lay people M = 3.24, when F = 6.72, p = 0.002). In all the dimensions showing dissensus between the user groups, the lay people showed higher scores than the UX professionals.

4. Discussion

4.1. Factors Influencing Usability of Medical Devices for Lower Limbs

From the EFA, in order to identify the main factors influencing the usability of the robotic device involving primary users, three general quality factors (visual pertinence, use confidence, and safety), and five device-specific factors (piggybacking, transitioning, supporting body weight, holding the body, sitting, and getting on/off) were determined. Considering the different names given to the factors by different researchers, the general quality factors seem to get assent from researchers in that they are frequently identified when establishing evaluation models or standards (of medical devices); visual pertinence probably can be interpreted as attractiveness or aesthetics, use confidence involves combining learnability and intuitiveness, and safety refers to trustworthiness or security. The repetitive emergence of these factors in usability studies demonstrates their importance as effective factors; they present strong possibilities of contributing toward measuring and then improving perceived usability during the development process at least in these types of devices. Another interesting observation was that the device-specific factors were so specific to the device tested in this study that it was difficult to see them in precedent. Posture, the identifier for discerning and regrouping the factors, is one of the frequently observed product-specific problems in medical devices. Many researchers have noted that right body positions matter to users. Unique positions necessarily required for using a particular medical device should be considered [24] in designing the device. If not, poor design could be a constraint that can cause users to use the device only once [51]. Proper body position is vital for providing comfort to people with reduced mobility like bedridden patients or patients with limited mobility [52]. All these factors support the necessity of a repository for posture-related factors, criteria, or metrics and a questionnaire factoring postures to measure the usability of medical devices. This new observation can help discover more usability flaws by providing another perspective for evaluating medical devices. The unfamiliarity of the new perspective is possibly a burden to the industry; however, it is not merely a matter of considering another factor when designing medical devices. Rather than considering this as one more relevant design criterion, we could think about how the product could be improved further. Moreover, in the long term, it could be a more effective way of ensuring a greater return on investment (ROI) to users with less effort and few critical usability flaws.

4.2. Discrepancy in Usability Evaluation among User Groups

To evaluate the usability of medical devices by considering user diversity, this study involved three primary user groups. ANOVA statistically compared the user ratings for the eight discovered factors and revealed disagreement between the lay people and UX professionals. These different results indicate that working only with a specific user group possibly results in lost opportunities to identify different perspectives and needs, which could be critical for consistent use and enhancing the ease-of-use of the device. Further, this difference requires the SMEs accept that their de facto standard is an economical way of measuring usability into something question worthy as for its authentic relevance. This low-cost technique, called the discount usability test, involves three to five users [53] and has shown its robustness with astounding ROI figures in usability analysis [54,55]. However, despite being a simpler, easier, faster, and thus discounted version of the original usability evaluation method, this method is controversial because it could cause oversimplification or distortion of results to some degree. The most controversial issue faced by the advocates of the discount usability test is the number of users to be employed to evaluate the usability of any given device [56]. A popular claim is that diminishing returns kicks in after testing with approximately five users [57]. However, many researchers attempting to define acceptable upper bounds argue that more user tests are necessary [58,59,60]. Moreover, there are some debates with respect to the number of users and user heterogeneity in the real world. Though the possibilities for including different types of users increase as more users are tested, this does not guarantee user diversity. Considering the environment in which the industry operates, the number of user types depends on economic feasibility. Thus, an effective and efficient range for the lower and upper bounds in terms of user variety for usability tests is required. Nevertheless, before identifying methods requiring the smallest amount of effort (the number of user types) that will return the greatest amount of data (the number of usability flaws), ‘do it’ for user benefits is the invariable rule of usability tests. A successful product is not only the one that is commercially beneficial to the manufacturer [61] but also the one that is beneficial to its users [30]. Even while considering the increased use of medical devices used in care homes, private homes, and elsewhere [62,63], laypeople and caregivers as well as medical device professionals (e.g., medical staff and UX professionals) should be included in usability tests to evaluate the medical device [33]. Therefore, because of diverse user groups, it is necessary to follow a human-centered approach in designing medical devices, rather than a user-focused approach which constrains the human-centric nature of products [64]. However, if companies operating in the challenging environment of the industry cannot wait until designers and researchers consider all segments of the population, finding an equilibrium point between the real and ideal design can be a feasible approach. Inclusiveness of products is “negotiable” to find an equilibrium point between the range of requirements of varied users and the population of target users, which is linked to the lucrativeness of the market [65]. Therefore, considering that SMEs manufacturing medical devices are unlikely to carry out rigorous user research, as a strategic “negotiable” progress, taking the views of primary users into consideration can contribute towards developing a medical device with better usability.

4.3. Contribution of Expected Usability to Improved Visual Design

This study attempted to investigate the factors influencing the usability of medical devices and identified differences in terms of the evaluation of these factors by different user groups. However, some might argue that the findings are not sufficiently convincing because the results are related to expected usability, i.e., evaluations before actual use. Scholars diverge on this issue of validity of the results with respect to expected usability. On the one hand, some studies found that expected usability has an influence on the perceived overall quality [66], overall impression [67], and very early ratings of perceived usability; these results were consistent across studies [68]. On the other hand, several studies have shown that expected usability is connected only to visual aesthetics of a system [69,70] or a website [71,72,73]. Despite this controversy, we may conclude that the factors influencing the evaluations regarding expected usability and different evaluations by different user groups are effective for improving the visual design of the device at least. Thus, by constructing a questionnaire reflecting the research outcomes and utilizing it to measure the usability of a medical device, we expect to obtain some valuable inputs for enhancing the device design. If so, it is worth considering that an aesthetically appealing design helps users to perceive that a product is usable [74]. This argument implying “what is beautiful is usable” [70], despite having a counterargument implying “what is usable is beautiful” [75], is supported by a widely acknowledged notion that people positively infer from attractive objects to preferably keep consistent judgments about these objects [76]. This is connected to the so-called halo effect of aesthetics that outshines all other features of the object and influences the evaluation of the object by users [70]; this effect is based on the social psychological principle that a physically attractive person is considered to possess more positive personality traits than an unattractive person [77]. Likewise, in usability tests, the attitude toward a product designed aesthetically during the very early stage of product interaction [73] influences the later appreciation of the other characteristics of the product. This indicates that, aside from the debate on whether expected usability is linked with objective measures of performance in usability tests, the very first impression formed regarding the product before using it is extremely important. Therefore, it is essential to create a positive impression in terms of aesthetics from the viewpoint of usability.

5. Conclusions

The usability of robotic devices has drawn considerable attention since their first introduction in rehabilitation medicine. However, most companies in the medical device market of South Korea, where SMEs make up the majority of the market, are focusing on using simplified methods to test device usability. Despite the efficiency of the discount test methods, this is not only because of the lack of financial and human resources and time for SMEs to test their devices but also because of the deficiencies in the tools used to measure the usability of medical devices. Thus, as an effort to establish a base for developing inclusive usability evaluation tools to reflect diverse users’ voices, this study empirically analyzed the factors influencing usability of a pre-gait rehabilitation robotic device with three primary user groups and compared the evaluations of the factors by the groups. We identified visual pertinence, use confidence, and safety as the critical factors influencing usability and the necessity of adopting a posture-centric approach in developing a usability evaluation questionnaire pertaining to rehabilitation devices for lower limbs. Moreover, we found that a wider range of user groups was necessary owing to differences in the evaluations performed by the user groups considered in this study. In addition, considering the challenges faced by SMEs producing medical devices, whether they will follow this inclusive approach to integrate grass-root ideas from various users in a reciprocal, responsible, and respectful manner [78] is still an open question. However, the persistently unmet needs, referred to as wicked problems, and unaddressed problems of usability indicate that the existing approach is inadequate for the purpose. Innovations are imperative [79]. Thus, to resolve limitations of testing a rehabilitation robot, the development of a questionnaire based on the factors and posture centric perspectives discovered in this study, and the identification of the validity and reliability of the study as an assessment tool need to be considered through different rehabilitation equipment in a future study.


This research was funded by the Ministry of Education of the Republic of Korea and the National Research Foundation of Korea (NRF-2017S1A5A8021105) and the Korea Evaluation Institute of Industrial Technology (KEIT-10077757).


I would like to thank my colleagues Donghun Kim, Sunyoung Hwang, and Gyohwan Ku at the company of Mind On for fruitful collaborations and discussions on the work presented here.

Conflicts of Interest

The author declares no conflict of interest. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results.


  1. Laut, J.; Porfiri, M.; Raghavan, P. The present and future of robotic technology in rehabilitation. Curr. Phys. Med. Rehabil. Rep. 2016, 4, 312–319. [Google Scholar] [CrossRef] [PubMed]
  2. Kim, T. Identifying Stakeholders and Interactions in the Dementia Café in Seongju through Empathic Service Design Approaches. J. Open Innov. Technol. Mark. Complex. 2018, 4, 28. [Google Scholar] [CrossRef]
  3. Statistics Korea. 2018 The Statistics of the Elderly. Available online: (accessed on 20 August 2019).
  4. Hayat, H.; Lock, R.; Murray, I. Measuring software usability. In Proceedings of the BCS Software Quality Management Conference, Loughborough University, Loughborough, UK, 30 March 2015. [Google Scholar]
  5. Meng, W.; Liu, Q.; Zhou, Z.; Ai, Q.; Sheng, B.; Xie, S.S. Recent development of mechanisms and control strategies for robot-assisted lower limb rehabilitation. Mechatronics 2015, 31, 132–145. [Google Scholar] [CrossRef]
  6. Kim, K. Policy Research for Cultivating Physiotherapists, Korea Health Personnel Licensing Examination Institute. 2017. Available online: (accessed on 20 August 2019).
  7. Zhou, Z.; Meng, W.; Ai, Q.; Liu, Q.; Wu, X. Practical –velocity tracking control of a parallel robot based on fuzzy adaptive algorithm. Adv. Mech. Eng. 2013, 5, 574896. [Google Scholar] [CrossRef]
  8. Nef, T.; Mihelj, M.; Kiefer, G.; Perndl, C.; Muller, R.; Riener, R. ARM in-Exoskeleton for arm therapy in stroke patients. In Proceedings of the 2007 IEEE 10th International Conference on Rehabilitation Robotics, Noordwijk, The Netherlands, 13–15 June 2007; pp. 68–74. [Google Scholar]
  9. Lu, R.; Li, Z.; Su, C.Y.; Xue, A. Development and learning control of a human limb with a rehabilitation exoskeleton. IEEE Trans. Ind. Electron. 2013, 61, 3776–3785. [Google Scholar] [CrossRef]
  10. Song, W.K. Trends in rehabilitation robots and their translational research in National Rehabilitation Center of Korea. Biomed. Eng. Lett. 2016, 6, 1–9. [Google Scholar] [CrossRef]
  11. Kang, C. The Market of Rehabilitation Robotics Rapidly Growing but Asking a Critical Technology. Available online: (accessed on 24 August 2019).
  12. Dollar, A.M.; Herr, H. Lower extremity exoskeletons and active orthoses: Challenges and state-of-the-art. IEEE Trans. Robot. 2008, 24, 144–158. [Google Scholar] [CrossRef]
  13. Song, W.K. Technology trend of rehabilitation robots. J. Korean Inst. Commun. Inf. Sci. 2016, 33, 10–18. [Google Scholar]
  14. Lee, B.; Kim, K. Technology trend and industrial prospect of surgical robots. KEIT PD Issue Rep. 2017, 17-3, 62–76. [Google Scholar]
  15. Seo, G. Situation of exports of medical devices of South Korea; focusing on recent five years. KHIDI Brief. 2014, 151, 1–8. [Google Scholar]
  16. Ministry of Trade, Industry and Energy. Development Plan for the Robot Industry of South Korea. 2019. Available online: (accessed on 30 August 2019).
  17. Ministry of Trade, Industry and Energy. Research of Competitiveness of Robot Industry. 2017. Available online: (accessed on 30 August 2019).
  18. Qian, Z.; Bi, Z. Recent development of rehabilitation robots. Adv. Mech. Eng. 2015, 7, 1–11. [Google Scholar] [CrossRef]
  19. Lemke, M.R.; Winters, J. Removing barriers to medical devices for users with impairments. Ergon. Des. Q. Hum. Factors Appl. 2008, 16, 18–25. [Google Scholar] [CrossRef]
  20. Brockhoff, K. Customers’ perspectives of involvement in new product development. Int. J. Technol. Manag. 2003, 26, 464–481. [Google Scholar] [CrossRef]
  21. Shah, S.G.S.; Robinson, I. User involvement in medical device technology development and assessment: Structured literature review. Int. J. Health Care Qual. Assur. Inc. Leadersh. Health Serv. 2006, 19, 500–515. [Google Scholar] [CrossRef]
  22. Gosbee, J. Human factors engineering and patient safety. Qual. Saf. Health Care 2002, 11, 352–354. [Google Scholar] [CrossRef]
  23. Gallivan, M.J.; Keil, M. The user–developer communication process: A critical case study. Inf. Syst. J. 2003, 13, 37–68. [Google Scholar] [CrossRef]
  24. Wegge, K.P.; Zimmermann, D. Accessibility, usability, safety, ergonomics: Concepts, models, and differences. In Universal Access in Human-Computer Interaction. Interaction Techniques and Environments, Proceedings of the International Conference on Universal Access in Human-Computer Interaction, Toronto, ON, Canada, 17–22 July 2016; Antona, M., Stephanidis, C., Eds.; Springer: Berlin/Heidelberger, Germany, 2016; pp. 294–301. [Google Scholar]
  25. Martin, J.L.; Norris, B.J.; Murphy, E.; Crowe, J.A. Medical device development: The challenge for ergonomics. Appl. Ergon. 2008, 39, 271–283. [Google Scholar] [CrossRef]
  26. World Health Organization (WHO). Medical Device Regulations: Global Overview and Guiding Principles; WHO: Geneva, Switzerland, 2003; pp. 1–8. [Google Scholar]
  27. Samore, M.H.; Evans, R.S.; Lassen, A.; Gould, P.; Lloyd, J.; Gardner, R.M.; Abouzelof, R.; Taylor, C.; Woodbury, D.A.; Willy, M.; et al. Surveillance of medical device–related hazards and adverse events in hospitalized patients. J. Am. Med Assoc. 2004, 291, 325–334. [Google Scholar] [CrossRef]
  28. Kaye, R.; Crowley, J. Medical Device Use-Safety: Incorporating Human Factors Engineering into Risk Management; Food and Drug Administration, Guidance Document; Center for Devices and Radiological Health, Department of Health and Human Services, U.S.: Silver Spring, MD, USA, 2000; pp. 1–33. Available online: (accessed on 10 March 2006).
  29. Department of Health. EC Medical Devices Directives: Guidance Notes for Manufacturers of Custom-Made Devices; Department of Health, Guidance Document No. 9; Medicines and Healthcare Products Regulatory Agency (MHRA): London, UK, 2003; pp. 1–12.
  30. Therapeutic Goods Administration. Medical Devices, TGA News-Issue; Therapeutic Goods Administration: Symonston, ACT, Australia, 2003; Volume 42 (November 2003), pp. 3–4.
  31. Shah, S.G.S.; Robinson, I. Medical device technologies: Who is the user? Int. J. Healthc. Technol. Manag. 2008, 9, 181–197. [Google Scholar] [CrossRef]
  32. Weinger, M.B.; Gardner-Bonneau, D.J.; Wiklund, M.E. General Principles. In Handbook of Human Factors in Medical Device Design; Weinger, M.B., Wiklund, M.E., Gardner-Bonneau, D.J., Eds.; CRC Press: Boca Raton, FL, USA, 2010; pp. 1–22. [Google Scholar]
  33. National Research Council. The Role of Human Factors in Home Health Care: Workshop Summary; National Academies Press: Washington, DC, USA, 2010; pp. 9–28. [Google Scholar]
  34. Baker, G.R.; Norton, P.G.; Flintoft, V.; Blais, R.; Brown, A.; Cox, J.; Etchells, E.; Ghali, W.A.; Hebert, P.; Majumdar, S.R.; et al. The Canadian adverse events study: The incidence of adverse events among hospital patients in Canada. Can. Med Assoc. J. 2004, 170, 1678–1686. [Google Scholar] [CrossRef]
  35. Martin, J.L.; Craven, M.P.; Norris, B.J. MATCH: A new industry-focused approach to medical device development. In Proceedings of the International Conference of Healthcare systems Ergonomics and Patient Safety, Florence, Italy, 30 March–2 April 2005; Tartaglia, R., Bagnara, S., Bellandi, T., Albolino, S., Eds.; Taylor & Francis: London, UK, 2005; pp. 294–297. [Google Scholar]
  36. Hass, C.; Berlin, D. Usability testing medical devices: A practical guide to minimizing risk and maximizing success. In Proceedings of the International Conference of Design, User Experience, and Usability; Springer: Berlin/Heidelberg, Germany, 2013; pp. 407–416. [Google Scholar]
  37. Choi, S. The Integrated Framework for Developing Design of and Evaluating Usability of Medical Devices. Ph.D. Thesis, Chung-Ang University, Ansan, Korea, 2018. [Google Scholar]
  38. Dumas, J.S.; Redish, J.C. A Practical Guide to Usability Testing; Intellect: Portland, OR, USA, 1993; pp. 119–134. [Google Scholar]
  39. Evans, G. Accessibility and user needs in transport design. In Proceedings of the International Conference on Inclusive Design, Royal College of Art, London, UK, 5–8 April 2005; pp. 32–44. [Google Scholar]
  40. Reed, D.; Monk, A. Inclusive design: Beyond capabilities towards context of use. Univers. Access Inf. Soc. 2011, 10, 295–305. [Google Scholar] [CrossRef]
  41. Amoore, J.; Ingram, P. Learning from adverse incidents involving medical devices. BMJ Nurs. Stand. 2002, 32517, 27241–27546. [Google Scholar]
  42. Bevan, N. Measuring usability as quality of use. Softw. Qual. J. 1995, 4, 115–130. [Google Scholar] [CrossRef]
  43. Cho, J.S.; Kwon, T.K.; Hong, J.P. A study of evaluation index development of healthcare rehabilitation device design. Korean J. Sci. Emot. Sensib. 2014, 17, 129–142. [Google Scholar] [CrossRef]
  44. Brooke, J. SUS: A retrospective. J. Usability Stud. 2013, 8, 29–40. [Google Scholar]
  45. Nielsen, J.; Landauer, T.K. A mathematical model of the finding of usability problems. In Proceedings of the INTERACT’93 and CHI’93 Conference on Human Factors in Computing Systems, Amsterdam, The Netherlands, 24–29 April 1993; ACM: New York, NY, USA, 1993; pp. 206–213. [Google Scholar]
  46. ISO DIS 9241-11 Ergonomic Requirements for Office Work with Visual Display Terminals (VDT)s—Part 11 Guidance on Usability, International Organization for Standardization. Available online: (accessed on 10 March 2006).
  47. Peres, S.C.; Pham, T.; Phillips, R. Validation of the system usability scale (SUS) SUS in the wild. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting, San Diego, CA, USA, 30 September–4 October 2013; SAGE Publications: Los Angeles, CA, USA; pp. 192–196. [Google Scholar]
  48. Schmidt, H.; Werner, C.; Bernhardt, R.; Hesse, S.; Krüger, J. Gait rehabilitation machines based on programmable footplates. J. Neuroeng. Rehabil. 2007, 4, 12–19. [Google Scholar] [CrossRef] [PubMed]
  49. Carr, J.H.; Shepherd, R.B. A Motor Relearning Programme for Stroke; Aspen Publishers: New York, NY, USA, 1987. [Google Scholar]
  50. Chan, D.Y.L.; Chan, C.C.H.; Au, D.K.S. Motor relearning programme for stroke patients: A randomized controlled trial. Clin. Rehabil. 2006, 20, 191–200. [Google Scholar] [CrossRef] [PubMed]
  51. Sharples, S.; Martin, J.; Lang, A.; Craven, M.; O’Neill, S.; Barnett, J. Medical device design in context: A model of user–device interaction and consequences. Displays 2012, 33, 221–232. [Google Scholar] [CrossRef]
  52. Díaz, I.; Gil, J.J.; Sánchez, E. Lower-limb robotic rehabilitation: Literature review and challenges. J. Robot. 2011, 2011, 1–11. [Google Scholar] [CrossRef]
  53. Nielsen, J. Guerilla HCI: Using discount usability engineering to penetrate the intimidation barrier. In Cost—Justifying Usability; Bias, R.G., Mayhew, D.J., Eds.; Academic Press: Boston, MA, USA, 1994; pp. 242–272. [Google Scholar]
  54. Donahue, G. Usability and the bottom line. IEEE Softw. 2001, 18, 31–37. [Google Scholar] [CrossRef]
  55. Siegel, D. The business case for user-centered design: Increasing your power of persuasion. Interactions 2003, 10, 30–36. [Google Scholar] [CrossRef]
  56. Virzi, R.A. Refining the test phase of usability evaluation: How many subjects is enough? Hum. Factors 1992, 34, 457–468. [Google Scholar] [CrossRef]
  57. Nielsen, J. Estimating the number of subjects needed for a thinking aloud test. Int. J. Hum.-Comput. Stud. 1994, 41, 385–397. [Google Scholar] [CrossRef]
  58. Spool, J.; Schroeder, W. Testing web sites: Five users is nowhere near enough. In Proceedings of the Extended abstracts on Human Factors in Computing Systems (CHI 2001), Washington, DC, USA, 31 March–5 April 2001; ACM Press: New York, NY, USA, 2001; pp. 285–286. [Google Scholar]
  59. Faulkner, L. Beyond the five-user assumption: Benefits of increased sample sizes in usability testing. Behavior Research Methods. Instrum. Comput. 2003, 35, 379–383. [Google Scholar] [CrossRef]
  60. Woolrych, A.; Cockton, G. Why and when five test users aren’t enough. In Proceedings of the IHM-HCI 2001 Conference, Lille, France, 10–14 September 2001; Cépaduès Editions: Toulouse, FR, USA, 2001; pp. 105–108. [Google Scholar]
  61. Glen, J.M.W.; Lord, M. New product development processes within the UK medical device industry. Med. Eng. Phys. 1996, 18, 670–676. [Google Scholar] [CrossRef]
  62. Lehoux, P.; Saint-Arnaud, J.; Richard, L. The use of technology at home: What patient manuals say and sell vs. what patients face and fear. Sociol. Health Illn. 2004, 26, 617–644. [Google Scholar] [CrossRef]
  63. Ward, J.R.; Clarkson, J.P. An analysis of medical device-related errors: Prevalence and possible solutions. J. Med. Eng. Technol. 2004, 28, 2–21. [Google Scholar] [CrossRef]
  64. Kessner, M.; Wood, J.; Dillon, R.F.; West, R.L. On the reliability of usability testing. In Proceedings of the Extended Abstracts on Human Factors in Computing Systems(CHI’01), Seattle, WA, USA, 31 March–5 April 2001; ACM Press: New York, NY, USA, 2001; pp. 97–98. [Google Scholar]
  65. Walker, M.; Takayama, L.; Landay, J.A. High-fi51delity or low-fidelity, paper or computer? Choosing attributes when testing web prototypes. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting, Baltimore, MD, USA, 30 September–4 October 2013; Sage Publications: Los Angeles, CA, USA, 2002; pp. 661–665. [Google Scholar]
  66. Van Schaik, P.; Ling, J. Modelling user experience with web sites: Usability, hedonic value, beauty and goodness. Interact. Comput. 2008, 20, 419–432. [Google Scholar] [CrossRef]
  67. Kim, H.; Fesenmaier, D.R. Persuasive design of destination web sites: An analysis of first impression. J. Travel Res. 2008, 47, 3–13. [Google Scholar] [CrossRef]
  68. Lindgaard, G.; Dudek, C.; Sen, D.; Sumegi, L.; Noonan, P. An exploration of relations between visual appeal, trustworthiness and perceived usability of homepages. ACM Trans. Comput.-Hum. Interact. 2011, 18, 1–30. [Google Scholar] [CrossRef]
  69. Ben-Bassat, T.; Meyer, J.; Tractinsky, N. Economic and subjective measures of the perceived value of aesthetics and usability. ACM Trans. Comput.-Hum. Interact. 2006, 13, 210–234. [Google Scholar] [CrossRef]
  70. Tractinsky, N.; Katz, A.S.; Ikar, D. What is beautiful is usable. Interact. Comput. 2000, 13, 127–145. [Google Scholar] [CrossRef]
  71. Lee, S.; Koubek, R.J. Understanding user preferences based on usability and aesthetics before and after actual use. Interact. Comput. 2010, 22, 530–543. [Google Scholar] [CrossRef]
  72. Katz, A. Aesthetics, usefulness and performance in user—Search-engine interaction. J. Appl. Quant. Methods 2010, 5, 424–445. [Google Scholar]
  73. Lindgaard, G.; Dillon, R.; Trbovich, P.; White, R.; Fernandes, G.; Lundahl, S.; Pinnamaneni, A. User Needs Analysis and requirements engineering: Theory and practice. Interact. Comput. 2006, 18, 47–70. [Google Scholar] [CrossRef]
  74. Sonderegger, A.; Sauer, J. The influence of design aesthetics in usability testing: Effects on user performance and perceived usability. Appl. Ergon. 2010, 41, 403–410. [Google Scholar] [CrossRef] [PubMed]
  75. Tuch, A.N.; Roth, S.P.; Hornbæk, K.; Opwis, K.; Bargas-Avila, J.A. Is beautiful really usable? Toward understanding the relation between usability, aesthetics, and affect in HCI. Comput. Hum. Behav. 2012, 28, 1596–1607. [Google Scholar] [CrossRef]
  76. Mugge, R.; Schoormans, J.P. Product design and apparent usability. The influence of novelty in product appearance. Appl. Ergon. 2012, 43, 1081–1088. [Google Scholar] [CrossRef]
  77. Dion, K.; Berscheid, E.; Walster, E. What is beautiful is good. J. Personal. Soc. Psychol. 1972, 24, 285–290. [Google Scholar] [CrossRef]
  78. Gupta, A.K.; Dey, A.R.; Shinde, C.; Mahanta, H.; Patel, C.; Patel, R. Theory of open inclusive innovation for reciprocal, responsive and respectful outcomes: Coping creatively with climatic and institutional risks. J. Open Innov. Technol. Mark. Complex. 2016, 2, 16. [Google Scholar] [CrossRef]
  79. Gupta, A.; Dey, A.; Singh, G. Connecting corporations and communities: Towards a theory of social inclusive open innovation. J. Open Innov. Technol. Mark. Complex. 2017, 3, 17. [Google Scholar] [CrossRef]
Figure 1. Rehabilitation robotic device tested in this study.
Figure 1. Rehabilitation robotic device tested in this study.
Sustainability 12 00598 g001
Table 1. Standardized usability assessment tools focusing on two-dimensional screens in computer systems.
Table 1. Standardized usability assessment tools focusing on two-dimensional screens in computer systems.
ApproachDimensions and Factors to Measure Usability
QUISOverall satisfaction, interface (screen, terminology and system feedback, learning factors, system capabilities, technical manuals, online tutorials, multimedia, teleconferencing, and software installation)
SUMIAffect, efficiency, learnability, helpfulness, control
PSSUQSystem usefulness, information quality, interface quality, overall satisfaction
SUSease of use (learnability and usability)
Table 2. Questionnaire structure to measure usability of a rehabilitation robotic device for lower limbs.
Table 2. Questionnaire structure to measure usability of a rehabilitation robotic device for lower limbs.
Question CategoriesAttributesNumber of Questions
Demographic InformationAge4
Period of career
Prior experience
Generality across devicesEffectiveness3
Device-specific questionsRail5
Chest board6
Knee Board3
Handle strap2
Footrest strap2
Table 3. Outline of survey respondents’ demographic characteristics.
Table 3. Outline of survey respondents’ demographic characteristics.
ClassificationUX Design ProfessionalsRehab ProfessionalsLay PeopleTotal Users
Age21–3015153363 (32.14%)
31–4024152968 (34.69%)
41–5012141541 (20.9%)
≥51–6007916 (8.16%)
≥610088 (4.08%)
Total515194196 (100%)
SexMale23362483 (42.35%)
Female281570113 (57.65%)
Total515194196 (100%)
Years of professional experience1–5 years2420N/AN/A
6–10 years2310N/AN/A
≥10 years421N/AN/A
Table 4. EFA results.
Table 4. EFA results.
Factor InterpretationVariable No.Factor LoadingCommunalityEigen Value% of Variance ExplainedCronbach’s Alpha
Sitting on/off010.6130.6453.7137.7360.886
Holding the body350.7230.7253.2746.8220.857
Visual pertinence020.7710.7293.2336.7360.806
Use confidence060.8080.8043.1846.6340.851
Table 5. Comparison of ANOVA results between the primary user groups.
Table 5. Comparison of ANOVA results between the primary user groups.
General QualitiesUse confidenceUX professionals (a)513.730.822.340.099
Rehab professionals (b)513.660.90
Lay people (c)943.430.74
SafetyUX professionals (a)513.880.810.120.885
Rehab professionals (b)513.830.87
Lay people (c)943.910.75
Visual pertinenceUX professionals (a)943.930.805.620.004
Rehab professionals (b)513.660.82
Lay people (c)513.480.71
Device SpecificsPiggybackingUX professionals (a)513.750.74752.740.067
Rehab professionals (b)513.660.7172
Lay people (c)943.460.7401
TransferringUX professionals (a)513.380.89513.110.047
Rehab professionals (b)513.280.8058
Lay people (c)943.030.6787
Supporting the bodyUX professionals (a)944.040.70261.830.164
Rehab professionals (b)513.950.6808
Lay people (c)513.810.7032
Sitting on/offUX professionals (a)513.500.85251.910.150
Rehab professionals (b)513.500.8448
Lay people (c)943.240.6811
Holding the BodyUX professionals (a)513.730.83716.720.002
Rehab professionals (b)513.620.7876
Lay people (c)943.240.6227
Back to TopTop