Design of Medical Devices with Usability in Mind: A Theoretical Proposal and Experimental Case Study Using the LEPRE Device

Abstract

Usability is a critical product feature and is required for widespread market adoption. Standards on usability are highly focused on evaluation procedures and specific aspects, such as software issues or human–machine interaction, whereas the relative scientific literature is very normative oriented. The few methodological works dealing with usability either consider it as one of the many attributes that a particular project must satisfy or present very general methods. No design methods systematically oriented toward the integration of usability and usability-related constraints have been developed to date. This paper presents a usability-oriented model for the design of medical devices and its application to the design of LEPRE, a medical device for upper- and lower-limb robotic rehabilitation. Two methods were used to assess the device’s usability: interviews with experts to outline qualitative evaluations and System Usability Scale (SUS) questionnaires on eight physiotherapists, two physiatrists, and 12 patients, enabling a quantitative assessment. Results support the intention of providing an integrated methodological approach to be applied from the early stages of the project, thus saving time and costs, leading to a more linear product development for this application.

1. Introduction

The usability of a product or service, in conjunction with its safety, is a critical feature required for widespread market adoption of the good itself [1], and even more in the field of medical devices. Nonetheless, few scientific works and regulations address usability methodologically, considering it one of the many attributes that the project must satisfy [2], or otherwise proposing very general methods [3].

According to International Organization for Standardization (ISO), usability is defined as “the extent to which a system, product or service can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use” [3]. The concept of usability is, therefore, extremely broad, and the scientific literature includes further attributes to the list of ISO parameters, interpreting and offering additional shades of meaning to the ISO definition, such as: ease of use, learnability, flexibility, attitude, and memorability [2,4]. Furthermore, since usability is applicable to both software and hardware, it encompasses a wide range of applications, including apps, websites, computer mice, smartphones and machinery. For this paper, usability is meant as described by the ISO definition.

The scientific literature on usability, particularly in the medical field, is very normative-oriented [5,6,7]. Themes such as usability risk assessment [8], usability reports [9], documentation [10] and protocols [11] are of particular interest. Moreover, most of the articles are software-related, with an emphasis on the human–machine interface [12,13,14], evaluation procedures [15,16,17,18], and numerous but very specific application cases [19,20,21]. For instance, Pei et al. [22] described the usability of a novel robotic bilateral arm rehabilitation device for patients with stroke [22], Surma-aho et al. [23] analyzed the usability issues of operating room devices [23]. Heinemann et al. [24] focused on the usability of medical devices for patients with diabetes who are visually impaired or blind [24]. As the application cases cited above demonstrate, in addition to being circumstantial to a particular medical field (stroke, operating room and diabetes), they often do not cover the entire relative range of design possibilities. For instance, regarding stroke technologies, the first application example [22] is restricted to a robotic bilateral arm rehabilitation device, while the case regarding diabetes [24] is confined to patients who are visually impaired or blind. Apart from the application cases, all the usability-related literature mainly concerns software applications and the human–machine interface so it does not extend beyond what is outlined in the standards. Most of all, it does not address the topic of usability with a methodological approach.

For the European market, the normative framework that governs the usability domain of devices in the medical field is comprised of three references: (i) IEC 62366-1:2015 “Application of usability engineering to medical devices” [25], which describes a process for a manufacturer to analyze, specify, develop, and evaluate the usability of a medical device in terms of safety; (ii) ISO/TR 16982:2002 “Usability methods supporting human-centered design” [26], which provides information on human-centered usability methods that can be used for design and evaluation; and (iii) ISO 9241-11:2018 “Ergonomics of human–system interaction—Part 11: Usability: Definitions and concepts” [3], which specifies a framework for understanding the concept of usability and applies to situations where people use interactive systems, products (including industrial and consumer products) and services. As the last reference provides a global perspective on usability, the first two are primarily concerned with the software domain.

In other words, the normative framework analyzes usability almost as a final output of the design process for a medical device, providing indications, for instance, on its post-hoc evaluation. On the contrary, fewer indications are provided about how to actively manage usability-related aspects during the design evolution itself. In addition, the fulfillment of usability-based requirements could strongly affect the final design of a device, especially in the medical field, and it is a given that every unexpected change applied to a process introduces additional costs to a project, e.g., personnel hours, economic costs, or resources.

Accordingly, several works describe design techniques and methodologies devoted to identifying device requirements [27,28], as well as detecting or anticipating, if possible, project criticalities. Among them, design-for-X [29] and open innovation approaches [30] especially emerged, although not specifically devoted to the integration of usability in the design process.

Other approaches, such as Design Thinking and Human-Centered Design, represent valuable, complementary design tools [31,32], even if they do not properly consider design models, as they are not contextualized into all the various stages of product design. In fact, Design Thinking is a new paradigm for dealing with problems in many professions, most notably Information Technology (IT) and Business [33,34], but does not result in either specifically to the medical field or referable to one or more specific stages of the product design. Likewise, Human-Centered Design is an approach to interactive systems development that aims at making systems usable and useful by focusing on the users, their needs and requirements, and by applying human factors/ergonomics and usability knowledge and techniques [35,36]. Although Human-Centered Design does provide guidance on the key aspects to be considered throughout the entire design process, significantly affecting the definition of requirements, this tool is broad and does not define specific engineering tasks. Therefore, according to the authors’ knowledge, no design methods systematically oriented towards the integration of usability and usability-related constraints at all the phases of the design process have been developed up to date.

In this context, the objective of this study is to propose a usability-oriented model for the design of medical devices based on an extension of the concept of usability, which is intended not only as an outcome, but more significantly, as an integrated methodological approach to be applied during the early stages of the project. Therefore, this approach embraces all the stages of the design process. Involves a more linear product development and allows for reducing time and costs

Furthermore, the innovative aspect of this process is also grounded in its practicality and broad applicability, distinguishing it, for example, from the generic iterative approach proposed by the standards [35].

The usability-oriented model will be discussed by demonstrating its application in the design process of the LEPRE medical device for robotic rehabilitation of the lower and upper limbs. This robotic device has an end-effector-based architecture and can perform rehabilitation treatment in different modalities: passive (CPM), active-assisted, or active with biofeedback. It is composed of three main groups, i.e., the frame, the main body, and the monitor. A compact differential system in the main body enables the implementation of any motion profile in the desired plane. The extended description of the device is given in Section 2.2: Case Study: LEPRE Rehabilitation Device.

2. Method

2.1. Usability-Oriented Model for the Design of Medical Devices

The Usability-Oriented Model for the Design of Medical Devices is represented in Figure 1. This results from more than 15 years of research and analysis of scientific literature, regulatory framework, and practical applications. It is the outcome of two implementation steps that transformed and incrementally improved an original model. This original model, called the “soft open innovation model” [37], has been applied in the last 10 years to the design of the five medical devices depicted in previous literature by the authors [37] and was implemented from two points of view: (a) through a more detailed specification of the actors involved in the various stages of the design process, and (b) with the integration of the regulatory aspect [38]. The result of this first improvement is the model named “Open innovation-inspired model for the design process of medical devices” [39]. From this second model, the role of usability was investigated and contextualized within the design process, obtaining the approach proposed in this work.

According to the Usability-Oriented Model for the Design of Medical Devices, shown in Figure 1, the design process begins with a specific unmet need that requires innovation and progresses through the following phases to the final prototype: “Concept Definition”, “Simulations”, “Prototype”, “On-Field Evaluation”, and “Final Prototype”.

In some cases, the emergence of usability issues itself may stimulate the launch of a new innovative project. Referring to Figure 1, the “Goal” is not included in the usability area (green dashed box) because this is not a condition that always occurs. Nevertheless, usability desiderata can be the reason for defining the “Goal”, i.e., the model’s starting point. For example, suppose products on the market in a given area of use have poor usability with regard to a specific function. In that case, an industrial company could cover that market segment by developing a new innovative solution that satisfies users in terms of usability.

Focusing communication on usability is critical during the “Concept Definition” and, in particular, when identifying “Functional Specifications” of the project by addressing with stakeholders the main features that are involved, such as functionality, ease of use, comfort, flexibility and learnability. Interviews are important for eliciting usability requirements [40], particularly during the “Concept Definition” and subsequent phases. Interviews are more effective when conducted in groups [41,42] because conflicting requirements are reduced. These are primarily used in the search for requirements rather than for a posteriori evaluation, emphasizing the usability paradigm shift once more.

The detailed design and early production phases, represented in the scheme by “Simulations” and “Prototype”, remain primarily engineering-related tasks. As a result, usability improvements are less likely to emerge at these stages. However, there are times when technicians become aware of issues during the detailed design stages of a project, like in the case of previously unknown conflicts of requirements. These issues must be discussed with stakeholders, and as a result, new usability requirements may emerge.

Once the prototype has been created, it must pass a test before being used in the field. This check reveals “unexpressed requirements”, or requirements deemed necessary by a stakeholder but not expressed until that point. In most cases, these requirements are related to usability.

If the prototype passes the test, the next phase is the “On-Field Evaluation”. This is the most important phase after the “Concept Definition” because it validates the prototype’s usability. Typically, if the previous steps of the design process were correctly carried out using the usability-oriented approach; in this case, the usability requirements emerge that do not imply design revisions but rather improvements to the prototype under evaluation.

The “Final Prototype” is the result of passing the “On-field evaluation” check, although the implementation of further minor design changes could still occur also after the “Final Prototype” is completed.

The Beginning-of-Life (BOL) phase concludes with the production of the “Final Prototype”. The usability-validated product is then allowed to proceed through the remaining phases of the product life cycle [43]: the Middle-of-Life (MOL), which includes use and maintenance, and the End-of-Life (EOL), which comprises disposal or recycling.

Devices intended for the European market are partially guided to be compliant with usability since they require the set of mandatory procedures of the CE marking to be freely commercialized within the European market itself. Nevertheless, additional usability requirements that lead to improvements may also emerge at subsequent stages, like in after-sales phases, for example, during the post-market surveillance procedures required by the normative framework

2.2. Case Study: LEPRE Rehabilitation Device

The LEPRE Rehabilitation Device is an end-effector-based robotic device for upper- and lower-limb rehabilitation. It is characterized by a compact differential system, with two degrees of freedom that enable the implementation of any motion profile in the desired plane [44], and can work either through pre-set exercises or with exercises customized ad hoc by the clinician. The device is suitable for rehabilitation in passive modality (Continuous Passive Motion, CPM), active assisted, and active modality with biofeedback. Figure 2 depicts the current CE-marked version of the device, validated for use in clinical practice [45,46].

Rehabilitation of both upper and lower limbs is due to the device’s central body’s ability to tilt and thanks to the interchangeability of accessories, allowing for two setups: upper-limb configuration (Figure 3a) and lower-limb configuration (Figure 3b).

The device can also perform rehabilitation treatment in different modalities: passive (CPM), active-assisted or active with biofeedback.

2.3. Suitability Analysis of Usability Evaluation Methods

Several tools exist to elicit requirements. Especially in relation to the usability field, the standard ISO/TR 16982:2002 [26] is fundamental. This technical report provides an overview of existing usability methods that can be used on their own or in combination to support design and evaluation. Methods presented in the standard include the observation of users, performance-related measurements, critical incidents analysis, questionnaire, interviews, thinking aloud, collaborative design and evaluation, creativity methods, document-based methods, model-based approaches, expert evaluation and automated evaluation. In addition, the standard provides guidelines for choosing the optimal method based on the life-cycle process. Specifically, the standard divides the Life Cycle of a project into three macro phases: acquisition and supply processes, development process, and operation and maintenance processes. Since the model presented in this paper concerns the development phase, only the macro phase of development was considered for choosing the most suitable usability evaluation method. In turn, the development process is divided by the standard into three stages: requirements analysis, architectural design, and qualification testing.

In fact, no linear correspondence between the model presented in this paper and the model proposed by the standard can be established. In fact, the latter defines the stages of architectural design and qualification testing as follows:

• Architectural design: “During the design phases, usability methods will be implemented to confirm, modify or refine the previous findings”;
• Qualification testing: “Is the activity where usability methods are applied to test the match with the requirements”.

In the Usability-Oriented Model for the Design of Medical Devices, on the other hand, there is no clear distinction between these two phases because both are evaluated in the various testing phases: the fulfillment of initial requirements is provided by the qualification testing phase of the standard, and the architectural design phase provides the possibility of changes and improvements.

Figure 4 synthesizes the correspondence between the different phase nomenclatures of the standard and the model presented in this paper.

For each of the three stages of development, the standard indicates which usability evaluation methods are most appropriate (see

Table 1. Usability evaluation methods recommended based on the development stage.

Development Stages	Usability Evaluation Methods
	Observation of Users	Performance-Related Measurements	Critical Incident Analysis	Questionnaires	Interviews	Thinking Aloud	Collaborative Design and Evaluation	Creativity Methods	Document-Based Methods	Model-Based Methods	Expert Evaluation	Automated Evaluation
Requirements analysis	++	+	+	++	++	++	+	+	+	+	+
Architectural design	+	++		+	+	++	+	++	++	+	+	+
Qualification testing	+	++	+	++	++	+	+		+	+	+	+
Legend
++	Recommended
+	Appropriate
empty cell	Neutral

for an extract from the standard).

For the requirement analysis phase, the standard identifies the following methods as most recommended (i.e., methods marked “++”): observation of users, questionnaires, interviews, and thinking aloud. The observation of users, in the sense of a collection of information about the behavior and the performance of users, is not feasible in our case as both clinical actors and patients cannot fully interact with the device as a functioning prototype has not yet been produced at this early stage of defining the specifications. For the same reason, thinking aloud, intended as the verbalization by users of their ideas, beliefs, expectations, doubts, and discoveries during their use of the system under test, is not applicable as there is no complete prototype. Moreover, valid questionnaires in these early stages are qualitative and aimed at eliciting opinions on the device. Consequently, interviews are the best tool for evaluating usability in the requirements analysis phase as they allow for greater flexibility than the qualitative questionnaires and enable unexpressed opinions and requirements to be more easily revealed.

Since, for the model presented in this paper, the two phases of architectural design and qualification testing are not separate, a combined evaluation was carried out to understand the best usability evaluation method. The method that would be most suitable for these phases is the performance-related measurements, as it has the higher mark (“++”) for both phases. The commonly used quantifiable performance measurements include time spent to complete a task, number of errors, and number of tasks that can be completed within a predefined duration. It was decided not to adopt this method for the evaluation of usability in these phases because the large number of tasks that can be performed with the device would have made a complete evaluation difficult and because this tool does not allow an evaluation of the overall device but in terms of human–machine interface only.

Consequently, the methods most suited by the standard to assess usability at these stages are questionnaires, interviews, thinking aloud and document-based methods (i.e., those marked with a “++” and a “+”). The latter method was not considered, as it involves pre-existing documentation, which is, however, not available as the first prototype is being evaluated. Interviews were continually held with healthcare personnel, who also reported the patients’ feelings about the device in their care. Interviews also made it possible to replace and supplement the thinking aloud, avoiding specific tests and leaving the clinical actors free to use the device according to instructions for use and clinical practice. Questionnaires are mostly employed in this phase, as they allow a quantitative evaluation of usability concerning the whole device and can be applied to both clinical actors and patients.

Following the instructions of the standard, the tools that emerged as the best ones to support the design and evaluate usability were interviews, especially in the initial stages indicated by the standard under the heading of “Development–Requirements Analysis,” and questionnaires, especially in the final stages indicated by the standard under the heading of “Development–Architectural Design” and “Development–Qualification Testing”. Therefore, in the present work, the Usability-Oriented Model for the Design of Medical Devices has been applied to the LEPRE Rehabilitation Device (Polibrixia s.r.l., Padova, Italy), and interviews and questionnaires were used to evaluate the usability of the device.

2.4. Usability Evaluation Protocol

Two methods available in the scientific literature [6,15] and the standards [26] were used to evaluate the obtained usability of the LEPRE Rehabilitation Device: interviews to outline a first qualitative evaluation and questionnaires to enable a quantitative assessment. The subjects provided the proper consent for the data treatment.

The interviews were conducted with more than 15 physiatrists (at least 45 years old) and physiotherapists (at least 35 years old) with at least 10 years of experience from different rehabilitation clinical facilities, treating patients with acute-phase impairments with orthopedic and neurological genesis by the same trained operator. The interviews were conducted throughout the different stages of the device design. In particular, as the phases progressed, the interview questions went from being open-ended to increasingly specific ones. For example, in the early stages of “Concept Definition”, typical questions were: “What are the rehabilitation modalities that you expect from the device?” or “What are the trajectories that the device should be able to perform?”. Differently, at the end of the prototyping phases, the usual questions were: “Which usability criticalities do you think can be present with the implemented structural shape?” or “Can the usability of the 3D mouse be improved by modifying its position?”. A multidisciplinary team of three engineers analyzed the answers provided by the experts to capture suggestions or desiderata aimed at improving the device’s usability. In the current work, usability desiderata were considered new design constraints and translated into device requirements.

For the quantitative analysis, the System Usability Scale (SUS) [48,49,50] was used, i.e., a survey composed of ten statements with a final score ranging from 0 to 100, which is the most valid and widely used questionnaire, both scientifically and industrially [51]. In particular, the applied SUS totally coincides with the original version proposed by Brooke [48,49,50], but for the translation of the statements in the Italian language. Appendix A presents the detail of the questions used and the formula to evaluate the SUS value. Generally, a usability rating is considered fully acceptable when it exceeds the SUS value of 70:

Table 2. Adopted convention for the interpretation of the SUS values as Adjective Ratings. The interpretation complies with Bangor et al.’s classification [52].

Adjective	SUS Value
Worst Imaginable	12.5
Awful	20.3
Poor	35.7
OK	50.9
Good	71.4
Excellent	85.5
Best Imaginable	90.9

collects the adopted convention for interpreting the ratings, compliant with the values proposed by Bangor et al. in [52].

In particular, the SUS questionnaire tool was used to assess the usability of the device at three distinct stages of the development of LEPRE. The device usability was evaluated for the first time after the first prototype was created: this moment is indicated in the following elaborations as T0. As it was impossible to assess patient usability because the device had not been CE marked yet, the questionnaire was distributed to eight physiotherapists and two physiatrists. After the device was CE-marked, a clinical investigation in accordance with ISO 14155:2020 [53] was carried out to evaluate the LEPRE device’s usability on patients. At that moment, as indicated in the following elaborations as T1, the questionnaire was submitted to 12 post-stroke patients, both ischemic (50%) and hemorrhagic (50%), and once again to the clinical operators: the same eight physiotherapists and the same two physiatrists. The SUS evaluation was then repeated on the same patients at the end of the second therapy session with the device to capture possible evolution in the evaluation. This moment is indicated in the following elaborations as T2. For both the questionnaires, the SUS values were analyzed with descriptive statistics, and in particular, computing minimum and maximum values (min and max), as well as mean and standard deviation (SD) values, of the data samples (combined in mean ± SD). In addition to the exposed parameters, the p-value calculated according to the Mann–Whitney U-test with a level α = 0.05 is also reported to analyze statistical consistency.

In addition to the overall SUS values as a function of groups of participants (physiotherapists, physiatrists, and patients), which represent the primary endpoints, SUS results were also analyzed by considering the two components of which it is composed: Usability (SUS(U)) and Learning (SUS(L)). In fact, among the 10 questions of which the SUS questionnaire is constituted, in accordance with what is reported in the literature [22], eight questions (Q1, Q2, Q3, Q5, Q6, Q7, Q8, Q9) are intended to express an evaluation in the usability domain (SUS(U)) and two (Q4, Q10) in the learning domain (SUS(L)). Appendix A presents the detail of the formula to evaluate the value of the SUS components: Usability (SUS(U)) and Learning (SUS(L)).

Moreover, the Mann–Whitney U-test and single-sample t-test with significance level α = 0.05 are applied to compare the physiotherapists’ scores to the average SUS score obtained by Sauro [54], who used a population of 5000 different products that have been sold in the market.

Regarding the quantitative analyses, the flowcharts of the activities conducted to obtain the resulting SUS values are presented in Figure 5 and Figure 6.

Table 3. Study population description.

Category	Parameter	Value
Patients	Age	54.08 ± 9.77
	Sex	M = 8, F = 4
	Stroke type (I = Ischemic, H = Hemorrhagic)	I = 6, H = 6
Physiotherapists	Age	44.88 ± 7.79
	Sex	M = 5, F = 3
	Experience (years)	19.63 ± 7.48
Physiatrists	Age	48.00 ± 1.41
	Sex	M = 1, F = 1
	Experience (years)	18.50 ± 2.12

summarizes the descriptive data of the participants involved in the study.

3. Results

The Usability-Oriented Model for the Design of Medical Devices was adopted to develop the LEPRE Rehabilitation Device (Polibrixia s.r.l., Padova, Italy).

The following are examples of changes, improvements, and implementations that occurred in the LEPRE design process due to implementing the usability-oriented method. These are usability requirements, in addition to those prescribed by regulations, that results from the extension of the concept of usability foreshadowed by the model’s paradigm change. The following examples are graphically depicted in Figure 5: Carter Shape, Structural Shape, Mirror Function, and Mouse 3D Height. The figure also indicates the suitability of the proposed model with respect to the project timeline, indicating at which stage of the LEPRE design process the usability changes were performed.

Furthermore, usability evaluation tests were conducted at various stages of the device development to assess the usability achieved thanks to the application of the usability-oriented model. The moments when usability measurements were performed, i.e., T0, T1 and T2, are shown in blue in Figure 7.

3.1. Carter Shape

The original LEPRE Carter was designed to have the shape depicted in Figure 8a to maximize the internal spaces. However, during the early stages of the “Concept Definition”, the following usability requirement emerged from interviews with doctors and physiotherapists: (u1) the device must allow patients to lean on its frontal part (circled in green in the figure) during the rehabilitation therapy on the upper limbs. This necessity does not arise from a specific pathology but might arise because of a particular physical condition, prominent breasts, or because of the set law of motion, which involves reaching the device workspace most distal from the patient’s trunk (a condition to be avoided in the exercise setting).

The newly designed LEPRE carter (Figure 8b) presents a radius of curvature of 90 mm, in contrast to the initially planned 8 mm. In this way, through simple geometric considerations, it turns out that the reduction of the interior space (considering the four corners of the crankcase) results to be about 6900 mm².

The carter usability requirement impacted aesthetics, safety, mechanical and electronic structure and device market share.

The first affected aspect is the aesthetics. For the carter to be useful, the radii of curvature must be much more pronounced.

Instead, safety is strengthened on two fronts. First, mechanical safety improves, as more pronounced radii of curvature allow for safer and more comfortable support of the patient’s trunk (Figure 9). Above all, the new usability requirement necessitates the relocation of the emergency push bottom (element in red in Figure 8), which would otherwise be difficult to access by the operator (physician or physiotherapist) in the event of an emergency, especially when the device is in the lower-limb rehabilitation therapy configuration.

Furthermore, the new usability requirement affected the mechanical and electronic structure within the carter, particularly in terms of overall dimensions. More pronounced curvature radii, the carter’s longitudinal dimensions being equal, imply a reduction in the space available for the internal mechanical and electronic structure.

Finally, thanks to the possibility of the carter supporting the trunk, the inclusion criteria of the device are expanded, allowing even patients who need frontal support to perform upper-limb rehabilitation therapies with the device.

3.2. Structural Shape

The Structural Shape of the LEPRE chassis was initially designed, as represented in Figure 10a, to maximize the device stability. After the first prototype was built, the primary usability requirement from clinician interviews was: (u2) the ability to perform onoliteral rehabilitation therapies laterally with respect to the device’s central body. This unexpressed requirement was translated into the possibility of positioning the patient on the wheelchair at the side of the device. For example, if the patient needed to perform rehabilitation exercises only on the right limb, they would position themselves to the left of the device’s central body. This functionality was not possible with the original structural form. The structure currently implemented in the CE-marked device (Figure 10b) was designed to fulfill the new requirement.

The new usability requirement for the LEPRE structural shape influenced aesthetics, mechanical structure, safety, and electronic and software features.

Due to the revision of the structural shape of the chassis, the device aesthetics have been drastically altered.

New load analyses and structural simulations were required from a mechanical structural standpoint to develop a new reliable, and robust solution.

Furthermore, the device’s stability must be tested to ensure safety according to standards [55]. As a result, new stability tests were required.

However, the new requirement had the greatest impact on electronics and software functionality. In fact, it was necessary to implement the device and the accessories of new electronic components capable of recognizing the working side of the device and implementing all relevant software features. It was mandatory, for example, to prevent the setting of a bilateral rehabilitation exercise when only one tool is mounted or to set a rehabilitation exercise on the incorrect side.

3.3. Mirror Function

During the “on-field evaluation”, the following new usability requirement emerged: (u3) the clinical operator must be able to perform the device movements when setting the trajectory foreseen by the rehabilitation exercise according to their position (relative reference system) rather than the device absolute reference system. The mirror function (Figure 11) was implemented to meet this requirement. This function allows users to mirror motion commands, invert the clockwise and anticlockwise sense, and change the horizontal translation direction on both the Mouse 3D and the virtual console.

The new usability requirement for the Mirror Function has changed how the software operates and the device’s safety.

The first domain affected by the new requirement was undoubtedly software, as it was necessary to implement the previously unanticipated Mirror Feature.

However, the most significant impact was on device safety, particularly in the medical field. The new function implies that the clinical operator better manages the fundamentals for the setting of robotic rehabilitation therapies. The new function prevents the device from unwanted and potentially dangerous motions that could harm the patient.

3.4. Mouse 3D Height

Although the device was successfully operating at full capacity in the hospital after receiving the CE mark, the periodic interviews revealed a new usability requirement: (u4) Mouse 3D Height must be increased by a few centimeters. Therefore, a minor design review was required (Figure 12), based on which the height of the Mouse 3D was raised by 40 mm.

The new usability requirement for Mouse 3D Height affected both the mechanical aspect and the ease of use.

A minor design review was performed to meet the requirement, resulting in mechanical modifications to the monitor support.

However, this new usability requirement’s major impact was improving the ease and comfort of using the Mouse 3D.

3.5. Usability Evaluation

As expected, the performed interviews did not reveal quantitative values to be analyzed but rather qualitative suggestions and requirements regarding functionality and usability to be implemented in the device as modifications, improvements, or new functions. In particular, it was possible to elicit requirements that involved: the modification of the Carter Shape (u1) and the Structural Shape (u2); the improvement of the usability of the Mouse 3D (u4); and the implementation of the new Mirror Function (u3). In addition, the involved changes impact multiple areas of influence, from Aesthetics to Safety.

Concerning the quantitative analyses, Appendix B presents the collected data’s details. Instead, the SUS results elaborated for groups of participants (physiotherapists, physiatrists, and patients), i.e., the primary endpoints, are shown in

Table 4. SUS results for groups of participants.

Category	T0	T1	T2
Patients	\	75.00 ± 11.48	85.00 ± 9.35 (p = 0.028)
		min = 60.00	min = 75.00
		Max = 100.00	Max = 100.00
Physiotherapists	64.38 ± 12.30	78.13 ± 9.98 (p = 0.028)	\
	min = 40.00	min = 60.00
	Max = 77.50	Max = 90.00
Physiatrists	85.00 ± 3.54	86.25 ± 1.77 (p = 0.667)	\
	min = 82.50	min = 85.00
	Max = 87.50	Max = 87.50

.

In addition, secondary endpoints are shown in

Table 5. SUS(U) and SUS(L) results for groups of participants.

Category		T0	T1	T2
Patients	SUS(U)	\	77.34 ± 10.92	86.56 ± 8.87 (p = 0.039)
			min = 62.50	min = 75.00
			Max = 100.00	Max = 100.00
	SUS(L)	\	65.63 ± 17.78	79.17 ± 15.39 (p = 0.114)
			min = 37.50	min = 62.50
			Max = 100	Max = 100.00
Physiotherapists	SUS(U)	63.67 ± 11.80	78.91 ± 8.64 (p = 0.010)	\
		min = 43.75	min = 65.63
		Max = 78.13	Max = 93.75
	SUS(L)	67.19 ± 22.10	75.00 ± 21.13 (p = 0.442)	\
		min = 25.00	min = 37.50
		Max = 100.00	Max = 100.00
Physiatrists	SUS(U)	82.81 ± 2.21	90.63 ± 4.42 (p = 0.333)	\
		min = 81.25	min = 87.50
		Max = 84.38	Max = 93.75
	SUS(L)	93.75 ± 8.84	68.75 ± 8.84 (p = 0.333)	\
		min = 87.50	min = 62.50
		Max = 100.00	Max = 75.00

: that is, the components of SUS, Usability and Learning.

The mean SUS score incurs a statistically significant change (p = 0.028) from 75.00 ± 11.48 to 85.00 ± 9.35 for patients and from 64.38 ± 12.30 to 78.13 ± 9.98 for physiotherapists. In contrast, for physiatrists, the change in mean SUS score from 85.00 ± 3.54 to 86.25 ± 1.77 is not statistically significant (p = 0.667).

The mean value of the usability component SUS(U) always increased, but the change is statistically significant only for patients and physiotherapists. Indeed, SUS(U) increased from 77.34 ± 10.92 to 86.56 ± 8.87 (p = 0.039) for patients, from 63.67 ± 11.80 to 78.91 ± 8.64 (p = 0.010) for physiotherapists, and from 82.81 ± 2.21 to 90.63 ± 4.42 (p = 0.333) for physiatrists. On the other hand, the mean value of the learning component SUS(L) increased only for patients and physiotherapists, and in all cases, the changes were not statistically significant. The SUS(L) increased from 65.63 ± 17.78 to 79.17 ± 15.39 (p = 0.114) for patients, from 67.19 ± 22.10 to 75.00 ± 21.13 (p = 0.442) for physiotherapists and decreased from 93.75 ± 8.84 to 68.75 ± 8.84 (p = 0.333) for physiatrists.

In addition, the Mann–Whitney U-test and single-sample t-test comparing the physiotherapists’ score to the average SUS score obtained by Sauro revealed a significant difference: p_Mann = 0.007, p_Ttest = 0.005, with the LEPRE mean SUS 12.07 higher.

4. Discussion

The presented Usability-Oriented Model for the Design of Medical Devices addresses usability methodologically, overcoming the limitations revealed by the analysis of scientific literature and regulations. To apply this method, no specific conditions must be a priori met, so we expect that the model could be easily extended to other contexts or, in general, to medical product development, provided the proper fine adaptations. Indeed, the involved stakeholders could greatly vary, depending on the product’s function and the main entailed analysis aspects. For surgical devices, the focus will be, for instance, on the grip and/or the movement ability, whereas the elements evaluated in the case of the LEPRE device are expected to be shared among electromedical devices for rehabilitation. Nonetheless, the continuous interaction provided by the proposed method allows for the integration of the different visions of the various stakeholders, identifying solutions that represent a trade-off between the technical needs indicated by designers and the demands of medical stakeholders.

As the proposed model emphasizes, the usability aspect encompasses the entire development process and regulation framework. The paradigm shift introduced by this model corresponds to the extension of the notion of usability, intended as an integrated methodological approach to be applied since the early stages of the project rather than as an outcome or a simple attribute, as it is commonly seen in literature or in normative contexts. Furthermore, within this model, the concept of usability has been broadened to all elements of the product or service being designed rather than being limited to software aspects or the human–machine interface.

Finally, the scientific literature and standards cover the topic of usability in a broad sense. For example, ISO 9241-210:2019 proposes a general design approach based on iterative repetition of four steps: (i) understanding and specifying the context of use; (ii) defining user requirements; (iii) producing design solutions; and (iv) evaluating the design. In contrast, the usability-oriented model proposed in this paper contextualizes and specifies the influence of usability in the typical phases of a medical device design process.

Furthermore, the introduced paradigm shift requires that usability represents both the goal to be achieved and the guideline as well of the whole design process and product development. In these terms, compared to the models in the literature for both design process, such as the model of Pahl and Beitz [56], and product development, like the one described by Ulrich and Eppinger [57], the proposed usability-oriented model presents two main peculiar characteristics: (a) details the phases of the design process and product development for the medical field, showing the involved actors (Users, Physicians, Operators and Engineers), and (b) emphasizes the need to focus throughout the development of the device, in addition to the regulations, on usability. Moreover, the application of the usability-oriented model enables the identification of all the usability requirements in reference to a particular functional group but does not exclude the possibility of applying the approaches indicated by the literature in the process of design and development of its constituent components. For example, once the shape of the Carter has been determined thanks to the usability requirements that emerged, the design and development of its components can follow the phases indicated by the method of Pahl and Beitz (Concept, Embodiment and Detail) or those indicated by Ulrich and Eppinger (Planning, Concept Development, System-Level Design, Detail Design, Testing and Refinement). However, without the definition of the usability requirements resulting from the application of the usability-oriented model, the approaches of the literature would generate products that work and perform well but could reveal to be usability non-compliant.

Compared to other design support tools existing in the scientific literature, such as Design Thinking and Codesign, the proposed model presents strong affinities and, in some aspects, embodies them. In fact, the five stages of the design thinking defined by the Hasso Plattner Institute of Design at Stanford [58] (empathize, define, ideate, prototype, and test) or the four ones of the typical design thinking framework known as Double Diamond [59] (Discover, Define, Develop and Deliver) reasonably reflect the macro phases of the proposed usability-oriented model (“Concept Definition”, “Simulations”, “Prototype”, “On-Field Evaluation”, and “Final Prototype”). At the same time, the principle of involving end users in the design established by the Codesign paradigm [60] finds correspondence in the usability-oriented model in making explicit the actors involved in the individual stages and in dedicating extensive attention to constructive confrontations, e.g., through interviews and questionnaires, involving various stakeholders throughout the design process.

However, both Design Thinking and Codesign are not contextualized to the various stages of product design. Furthermore, even the integration of the two paradigms [61] does not resolve the weaknesses of the two individual approaches since it results in the introduction of co-design sessions in models composed of generic steps (seeing, knowing, thinking, acting and reflecting) that are not related to the stages that occur in industrial practice. In contrast, the presented usability-oriented model overcomes the two paradigms, although it implements their principles. In fact, whereas the frameworks described by Design Thinking are general, the proposed model details the specific stages; for example, by distinguishing between functional and technical specifications in the “Concept Definition” or by differentiating the “Simulations” from the “Prototype” in the detail stage.

Moreover, in contrast to Codesign, the usability-oriented model not only indicates the importance of end-user involvement but also delineates the stakeholders involved in the individual stages of the design process. For example, there are phases, such as in the definition of “Functional Specifications”, where the involvement of patients (users), operators, and physicians is essential. In contrast, other phases, such as “Simulations” or “Prototype,” remain primarily engineering-related tasks. In addition, compared to the two paradigms, the presented model explicitly introduces the influence of usability and regulations affecting all stages of design, anticipating various critical issues and encouraging the arising of unexpressed requirements.

Therefore, the novelty and usefulness of the presented usability-oriented model consist of the extension of the concept of usability applied to a detailed design model in which the various actors involved in the different stages are also made explicit. More specifically, the novelty lies in the paradigm shift of considering usability not as an outcome but systematically throughout the various stages of the design process. The usefulness lies in the contextualization of this paradigm shift using a practical and specific design model that presents the various phases pursued in the medical design practice.

Table 6. Overview of the comparison between the usability-oriented model and the presented paradigms/models (Y: Yes, N: No).

Paradigm/ Method	A Priori Condition	Different Vision	Different Stakeholders	Define Engineering Tasks	Regulation Oriented	Usability in All Stages
Design Process	Y	N	N	Y	N	N
Product Development	Y	N	N	Y	N	N
Design Thinking	N	Y	N	N	N	N
Human-Centered Design	N	Y	Y	N	Y	Y
Codesign	N	Y	Y	N	N	N
Usability-Oriented Model	N	Y	Y	Y	Y	Y

summarizes the comparison with respect to the presented paradigms and methods.

With specific reference to the LEPRE medical device application case, the adoption of the Usability-Oriented Model for the Design of Medical Devices highlighted several benefits. The qualitative investigation performed through informal interviews with experts emphasized suggestions for design improvement, as the analyzed usability requirements.

Quantitative and statistical analyses could be useful tools to assess the changes and improvements that have been described. However, these evaluations cannot be applied if no additional measurable data is introduced. In our case study, the Carter Shape improvement, where the numerical values of the radii of curvature were given, and the Mouse 3D Heigh modification, where the specific value of the uplift was given, were the only cases suitable for analyses based on the numerical evaluation. In fact, the case of the Structural Shape resolves in a radical change in shape; therefore, no values are meaningful for numerical comparison, while the Mirror Function is a newly introduced function, which thus has no comparison term. In addition, it would have been useful to punctually compare results in terms of usability evaluation before and after each change to understand its impact through differentials. However, this was not possible because the best design support and usability assessment tool, according to the standard ISO/TR 16982:2002, was in the early stages of the device development interviews. Nonetheless, these tools did not provide quantitative results to be analyzed but rather qualitative suggestions and requirements regarding functionality and usability to be implemented in the device as modifications, improvements, or new functions.

Table 7. Overview of fields affected by usability requirements that emerged in the illustrative case.

Impact Areas	Usability Requirements
	(u1) Carter Shape	(u2) Structural Shape	(u3) Mirror Function	(u4) Mouse 3D Height
Aesthetic field	Aesthetics	Aesthetics	x	x
Safety field	Safety	Safety	Safety	x
Mechanical field	Mechanical Structure	Mechanical Structure	x	Minor Design Review
Electronic field	Electronic Structure	Electronic Features	x	x
Software field	x	Software Features	Software Features	x
Other fields	Market Share	x	x	Ease of use

summarizes the areas affected by these requirements, detailing the nature of the modification required for each field.

Table 6 shows for each usability requirement the impacted areas, specifying their nature where appropriate. For example, focusing on the “Mechanical field”, the first two usability requirements ((u1) and (u2)) affected the “Mechanical Structure”, while the modification regarding the Mouse 3D (u4) introduced “Minor Design Review”. At the same time, in terms of the “Electronic field”, the revision regarding the Carter (u1) changed the “Electronic Structure”, while the Structural Shape (u2) concerned “Electronic Features”. Similarly, in the “Other fields,” the usability requirements related to the Carter (u1) expanded the “Market Share”, while changing the height of the Mouse 3D improved the “Ease of Use”. For “Aesthetic field”, “Safety field” and “Software field”, no further detail about the impact area was provided as it would not carry further relevant information. In addition to specifying the impacted areas, the table also provides an overview of which specific fields were impacted. Table 6 also depicts the broadening of the concept of usability since the presented usability requirements are not limited to influencing the software area or the human–machine interface

As a general outcome, time-to-market and project costs are significantly reduced, as critical design issues can be anticipated and emerging usability requirements promoted thanks to the usability paradigm shift. Table 6 reveals that the earlier emerging usability requirements are also the most impactful in terms of the number of affected fields and relative impact on the overall development process.

As a result, the number of pre-validation prototypes is reduced, and the design process follows a linear path, as evidenced by the increase in SUS score recorded for all categories of participants involved. The analysis of SUS results shows, as summarized in Table 4 that for physiotherapists, an increase from 64.38 ± 12.30 to 78.13 ± 9.98 (from T0 to T1), for physiatrists from 85.00 ± 3.54 to 86.25 ± 1.77 (from T0 to T1), and for patients from 75.00 ± 11.48 to 85.00 ± 9.35 (from T1 to T2). In particular, for physiotherapists and patients, there is statistical consistency as the p-values recorded, in both cases, p = 0.028, are such as to reject the null hypothesis. On the contrary, there is no statistical consistency in the case of physiatrists as the p-value is found to be p = 0.667 and, therefore, not such as to reject the null hypothesis.

These last statistical results are besides strongly affected by the small sample size available for physiatrists, which corresponds to only two participants. In addition, Table 4 also reveals that for all three categories of participants, standard deviation decreases and minimum recorded values increase between consecutive measures of SUS. These are two very significant results since a reduction in standard deviation indicates greater agreement in usability assessment. The increase in the minimum recorded values means that an increasing number of participants consider the device to be at least acceptable in terms of usability. In fact, for physiotherapists, the standard deviation decreased from 12.30 to 9.98 (from T0 to T1), for physiatrists from 3.54 to 1.77 (from T0 to T1), and patients from 11.48 to 9.35 (from T1 to T2).

On the other hand, in terms of minimum value, it increased from 40.00 to 60.00 for physiotherapists (from T0 to T1), shifting from a usability between “Poor” and “OK” to a usability between “OK” and “Good”, increased from 82.50 to 85.00 for physiatrists (from T0 to T1), but still reg at an almost “Excellent” level of usability, and increased from 60.00 to 75.00 for patients (from T1 to T2), upgrading from a usability between “OK” and “Good” to a usability between “OK” to “Excellent”.

After implementing all the improvements that emerged from the application of the presented usability model, the CE-marked device (at instant T1) shows a usability level between “Good” and “Excellent” for the three categories of participants, which is definiable. Moreover, the SUS value emerging for physiotherapists and patients is rather close, whereas it is significantly higher in the case of physiatrists—this is the reason for the different uses of the device by the categories. Indeed, different users have different roles; thus, their usability scores provide different indications. In the case of the LEPRE device, physiatrists are primarily responsible for setting up the rehabilitation protocol, physiotherapists deal with setting up the exercise (defining the trajectories and the remaining parameters), whereas the patients perform the exercises guided by the on-screen instructions and feedback from the device. Therefore, SUS values at T1 indicate that the usability of the CE-marked device is acceptable with respect to rehabilitation protocol and exercise setting and exercise execution.

The results of the SUS components always increase their value, except in the case of physiatrists. In fact, the SUS(U) increased from 82.81 ± 2.21 to 90.63 ± 4.42; in contrast, the SUS(L) decreased from a value of 93.75 ± 8.84 to 68.75 ± 8.84 in the case of physiatrists. Therefore, the device is perceived as more complex to use. Nevertheless, the SUS score measured for physiatrists grew due to the increase of the SUS(U) component.

Ultimately, the SUS measures involving clinical stakeholders (physiotherapists and physiatrists) indicate the influence on the device usability evaluation of the changes introduced through the application of the usability-oriented model, whereas the SUS measures on patients reveal how much using a device multiple times affects the overall device usability evaluation. Given the numerical results, the introduced changes have improved the device’s usability, which grows further at a second session.

Furthermore, the results of the Mann–Whitney U-test and single-sample t-test comparing the physiotherapists’ score to the average SUS score obtained by Sauro indicate that the usability of the LEPRE device is statistically different from those of most market products, with LEPRE mean SUS higher of 12.07. Moreover, implementing the usability-oriented model has allowed for improvements or the introduction of new functions, influencing the mechanical, electronic, and software aspects and enabling the increase of the potential device market share.

More specifically, at the beginning of the project, three prototypes were planned to be realized before achieving the final one. However, a single, final prototype was produced thanks to the application of this model, whose changes impacted economically as one prototype. Therefore, the estimated economic saving compared to the forecast is 33%, corresponding to the saving of an intermediate prototype. On the other hand, the greatest advantage is in the product development timeline since this method saves time. Although no specific evaluation has been performed to quantify the overall time saving, this aspect can be concretely assessed, considering that only one prototype was made thanks to the early modifications. Half of the time required to design and make a prototype is estimated to be the impact of the early modifications. Consequently, the estimated time saving compared to the forecast is 50%, corresponding to the saving of one-and-a-half intermediate prototypes. The reported percentages are estimations, as there is no comparison term; that is, no comparison device is designed without the aid of the Usability-Oriented Model. Furthermore, the proposed estimates are for the specific case under consideration. However, economic savings of 10% to 40% and time-saving of 20% to 60% are estimated through the application of the model. These estimations could be demonstrated and further investigated in a specific study on the design of different devices and possibly in different fields of application.

5. Conclusions

The Usability-Oriented Model for the Design of Medical Devices presented in the current paper reduces project time and cost but also leads to linear product development. The novelty introduced by the model consists in the extension of the notion of usability, intended as an integrated methodological approach to be applied since the early stages of the project rather than as an outcome or a simple attribute, and in its practicality and broad applicability.

The LEPRE design applicative case demonstrates some of its potentiality. In fact, through the usability paradigm shift, many criticalities of the project have been anticipated, and several unexpressed requirements have emerged. For the application case, the efficiency of the model emerges from the economic-temporal saving values (33% in economic terms and 50% in temporal ones).

Future research could focus on analyzing the influence of usability in the post-CE-marking phases; in this work, it was only mentioned as the focus was on the design process. Specific studies could also be developed to better investigate and validate the reported values of time-economic saving resulting from the application of the proposed model with different application cases and in different application areas.

The various components that constitute SUS could also be investigated in future research, introducing and investigating new metrics, even more specifically, such as measuring usefulness, perceived benefit, comprehensiveness, interpretability and ease of use of the presented usability-oriented methodology.

Finally, although the presented model is centered on medical device design, its applicability could be easily extended to other fields, or further investigations could be performed within the same sector by broadening the involved stakeholders.