Usability Evaluation of the SmartWheeler through Qualitative and Quantitative Studies

Background: Intelligent powered wheelchairs remain a popular research topic that can improve users’ quality of life. Although our multidisciplinary research team has put a lot of effort into adding features based on end-users needs and impairments since 2006, there are still open issues regarding the usability and functionalities of an intelligent powered wheelchair (IPW). Methods: For this reason, this research presents an experience with our IPW followed by a study in two parts: a quantitative one based on the System Usability Scale (SUS) questionnaire and a qualitative one through open questions regarding IPW functionalities with novice users, e.g., IPW non-users. These users never used an IPW before, but are users and aware of the impacts of the technology used in our IPW, being undergraduate to postdoctoral students and staff (faculty, lecturers, research engineers) at the Faculty of Engineering of Université de Sherbrooke. Results: The qualitative analyses identified different behaviours among the novice users. The quantitative analysis via SUS questionnaire done with novice users reports an “okay” rating (equivalent with a C grade or 68 SUS Score) for our IPW’s usability. Moreover, advantages and disadvantages opinions were gathered on the IPW as well as comments which can be used to improve the system. Conclusions: The results reported in these studies show that the system, e.g., IPW, was judged to be sufficiently usable and robust by novice users, with and without experience with the software used in developing the IPW.


Introduction
There are currently more than 1 billion disabled people in the world [1]. According to the World Health Organization (WHO), a disabled person is anyone who has "a problem in body function or structure, an activity limitation, has a difficulty in executing a task or action; with a participation restriction". Moreover, 75 million people need a wheelchair on a daily basis. This represents 1% of the world's population, which is twice Canada's population. In 2012, Statistics Canada [2] reported that among the estimated 3,775,920 individuals with a disability, 8% used a wheeled mobility device, 10% reported an unmet need for an additional device: a power wheelchair (5%), a mobility scooter (4%) or a manual wheelchair (1%). While for non-users who needed a wheeled mobility device, the most common was a mobility scooter (60%), followed by a power wheelchair (21%) and a manual wheelchair (19%).
Powered wheelchairs (PW) were designed to improve the quality of life of people for which manual wheelchairs became difficult to use [3]. For similar reasons, intelligent power wheelchairs (IPWs) were developed to help the needs [4,5] of people with cognitive/motor/sensory impairment and their caregivers [6], whether it is due to disability or disease, by providing collision avoidance, navigation support and adapted control modes. ticipant. Secondly, (i) the IPW usability through an online questionnaire on Lime Survey (https://sondage.crir.ca/index.php?r=survey/index&sid=488847, accessed on 1 January 2021) consisting of the System Usability Scale (SUS) questionnaire [20] and (ii) the IPW's functionality through four open-ended questions were quantified.
Briefly, our results show that by involving novice users, new ways of improvements in terms of functionalities have been reported. Moreover, in terms of usability the IPW is reported to fit the requirements for which it has been designed.

Materials and Methods
In this study, we used an evolution of the IPW that was used in the original SmartWheeler project [13]. The original goal of the SmartWheeler project was the minimisation of the physical and cognitive load required in controlling it.
The IPW is composed of a commercially available Sunrise Quickie Freestyle, to which we have added front right, front-left and back light detection and ranging (LIDARs), wheel odometers, an emergency button, a touch-sensitive graphical display and an onboard computer. The LIDARs and odometers are used for navigation and obstacle avoidance. The onboard computer interfaces with the wheelchair's motor control board to provide autonomous navigational commands. All original hardware and electronic design were performed in-house by staff members and students at McGill's Center for Intelligent Machines (CIM) and École Polytechnique de Montréal. Further hardware maintenance and software development were done at Université de Sherbrooke's IntRoLab, notably on the interaction interface. The SmartWheeler used in this study uses the tactile/visual interface system as a communication way between the IPW and the user, and not only to provide visual feedback to the user regarding the state of the dialogue system as done previously [11,13]. Thereby, the touchscreen display and manual joystick (which comes with the commercial platform) are the main modes of communication with the user.
The touchscreen display is the new feature added as the result of the feedback provided by some users who found it difficult to hold the manual joystick in a stable position for a long time. The graphical interface proposed on the touchscreen display and reported on Figure 1 is designed using the Rviz (http://wiki.ros.org/rviz, accessed on 1 January 2021) graphical interface proposed by Robot Operation System (ROS) (https://ros.org, accessed on 1 January 2021), which is an open source set of libraries popular in robotics. The screen displays: • On the right side, a virtual IPW in its environment (described by a map) along with the obstacles (reported in blue color in Figure 2) as detected by the LIDAR sensors. The environment is constructed and updated using simultaneous localization and mapping (SLAM). • On the left side, the four interaction modes as described in the following.
The IPW can be controlled by the user with four interaction modes: • Manual mode, using the joystick which comes along with the commercially available wheelchair platform and three other modes which can be activated via the touch screen as reported on Figure 1. • Semi-autonomous mode, designed by our teams as a new feature added to the SmartWheeler, composed of: -A virtual joystick, as represented in Figure 2 upper left side, which allows the user to move the IPW in the desired direction by moving the black button in the circle and -An IPW-shaped-joystick, side, which is represented as an IPW shape shadow on the map, as represented on Figure 2 upper right, and allows the user to move the IPW in the desired direction by dragging the IPW's shadow on the map; The difference between the two semi-autonomous modes consists in their design. • Autonomous mode (represented on Figure 2) lower left and right, which consists, first, of indicating a desired destination on the map using a virtual representation of the IPW, and second, the wheelchair will autonomously generate the path to reach the desired destination.  In order to validate the development carried out on the IPW and to correct problems that could appear due to the placement of the LiDAR sensors (for the detection of obstacles), the sensitivity of the touchscreen or the development of the semi-autonomous and autonomous operating modes, this study carries out an experiment with novice users on our IPW, as reported in the following. The study was approved by the Centre for Interdisciplinary Research in Rehabilitation of Greater Montréal (CRIR) Research Ethics Committee.

Participants and Recruitment Procedure
Using a convenience sample, eleven IPW novice participants were recruited for this study, as following: an email invitation to participate was sent to the mailing list of students, enrolled in all cycles of study from undergraduate to postdoctoral, and staff (faculty, lecturers, and research engineers) at Université de Sherbrooke. The email summarised our study. Interested persons contacted our research team by email, and a meeting was scheduled based on each participant's availability. Moreover, the inclusion criteria were: (a) being a minimum of 18 years of age, (b) being a student or staff member of the Université de Sherbrooke (as access to the institute restrictions was restricted to faculty members during the COVID-19 lockdown) and (c) fluent in English or French, while the exclusion one was (a) to have a cognitive, visual or auditory deficit.
Each participant received, after the experiment, a compensatory allowance of CAD 20 for their participation.

Experimental Setup and Protocol
The purposes of using novice users in the proposed experimental are to: • Analyze, identify and/or evaluate the control graphical interface, reported on the touch screen, as represented on Figure 1, for improvements and complementary and/or alternative interaction modalities; • Identify the specifications retained in connection with the comfort improvement and preferably in the planning of the IPW driving.
To do so, participation in the experiment was meant to last approximately one and a half hours and could be held in one session or more depending on the availability of the novice participants.
The experimental room at the Interdisciplinary Institute for Technological Innovation (3IT) at Université de Sherbrooke used for the experiments in this study was mapped prior to welcoming the participants by a member of our team.
First, the researcher went through the consent form and if the participant agreed to sign it then the researcher presented the physical IPW and verbally described all its functionalities and operation modes. Secondly, the novice users were asked to complete the following tasks in the following order: (1). Familiarise with the IPW The objective was to become familiar with the IPW, meaning each participant manipulated all the options of the IPW while the team member was at their side to answer all their questions. For safety reasons, an emergency stop for the IPW was available on the IPW and a second one was available to the research member.
(2). Run a predefined path Participants were asked to use the IPW to run a predefined path which consisted in following the green lines attached to the floor of the experimental room used for the experiments. The only indication given by the researcher was to follow the path as many times as they want but the only constraint was to use, at their convenience, one after the other on the same round or on separate rounds, the four interaction modes present on the IPW: • manual mode via a physical joystick, • semi-autonomous mode represented by (a) a virtual joystick present on the graphical interface and (b) a virtual IPW-shape joystick, • and an autonomous mode, by indicating (dragging the virtual IPW present on the touch screen on the desired location).
(3). Filling out a questionnaire At the end of the experiment, the participants were asked to complete an online questionnaire via Lime Survey to assess their ability to use the IPW and comments regarding its functionalities during the experiment.

Data Collection and Analysis
In order to capture data from the involvement of novice participants, the following methods were used: • Qualitative method each participant was observed using the participant observation technique (https://research.utoronto.ca/participant-observation, accessed on 1 January 2021) by the researcher who conducted the experiment, in which the questions and discussions emerge during the involvement of the participant, during the familiarisation with the IPW by novice users before executing the predefined path (which consisted of following the green tracks on the floor of one of our experimental room at Université de Sherbrooke as represented on Figure 3). • Quantitative method after performing the experiment the participants filled (i) a Lime Survey (https://sondage.crir.ca/index.php?r=survey/index&sid=488847, accessed on 1 January 2021) online questionnaire on the IPW usability, consisting of the System Usability Scale (SUS) questionnaire and answered to (ii) four open-ended questions related to IPW functionality developed by our research team.
The questionnaire used for the quantitative method was available on a desktop PC installed in the same room where the tests were performed. The collected data were analysed as follows: • Qualitative analyses The qualitative analyses, which in our study was of the participant observation type, involved the observer asking questions and discussing with the participant the function of the participant's involvement. Moreover, this technique involves observing the participant's behaviour in the experimental environment. Thereby, the data collected by the observer include an unstructured interview with the participants, notes based on their observations and interactions on how the participants performed the (1) Familiarise with the IPW step: in what order they tested the modes and the IPW, what and how many questions they had during this step, how much time they needed to complete the three tasks and the ease or difficulty they had when familiarizing with the device. This information was written by the observer, reported to the other authors and analysed after a group decision. • Quantitative analyses The quantitative analyses is performed: on the SUS questionnaire, as described in the following, -on the answers provided by the participants on the four open-ended questions related to IPW functionality.
When the SUS is used, participants are asked to score the 10 items reported in Appendix A.1 with one of five responses that range from strongly agree to strongly disagree (Likert scale). It is a mixed-tone questionnaire in which the odd-numbered items have a positive tone and the even-numbered items have a negative tone.
The first step in scoring on the SUS is to determine each item's score contribution, which will range from 0 to 5. For positively-worded items (odd numbers), the score contribution is the scale position minus 1. For negatively-worded items (even numbers), the score contribution is 5 minus the scale position. To get the overall SUS score and obtain a range from 0 (very poor perceived usability) to 100 (excellent perceived usability) in 2.5-point increments as explained in [20], the sum of the item score contributions is multiplied by 2.5: X = Sum of the points for all odd-numbered questions − 5 Y = 25 − Sum of the points for all even-numbered questions SUS Score = (X + Y)·2.5 (1) In our study, Equation (1) is used to analyse the SUS for each of the involved participants, while the obtained score is analysed as previously described and also illustrated in Figure 4 according to Usability.gov (https://measuringu.com/interpret-sus-score/, accessed on 1 January 2021) which a SUS score above 68 is considered average, and anything lower than 68 is below average.

Participants
The participants (n = 11) involved in this study were called novices as they had never used a wheelchair before. Pseudonyms were used to protect participants' anonymity. The participants' age ranged from 22 to 38 (only one participant was female). All participants realised the three experimental steps, e.g., familiarisation with the IPW, running the predefined path and filling out the questionnaire, in less than 60 min (range: 45-60 min).

Findings
Some overarching themes resulted from our qualitative and quantitative data analysis, represented by (i) the participant observation method, (ii) the SUS questionnaire and the four open-ended questions related to the IPW functionality.

Qualitative Data Analyses: Participant Observation
During the participant observation stage, two categories of participants were observed during the familiarisation step: (a) 9 out of 11 participants tested each mode one by one in order to understand how to use them, (b) while the others 2 participants tested each of the control modes in a random way. Generally, all the participants were very autonomous in performing the indicated steps, with a desire of understanding the system by themselves without asking too many questions. Still, 8 out of 11 participants asked many pertinent questions related to the modules attached to the IPW, the limitations of the system or why the systems reacted in a certain way. For example, 2 out of 11 participants had problems during the trajectory, as they tried to push the system's limit, such as going too close to the obstacle or increasing the IPW's speed too much. Even so, they did not abandon the experiment and performed all the steps. They rather tried performing the scenarios involved in the experiment in order to understand if the problem came from the IPW or from their way of performing the trajectory. Still, one participant decided to not use the IPW-shape-joystick, as from his perspective it was not so easy to use. While ten out of eleven participants realised only three times the predefined trajectory, either using each control mode one after the other or mixing the control modes, only one participant performed the predefined trajectory more than three times, trying to use each mode as well as mixing the modes to follow the path.
Regarding the software used for the proposed graphical interface, ROS (https://www. ros.org/about-ros/, accessed on 1 January 2021 ), 2 out of 11 novice participants already made use of it previously in their career. For this reason, they really tried to put themselves in an IPW user's place. Moreover, at the same time they raised good questions and offered critical feedback on the proposed interface.
It was also interesting to see that even if 2 out of 11 participants were very tall and the touch screen could not be adjusted to a good position for them, they still went through the experiment steps and provided exploitable and good feedback. Table 1 shows the score contributions reported by each participant to the SUS questionnaire. The odds-numbered items, e.g., frequent usability (SUS1), perceived ease of use (SUS3), perceived system integration (SUS5), difficulty to learn (SUS7), confidence in use (SUS9) should have a positive tone, while the even-numbered ones, e.g., perceived complexity (SUS2), technical need (SUS4), perceived consistency (SUS6), perceived awkwardness/cumbersomeness/heaviness (SUS8), perceived learnability (SUS10) usually have a more negative tone. Each participant's choices in the SUS questionnaire are as follows:

Quantitative Data Analyses I: SUS Questionnaire
• P1 remains neutral to using frequently the system (SUS1), to the perceived ease of use (SUS3) and the need for technical support (SUS5), while answering in the expected way to other questions. • P2: indicates neutral response to perceiving the system as being complex (SUS2), to the perceived ease of use (SUS3) and to the perceived consistency (SUS7); while not feeling too confident in using it (SUS9), would not frequently use the system (SUS1) and needs to learn a lot of things before using it (SUS10); • P3: would not use frequently the system (SUS1) and remains neutral to perceiving the system as being complex (SUS2) and there being too much inconsistency with the system (SUS6); • P5: indicated that they would not use frequently the system (SUS1), found that the system would not be learned too quickly by others (SUS7), have not found confidence in the system (SUS9), perceived system integration (SUS4), though there was too much inconsistency with the system (SUS6), while being neutral to founding the system very cumbersome to use (SUS8) and the need for technical support (SUS5); • P7: would not use frequently the system (SUS1) and remained neutral regarding the confidence in the system (SUS9). • P10: indicated that the system does not have the features well integrated (SUS5) and does not feel too confident in using it (SUS9); while remaining neutral to perceiving the system as being complex (SUS2), to having too much inconsistency (SUS6) and found that the system would not be learned too quickly by others (SUS7). This participant is the only one who indicated that as a wheelchair user he would frequently use the SmartWheeler. • P11: indicated that he would not use frequently the system (SUS1), thought there was too much inconsistency with the system (SUS6) and does not feel to confident in using it (SUS9) • P4, P6, P8, P9: in all but one aspect, the frequency usability of the system (SUS1) to which P4 remained neutral, the others indicated a negative tone, and answered as expected. The SUS scores and grades reported in Table 2 and graphically represented on Figure 5, found that the participants can be classified in two (2) groups:

•
Group 1: 4 out of 11 participants scored the SUS below the average of 68, meaning they consider the IPW as a poor system or at the limit of being an ok one; • Group 2: 7 out of 11 participants scored the SUS above or equal to the average of 68, which indicates that they consider the IPW system as an ok, good and excellent system.
The reported results exhibited from our SUS questionnaire while analysing the data are as expected, meaning that the proposed IPW has an ok usability, a marginal acceptability and a passive net promotor score (NPS) and a 50% percentile rank (as Figure 4 suggests).  Table 2. Tabular representation of the SUS scores, for the odds items, even items and grades (as explained in Figure 4) obtained for each participant along the average one. The collected data of the four open-ended questions (reported in Appendix A.2) related to IPW functionality are analysed in this section. The results of the first three questions are grouped as Choice, advantages and disadvantages and question the choice of using a specific operation mode, the advantages and the disadvantages of such operation modes.

System Usability Scale (SUS) Score
For what reason(s) did you choose to activate/use one mode of operation rather than another at a specific point in the journey?
What advantage(s) do you find in each of the three modes?  The proposed questions were identified by the authors as an extension to the SUS questionnaire. As the SUS questionnaire gives answers to the overall "system" such as its performances and all its functionalities, these four questions allowed the participants to provide feedback regarding the interface in use and to propose improvements.

Choice, Advantages and Disadvantages
The data collected for this group were analysed using constructors presented within The Consolidated Framework for Implementation Research (CFIR) [21]. Each identified constructor was rated using the valence rating, which consists in identifying two factors: positive and negative influence on the system and interface. First the provided comments to each question were classified using the valence rating and reported in tables. Then, common comments are reported.

Manual, Semi-Autonomous and Autonomous Mode
The feedback represented by participants' comments is reported in the following. The collected comments are grouped in function of the variance rating positive, negative and neutral or no answer when there is no comment. The comments provided by the novice participants to the first group of questions regarding the choice, advantages and disadvantages of each control mode are reported in Tables 3-5 as follows: • Table 3 gathers comment for manual mode.
-(Choice of the mode) Two out of eleven participants reports negative feedback: Gives a false confidence linked to a habit and Acceleration was fast. These two comments are linked as the first one suggests that the use of a joystick can be intuitive but while activated the IPW can move very fast.

-
(Advantages) Only one participant out of eleven does not see an advantage in using the manual mode. -(Disadvantages) Two out of eleven participants do not see a disadvantage in using the manual mode.
• Table 4 gathers comment for the semi-autonomous mode.
-(Choice of the mode) Five out of eleven participants gave negative comments regarding the choice to use the IPW-shape-joystick, while nine out of eleven found the semi-autonomous mode intuitive, easy to use. -(Advantages) One participant indicates negative feedback, but the same participant also provides positive feedback as well. Moreover, P 3 was neutral by indicating I do not know, while P 4 does not provide an answer to this question.

-
(Disadvantages) Nine out of eleven indicate some disadvantages in using this mode but more precisely the IPW-shape-joystick one. P 9 prefers to not answer the question, while P 3 provides an advantage rather than a disadvantage It is advisable to use the virtual joystick.
• Table 5 gathers comment for the autonomous mode.
-(Choice of the mode) Five out of eleven participants report negative comments regarding their choice of using such mode. Still, all the comments are related to the choice our team made in leaving the IPW to choose the path to follow and not giving the participant the choice to select one. P 3 remains neutral, while P 6 preferred not to answer. -(Advantages) Only one participant, P 1 , did not report any advantage in using the autonomous mode. -(Disadvantages) P 9 preferred to not answer, while P 1 already reported its disadvantages in the previous questions. The rest of the participants found disadvantages related once again to the choice of our team to leave the IPW to choose the path to follow and the final orientation without giving the participant control over it.  The IPW-shape-joystick is very difficult to understand Less pleasant to use than a manual joystick We must always be aware of where we want to go Requires increased concentration on the screen It is difficult to predict the movement (IPW-shape-joystick) Speed control is more difficult The IPW-shape-joystick is more difficult to use No answer P 9

Proposed Improvements
The improvements provided by the participants are divided into two areas: (a) IPW software which will cover all feedback regarding the possible improvements to the interfaces/software available on the IPW and (b) IPW hardware where we considered all feedback regarding the improvements related to the system control and behaviour. The used rating is based on the strength (e.g., weak or/and strong) influence on implementation [21]. First, the improvements regarding the IPW software and IPW hardware are reported in the order collected from P 1 to P 11 , as follows:  Table 6 groups the common comments for both IPW software and hardware. Moreover, a strong or/and weak ranking is associated with each common comment. From the common comments regarding the IPW software it can be seen that (i) the autonomous mode was really appreciated, however, there is a requirement to involve the users more in the decision making and (ii) the graphical interface should be more user-friendly, which we already have. Both comments were expected to emerge due to the choices made by our team to use an open-source graphical interface library usually used by the robotics community and by not giving the choice of path to follow and orientation for the autonomous mode. The suggested improvements for the IPW hardware are appealing as the participants require improvements in order to increase the confidence they put in the wheelchair, such as Add vibrations feedback on the controls, Use of physical buttons for emergencies or Find ways to not look at the tactile screen. Table 6. Common areas of improvements along with the common improvement and the allocated ranking score. Overall, all the suggested improvements are consistent and are taken into account by our team.

Discussion
This qualitative and quantitative study is the first of its kind to explore the usability and functionality of an IPW, developed by our team, by novice participants without any experience in driving one. Previous studies done with the same IPW to explore the perceptions of IPW with end-users, via interviews before and after testing the IPW [6] or via interviews after illustrating video [5] with the IPW exhibited positive results and suggests that IPW would enhance social participation in a variety of important ways. This motivated us to continue developing the IPW until it can be capable of being fully usable by the intended users. Thereby, the feedback provided by novice participants who had a critical eye on the used technology helps us (a) adjust the physical IPW and the proposed graphical interface and (b) go further and perform tests in real environments with IPW users. The fact that novice users found the IPW acceptably usable with good functionalities suggests that the IPW can be adjusted by taking into account the feedback provided in this study, and could be used to improve user's mobility and lifestyle.
Still, along with positive feedback, the participants also raised technological and personal concerns, such as • the IPW's touch screen not being adaptable for tall participants; • the proposed graphical interface which proposes different control modes should be more user-friendly; • the autonomous navigation, even if it was very appreciated, should be improved; • the map used on the graphical interface should be more realistic; • the manual joystick, as well as other control modes of the IPW, should be thought for left and right-handed users.
Going further, the feedback and results collected through this study will allow us to improve the IPW before testing it with the users in real environments. It is also worth mentioning that the current word pandemic situation requires additional features to the IPW, such as (i) human-aware navigation, which makes the IPW capable of interacting with its environment (with the already existing autonomous mode) and taking decisions based on the human social interaction existing in it and can ensure a safe distance between the user and other humans; (ii) modules allowing the IPW to follow a person thus give the users the ability to "walk" alongside their caregivers, family or someone from the community while keeping a safe distance between them. These ideas already have received attention in the literature. For example, the socially acceptable navigation or human-aware navigation to socially interact with people [22] or to learned cost functions to be used for social navigation [23] are already studies which report positive outcomes prior to the current word pandemic. A review of sociological concepts of social robotics and human-aware navigation is proposed in [24]. We believe we can go further and use socially acceptable collision avoidance in an outdoor environment as the one proposed in [25] for a mobile robot, as the IPW can be considered a mobile robot and thus easily adopt such approaches. Additionally, ramp detection models can be implemented using deep learning [26], which could improve the mobility of the user, as well as modules which enable side-by-side following [27], which could make IPW users interact more with their caregivers instead of focusing on navigating the IPW. One can also find studies which propose low-cost and robust real-time eye-controlled wheelchair [28], user's voice [29], facial expression [30] and brain-controlled wheelchair to navigate in familiar environments with healthy [31] or patients [32] are proposed as solutions for people losing their sensing ability. Currently, our team is working on developing a low-cost BCI-based graphical interface in order to assist the control of the IPW, add human-aware navigation in the already existing navigation modes and use new technologies to allow the user to take a walk with a specific person indoors and outdoors.

Limitations
The reported study has some limitations. First of all, the use of novice participants can be considered a limitation of this study, as it is difficult to project in the daily needs required by an IPW end-user. Furthermore, even if the data analysis for the SUS questionnaire resulted in a positive outcome, it is worth underling the fact that the SUS questionnaire is intended for IPW users. Therefore, a bias can be exhibited in the first question, e.g., I think that I would like to use this feature frequently. However, each participant was instructed verbally by the observer and also indicated in writing, e.g., "To answer the questions, imagine yourself as a motorized wheelchair user", at the beginning of the questionnaire".
Secondly, even if done purposefully and knowingly, the use of a virtual map, generated by the Rviz graphical interface directly on the touchscreen did not give a good sense of spatial orientation to the users. For this reason, such input from the users was not a surprise. Moreover, future work will make use of a map which will better represent the environment in which the IPW navigates.
Moreover, the participants found that the autonomous mode, which gives the IPW the ability to navigate by itself from one destination to another, takes time to perform the trajectory and to stabilise. This is due to the fact that our team decided to use methods which allow the IPW to decide on the chosen path to the destination at a lower speed in order to avoid fearful reactions from the participant's side.
Finally, even if the novice users did not have experience driving an IPW prior to this experiment, the collected data exhibited that there were two participants, P 2 and P 5 , which associated the manoeuvre of the manual joystick with similar systems such as video games. Therefore, there may be a bias for these two participants, as they will prefer something which they already used. However, as regarding their feedback for the other modes, the authors did not find such a bias as their comments were coherent as reported on Table 4 for the semi-autonomous mode and Table 5 for the autonomous one.

Conclusions
In this study, eleven novice participants, with and without knowledge of the technology used to improve the IPW, agreed to participate in our qualitative and quantitative study. The improved IPW was first introduced within the SmartWheeler project.
The quantitative analyses suggest that the IPW is sufficiently usable and robust. Moreover, the qualitative analyses via the participant observation method allowed us to learn how to further form end-users at using the proposed control modes. Finally, the second quantitative analyses method helped us (i) uncover issues which we need to address before using it with end-users, for example distinguishing between left-and right-handed persons and (ii) understanding what are the first impressions on the proposed control and navigation modes.
Future work will involve testing improvements to our IPW prototype based on the feedback and insights from this study with the aim of enhancing mobility and quality of life of IPW end-users with and without loss of the sense of touch.  Informed Consent Statement: Informed consent was obtained from all subjects involved in the study.

Data Availability Statement: Not applicable.
Acknowledgments: The authors would like to thank Anne-Marie Hébert for her help in choosing the quantitative and qualitative questionnaires, as well as Réjean Prévost for creating the LimeSurvey questionnaire used during the experiments. The authors would also like to thank William Babin-Demers and Lucas Poquet for their contribution to the software of the platform. Finally, the authors would also like to thank all the subjects that accepted to be involved in the study.

Conflicts of Interest:
The authors declare no conflict of interest.

Appendix A. Used Questionnaires in the Experimental Setup and Protocol
Appendix A.1. System Usability Scale (SUS) Questionnaire SUS1 I think I will use this system frequently.
SUS2 I find this system unnecessarily complex.
SUS3 I think this system was easy to use. SUS4 I think I would need the help of a technician to be able to use this system. SUS5 I found that the different functions of this system have been well integrated.
SUS6 I think there are too many inconsistencies in this system. SUS7 I guess most people would be able to learn how to use this system very quickly.
SUS8 I found this system very cumbersome/awkward to use.
SUS9 I felt very confident using this system. SUS10 I need to learn a lot of things before I can use this system.
The answers were of Likert scale-type as follows: