To illustrate the advantages of the FULE methodology, we present a case study of the design process of a complementary autonomic breathing system by Hospitech®. The system aims to prevent ventilator-associated pneumonia (VAP) in respiratory patients. The design process was initiated even though the company already had a first functional prototype aiming to evaluate the reduction in VAP occurrences. Usability issues that were not considered during the first prototype development process resulted in operational challenges in hospitals, and nursing staff refused to handle the system due to its complexity. Hence, this project’s goal was to improve its usability, along with achieving acceptance and trust. We used FULE in the development of the second generation of the product in an effort to improve user interaction. After operation by the nursing staff at (Rambam Medical Center, Haifa, Israel) and following the FULE methodology, the product was evaluated for its usability, look-and-feel, and trust.
6.3. Case study—Usability
After laying the foundation and setting an initial framework considering all technical and functional requirements, it is time to proceed to the usability study. Our case study’s design process began while the company already had a first functional prototype, but usability issues were not considered during the development process. FULE was used to develop the second generation of the product to improve user interaction with it. We started this phase by examining the functional prototype in search of possible failures and risks. Failures were encoded on a scale between 1–4, where the lowest rank is 1 = user frustration, and the highest is 4 = physical or mental injury.
Among the findings were deficiencies in both the design of the system itself and of its disposable units. The buttons were all aligned, and were the same size, shape, and color, with no hierarchy, and without distinguishing between urgent and non-urgent actions. In addition, the connectors were placed without considering the wiring direction, linking the various lines, and tangling them.
In the next step, we conducted observations at Rambam Hospital, monitoring the system in action and at two other hospitals (Assaf Harofeh, Be’er Ya’akov, Israel and Rabin Medical Center, Petah Tikva, Israel) watching nurses’ regular work where there was no similar device. In addition, we interviewed the nurses and other medical staff to understand their needs, expectations, and requirements for the future product.
The next stage was creating usage scenarios for each possible human–technology interaction. Every scenario included all possible failures that might occur due to inappropriate usage. These usage scenarios indicated that the system’s dimensions and weight made its mobility in the hospital almost impossible, and it often was in the staff’s way. Nurses claimed it was often difficult to cope with the complex assembly of the system (before connecting it to a new patient), as it required gathering various components located in different rooms that were sometimes unavailable when needed. The assembling process itself appeared to be confusing for the staff, potentially causing user mistakes. It seems that at this stage, rather than making the task easier, the system was a burden for the staff.
Regarding the required information, all nurses agreed that the system should display only the most essential information (cuff pressure and critical alarms). Observations and interviews’ outcomes were used to help us design three different concepts of the new device. The top three suggestions for the system location were: (1) attached to patient’s bed, (2) connected to the back wall, and (3) using a designated pole. We used Harris’s profile [22
] to help us visualize each design concepts’ strengths and weaknesses according to our declared requirements (Figure 10
Using this four-scale scoring with a graphic representation helped us easily compare between our three options (derived from conducted interviews and observations) and we chose to design a designated pole that fit the design requirements in the best way.
6.7. Data Gathering
The evaluation phase was conducted at the ICU cardiac surgery units at Rambam Medical Center among trained nursing staff who were using the new AnapnoGuard system for six months and at (Kaplan Medical Center, Rehovot, Israel) where there is no similar autonomic system. Hence, the team there must hand operate the actions that the AG100s system performs automatically.
Phase one: exploratory interviews. Six semi-structured interviews were conducted. Participants included six volunteering nurses, three males and three females who had experienced the AG100s system. The interviews took place at Rambam Medical Center. The participants were asked to share their knowledge, opinions, and perceptions regarding ventilator-associated pneumonia (VAP), the AG100s system’s usability and appearance, and autonomic medical systems in general. All interviews were recorded and transcribed in full. The data were analyzed thematically.
Phase two: closed-ended questionnaires. This stage was conducted in two different groups: Group A: nursing staff who were using the AG100s system at Rambam Medical Center, and Group B: nursing staff who were not familiar with the AG100s system at Kaplan Medical Center. The second group had a short training process that was similar to the routine training procedure provided at the introduction of the AG100s system at Rambam the first time. Following training, participants were asked to complete the questionnaires based on their impressions. The questionnaire included closed-ended questions according to the following chapters: Chapter A—VAP, Chapter B—Technology, and Chapter C—The AG100s system.
The data collection process involved 14 questionnaires answered by medical staff who had experienced using the system at the Rambam Medical Center, two of which were eliminated because they were not fully completed. The remainder included 11 questionnaires filled out by the nursing staff and one completed by a physician. Thus, the results address only the 11 questionnaires completed by the nursing staff. Table 1
presents participant data from Rambam Hospital.
In addition, we gathered 22 questionnaires from members of staff who did not experience the system themselves but who underwent initial training at the Kaplan Medical Center. Because most of them chose to ignore the question of seniority, the analysis did not address this category. Table 2
presents participant data from Kaplan Medical Center.
Participants filled out two personal questionnaires regarding their attitude toward technology and their tendency to adopt new technologies. They were asked to rank their frequency of use of various technologies such as GPS and cloud backup on a scale of 0 = never to 5 = almost daily. The results suggest a slight gender difference, as women responded that they used a wider variety of technologies than men. The age factor did not impact the use of technology. In both hospitals, men attested to a slightly more positive attitude toward technology than women.
Evaluating the correlation between the use of technology and attitude toward technology did not reveal any significant results. Participants attesting to technological orientation did not necessarily employ new technologies, and participants using technologies were not necessarily technology proficient.
To evaluate the system’s aesthetic perception, the participants were asked to rank their impression of its visual appearance and graphical user interface (GUI). We used adjectives such as friendly, innovative, pleasing to the eye, etc. Women gave the system’s look-and-feel a higher score than men at both hospitals. Participants rated the visual appearance as pleasing to the eye (4.54 out of 5), medical (3.9 out of 5), and reliable (4 out of 5), with no significant differences between male and female responses except for device friendliness. Female nurses perceived the design of the system as friendlier (4 out of 5) than male nurses (3.16 out of 5) (Figure 13
To evaluate usability, we used a standard system usability scale (SUS) [40
] aiming to examine users’ impression of the comfort of use of the system in general, along with a designated questionnaire asking the respondents to describe the degree of comfort and clarity of use for each of the system’s operating phases. It was found that female nurses rated the comfort and clarity of use slightly higher than male nurses, and the scores given were mostly high (above four). Results also suggest that the older the respondent, the higher the ranking. However, the system’s SUS score was high on average (64.32 when the minimum required is 68). Grades ranged from 45 to 87.5 and indicate that there is a clear gender distinction. Male nurses gave higher grades in general (males: 70, females: 57.5). We also found an inverse ratio between the grades and age factor: the older the respondent, the lower the grade—Table 3
sums up the data from both questionnaires.
The last part relates to the issue of trust in the system. The questionnaire included 24 positive and negative statements regarding general confidence in the system and trust in specific actions. Participants were asked to rank their level of agreement for each one.
In order to calculate the degree of trust, we considered only questions addressing general trust in the system, excluding questions relating directly to specific operations or concerning trust in general. At the same time, we calculated the average grade participants gave specific operations. Table 4
sums up these data.
Women expressed a lower level of trust toward the AnapnoGuard system in every tool used for the study. In addition, women expressed a lower level of trust toward technologies in general, yet indicated they adopt and use new technologies more often than men (Rambam) or in an equal manner (Kaplan). Figure 14
illustrates the correlation between gender and trust.
Among male nurses, we found a definite link between the system’s SUS score and users’ trust; a positive correlation exists (R = 0.752) between the SUS score and the trust level (p
< 0.05), as shown in Figure 15
. These findings match previous studies’ conclusions that found a definite link between the two [16
The trust level expressed by men in both hospitals was almost identical. Experience with the system did not change their perception. On the other hand, among female nurses, we found differences between those who had experience with the system (Rambam) and those who only received the training (Kaplan). Kaplan’s nurses expressed a higher level of trust.
It seems that the factors affecting the women’s trust are different; some of these can be inferred from the interviews that we conducted, in which two main topics were raised. First is understanding the system mechanism. Participants were asked to explain how the AnapnoGuard system works. It seems that all the male nurses understood the system’s rationale; all of them were familiar with the CO2 sensor, as opposed to the female nurses who did not know how to explain what was happening. Perhaps this is the reason they found it difficult to trust the system. One of the men also pointed this out when he was asked if he trusted the system, saying: “At first, I was skeptical, but after understanding its rationale, I trust it.” The level of trust is possibly related to the level of understanding. Another possible explanation for the different results between males and females is their perception of the human factor. In our interviews, we asked the participants if they could imagine a future where autonomous systems would replace human nurses. While all said no, their reasoning was very different.
Women spoke in terms of compassion and contact: “The patient needs to have someone to hold his hand.” The word “communication” was repeated in all three interviews: “Without a doubt, patients need to communicate with a person.” Among male nurses, the reasons were related to the human brain: “Nothing can compare with the human mind in understanding the overall picture”; “There is a need for a person to integrate all the information.”
These results could help improving user trust by incorporating changes in the system itself, such as adding communication, written or verbal, or informing the user regarding its actions in real-time. Another adjustment could be to reach a higher level of transparency by explaining the reason for the system’s actions and revealing the next move. Second, user trust could also be improved by designing a slightly different training program to make sure all users fully comprehend the way the system operates.
The nature of our case study, however, is subject to several limitations. The research tools utilized to evaluate the system are subjective; therefore, respondents’ answers may be biased. In addition, due to staff turnover, we ended up with a relatively small sample size. This is a common situation due to the fact that we were evaluating a product for the first time with real users, employing a small group of participants.