Next Article in Journal
Effects of Incubation of Human Brain Microvascular Endothelial Cells and Astrocytes with Pyridostigmine Bromide, DEET, or Permethrin in the Absence or Presence of Metal Salts
Next Article in Special Issue
Study Protocol for Two-Steps Parallel Randomized Controlled Trial: Pre-Clinical Usability Tests for a New Double-Chamber Syringe
Previous Article in Journal
Personal and Lifestyle Determinants of HIV Transmission Risk in Spanish University Students
Previous Article in Special Issue
Facilitating Factors of Professional Health Practice Regarding Female Genital Mutilation: A Qualitative Study
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
IJERPH
Volume 17
Issue 22
10.3390/ijerph17228335
ijerph-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Usability Assessment of an Innovative Device in Infusion Therapy: A Mix-Method Approach Study

by
Pedro Parreira
1,
Liliana B. Sousa
1,*,
Inês A. Marques
2,3,4,
Paulo Santos-Costa
1,
Sara Cortez
5,
Filipa Carneiro
6,
Arménio Cruz
1 and
Anabela Salgueiro-Oliveira
1
1
Health Sciences Research Unit: Nursing (UICISA:E), Nursing School of Coimbra (ESEnfC), 3046-851 Coimbra, Portugal
2
Biophysics Institute, Faculty of Medicine, University of Coimbra, 3000-354 Coimbra, Portugal
3
Coimbra Institute for Clinical and Biomedical Research (iCBR), Area of Environment Genetics and Oncobiology (CIMAGO), Faculty of Medicine, University of Coimbra, 3000-354 Coimbra, Portugal
4
CNC.IBILI (Center for Neuroscience and Cell Biology and Institute for Biomedical Imaging and Life Sciences), University of Coimbra, 3000-354 Coimbra, Portugal
5
Muroplás-Plastic Engineering Industry, 4745-334 Muro, Portugal
6
PIEP—Innovation in Polymer Engineering, 4800-058 Braga, Portugal
*
Author to whom correspondence should be addressed.
Int. J. Environ. Res. Public Health 2020, 17(22), 8335; https://doi.org/10.3390/ijerph17228335
Submission received: 15 October 2020 / Revised: 2 November 2020 / Accepted: 10 November 2020 / Published: 11 November 2020
(This article belongs to the Collection A Focus on Healthcare from the Perspective of Gender, Culture, Management, and the Economy)
Browse Figure
Versions Notes

Abstract

Background: Flushing a venous access device is an important procedure to maintain their patency and prevent malfunctioning and complications. An innovative double-chamber syringe was developed, allowing for the assessment of catheter patency, drug delivery and final flush. This study aims to assess the usability of this new device, considering three development stages (concept, semi-functional prototype, functional prototype). Methods: An iterative methodology based on a mix-method design (qualitative and quantitative) enabled the assessment of the devices’ usability by their primary end-users. A usability questionnaire was developed and applied, along with focus groups and individual interviews to nurses. Results: The usability questionnaire integrated 42 items focused on four dimensions (usefulness; ease of use; ease of learning; satisfaction and intention to use). The initial psychometric findings indicate a good internal consistency and the conceptual relevance of the items. The scores seem to be sensitive to the usability evaluation of the medical devices in different stages of product development (with lower values on functional prototype evaluation), and related to nurses’ perceptions about functional and ergonomic characteristics. Conclusions: Quantitative and qualitative data provided a comprehensive overview of the double-chamber syringes’ usability from the nurses’ point of view, informing us of features that must be addressed.

1. Introduction

Peripheral intravenous catheters (PIVC) are the most used invasive devices in hospitals [1,2], with over a billion PIVC being inserted each year in hospitalized patients worldwide [3]. However, there is a wide range of complications associated with PIVC insertion [4,5,6], such as phlebitis [7,8,9], accidental removal, occlusion [10] and bloodstream infection [11,12,13]. Several PIVC complications can lead to their malfunctioning, followed by their premature removal and performance of subsequent puncture attempts until a new catheter is successfully placed [14]. To maintain catheter patency, international standards on infusion therapy recommend the recurrent performance of PIVC flushing, usually with sodium chloride 0.9%, before drug administration (to assess device patency), between multiple drug administrations (to prevent the occurrence of incompatibilities between substances), and after drug delivery (to clean and maintain catheter functioning) [15,16,17]. Currently, this whole process is performed using two or more syringes, which increases the time and costs of intravenous therapeutics, as well as PIVC manipulation.
To address this challenge, a new medical device was developed to allow intravenous drug administration and the flushing procedure, as recommended in international standards of infusion nursing [15,16,17]. With input from the nurse end-users that prepare and administer intravenous drugs regularly, a new double-chamber syringe was developed, with a single 20 mL barrel divided by a vertical plate into two chambers (with two plungers, presented in two independent channels to prevent the mixture of solutions). The two 10 mL chambers display plungers with different colors to facilitate the identification of the flushing and the drug chambers. Visually, this double-chamber syringe is very similar to the syringes traditionally found on the market, enhancing healthcare professional’s sense of familiarity with the device and their ease of use. The two chambers have been developed with equal volumes of 10 mL each to accomplish the general requirements of intravenous therapeutics and the international standards for flushing procedures, which recommend a minimum flush volume equal to twice the internal volume of the entire vascular access device system (catheter, extension set and/or needless injection system) [15]. Given that each independent chamber can be used without restrictions, this double-chamber syringe allows healthcare professionals to perform a catheter flush before and after drug administration (Figure 1), which is considered good clinical practice in infusion therapy [15,17,18].
According to specific Portuguese legal requirements [19], implemented by the Portuguese National Authority of Medicines and Health Products (INFARMED) under European directives [20,21], this double-chamber syringe was classified as a class I device (sterile, with a measurement function). This new medical device has several advantages. Along with the reduced number of syringes used for intravenous therapeutics, this syringe will reduce the time required for the whole process of preparing and delivering intravenous therapeutics, but also the number of catheter manipulations that can lead to several related complications such as vascular trauma, infiltration, occlusion, or mechanical phlebitis.
Engineering/ergonomics of human factors play an important role in the medical devices’ development process [22,23], during the phases of design, prototyping and manufacturing [24]. Considering the current European directives, a medical device refers to any apparatus, software, material, or other similar or related item intended to be used in the diagnosis, prevention, monitoring, treatment or alleviation of a disease or injury [20,21]. The search for medical solutions that improve patient well-being and reduce healthcare costs has been responsible for the fast-growing industry in these contexts [24], with associated challenges concerning medical devices’ assessment. There are specific requirements for the Health Technological Assessment (HTA) of medical devices when comparing them with other health-related technologies [25,26,27,28].
Usability methods must be implemented to define and design new products based on end-users’ needs and functional requirements [24], but also to validate potential prototypes in simulated settings or real-life environments to create the final device [29]. Usability is defined by the International Organization for Standardization (ISO) as “the extent to which a system, product or service can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use” (ISO 9241-11: 2018) [30]. Usability evaluation is intended to provide information about the medical devices’ effectiveness and efficiency, as well as end-users’ satisfaction, as required by the legal authorities [21], being highly useful during product development in the determination of significant outcomes concerning the interaction between the end-users and a given product [31].
The main purpose of usability testing is to refine and validate a medical devices’ proposed design [32], through an early prototype, working model, production-equivalent device, or marketable device [33]. Usually, these are pre-clinical validation studies, whose main purpose is to determine whether a given medical device will meet its intended users’ needs and preferences through the performance of representative tasks as a means to provide evidence of the strengths and opportunities for device improvement [33]. Usability tests can be used to determine design priorities, evaluate learning effects, assess legibility of written instructions or icon clarity, as well as explore other design options [33]. Through these tests, developers may improve the usability of a specific device, maximizing users’ satisfaction and safety during its use [29,34], also reducing device recalls and the need for ad hoc modifications [35]. Hence, it is important to identify and recruit representative users, whose selection criteria should be carefully well-defined to seek a secure and reliable data set [36]. The conceptual foundation of usability testing resides on the Human-Centered Design (HCD) model [29,37] since it provides information about potential design concerns, an important aspect for validation in performance requirements such as quality, efficiency or safety in healthcare systems [22,38]. Usability assessment involves several qualitative and quantitative methods [25,35,39] that should be implemented along with the technological and industrial development of the MD [33,39,40].
This study aims to assess the usability of an innovative double-chamber syringe, by their nurse end users, through a mix-method of quantitative and qualitative data, considering the development stages (concept, semi-functional and functional prototype). Due to the lack of standardized and validated usability measures for the assessment of medical devices’ usability, we also attempted to develop and provide a first exploratory validation of a usability questionnaire.

2. Materials and Methods

2.1. Design and Procedures

In the last decade, the HCD has been used in medical devices’ development [23,29,33] to comply with international standards in this field [ISO 14155:2011; ISO 14971:2012] [41,42]. According to this design, after the definition of initial specifications for the double-chamber syringe, based on the international recommendations and information provided by their nurse end-users, the technological development of this new medical device was also supported by an iterative methodology based on a mix-method design of qualitative and quantitative studies on the devices’ usability. This iterative methodology was implemented along with the initial stages of product development.
During the initial stages of concept and semi-functional prototype evaluation, focus groups with nurses were done to establish requirements and contexts of use, produce and assess design solutions, select the better design solution, and identify any modifications needed to improve the double-chamber syringe [18]. The participants also completed usability questionnaires focused on the design concept and a semi-functional prototype. Then, pre-clinical usability tests with the functional prototype were conducted through a two-arm parallel randomized controlled trial (RCT) in a simulated setting that enabled nurses to perform intravenous drug administration in a PIVC inserted on an upper arm simulator. The pre-clinical usability tests were performed in two phases: Phase (I): 10 nurses started the procedure using the double-chamber syringe and then the standard syringes, 10 nurses started the procedure using the standard syringes and then the double-chamber syringe; Phase (II): 10 nurses used the double-chamber syringe during the entire procedure; 10 used the standard syringes during the entire procedure. All the participants that used the double-chamber syringe during the usability tests were interviewed afterwards and completed a usability questionnaire.

2.2. Sample

According to the United States of America Food and Drug Administration (FDA), 15 participants per group or a minimum of 25 users should be enough to identify 90–97% of the usability problems [33]. Despite this, Borsci and colleagues [36] highlight that no definitive number of users equals reliable testing and consider that the number of users needed depends on the participants’ performance in identifying potential challenges/problems. Thus, concerning the usability tests with the double-chamber syringe, sample size was calculated according to the recommendations on ISO 62366-2 [43], which stipulates that a minimum sample of 10 users ensures the detection of potential challenges/problems and diminishes product returns. This is based on calculations determining the cumulative probability of detecting a usability problem: R = 1 − (1 − P) × n, where R = cumulative probability of detecting a usability problem, P = probability of a single test showing a usability problem, and n = number of participants. To comply with such standards, at least 10 nurses were involved in each stage of this study with the following inclusion criteria: have a bachelor degree (minimum academic title required); experience on intravenous drug administration; no previous contact with the newly developed double-chamber syringe (knowledge of the device or manipulation) and without any financial relationship with the device manufacturer and/or distributor. Thus, 16 nurses were involved during the concept stage, 22 nurses during the semi-functional prototype stage and 30 nurses during the testing of the final prototype (Table 1). To accomplish participants’ recruitment, the research team sent invitations to local tertiary hospitals in order to recruit nurses that meet the inclusion criteria previously presented.

2.3. Usability Questionnaire

Some usability questionnaires have been validated and developed for European Portuguese, such as the System Usability Scale (SUS) for products and users’ interfaces usability [44,45], as well as the Post-Study System Usability Questionnaire (PSSUQ) for system usability [46,47,48]. Moreover, a Usability Scale was developed based on the International Classification of Functioning, Disability and Health (ICF), the ICF-Usability Scale (ICF-US), specifically as part of a comprehensive framework for the design, development, and evaluation of ambient assisted living products and services for older adults [49,50]. However, in Portugal, there is a lack of standardized and validated usability measures specifically validated or developed for the assessment of medical devices’ usability in their development process. To address this gap, we attempted to develop and provide a first exploratory version of a usability questionnaire to be used in medical devices’ usability evaluation by their end-users in the several stages of product development. The usability questionnaire was developed through literature review and the revision of several assessment instruments for a selection of the initial pool of items for each dimension, namely: SUS [44,45]; PSSUQ [46,47,48]; Quebec User Evaluation of Satisfaction with Assistive Technology (QUEST 2.0) [51]; USE Questionnaire [52]; After Scenario Questionnaire (ASQ) [53,54]; Rating Scale Mental Effort (RSME) [55]. The Technology Acceptance Model (TAM) was used to establish a number of items concerning end-users’ technology acceptance and intention of use dimensions [56,57,58]. In the development of this questionnaire, the specificity of the medical devices regarding their efficacy, performance, and safety were considered as important features for a new medical device acquisition and use [59]. The expertise of academic, technological and industrial partners involved in medical devices’ development was also considered during the creation of an initial pool of items for the questionnaire to ensure its content validity.

2.4. Ethics

This study was reviewed and approved (Ref. P608-8/2019) by the Ethics Committee of the Health Sciences Research Unit: Nursing (UICISA: E) of the Nursing School of Coimbra (ESEnfC). Eligible nurses received all the information about the study and provided written informed consent and signed a non-disclosure agreement (NDA). All legal aspects regarding privacy and confidentiality were ensured with ID alphanumeric codes for the participants, which were used in all data collection instruments. Study participants’ ID codes and data were anonymized for analysis and publications.

2.5. Data Analysis

Demographics (gender, age), educational (academic degree) and professional (clinical experience, work setting) data from the participants were entered and analyzed using the Statistical Package for the Social Sciences–version 22.0 (SPSS 22.0, SPSS Inc., Chicago, IL, USA). For quantitative data, means, standard deviations, frequencies, and percentages were used as descriptive statistics. Cronbach alpha, item-total, and item-domain correlations were calculated. Given that the four domains of the usability questionnaire had a different number of items, the global score of each domain was divided by the number of items that composed it, ensuring an equally balanced scale. For qualitative data, after the transcription of the individual interviews and focus groups, the content analysis technique was used [60] to identify end-users’ perceptions regarding the double-chamber syringe’s usability.

3. Results

3.1. Usability Questionnaire Development and Pilot Study

A usability questionnaire was developed with 42 items in a 7-point Likert scale (from 1—totally disagree to 7—totally agree), within four distinct domains (i) usefulness; (ii) ease of use; (iii) ease of learning; (iv) satisfaction and intention to use (Table 2, Table 3, Table 4 and Table 5).
The internal consistency of the 42-items usability questionnaire was deemed excellent (α = 0.976), and the item-total correlation values are also good (all above 0.30), which indicates a good internal consistency of the questionnaire. The higher correlation coefficients (>0.80) were observed in the Satisfaction/intention to use domain (8 items), one in the Usefulness and one in the Ease of use domains. In the Ease of learning domain the item-total correlations were between 0.335 (p < 0.05) and 0.740 (p < 0.01). Considering the specific values in each dimension, the item-domain correlations were also deemed good (higher than 0.50). Several items presented a correlation coefficient higher than 0.90, four in the Satisfaction/intention to use domain and one in the Usefulness domain. The only item that increased the questionnaire’s internal consistency if deleted was number 28 (“there is no need for written instructions to use it”), although this was considered negligible.

3.2. Double-Chamber Syringe Usability

The adjusted global scores (total and domains) of the usability questionnaire are presented in Table 6, considering the specific stages of product development (concept evaluation, semi-functional prototype evaluation, and functional prototype evaluation). Throughout the different stages, usability scores were consistently higher than the mean ponderation value of the 7-point Likert scale, generally indicating a good usability evaluation of the double-chamber syringe.
Qualitative data collected during the focus groups (in concept and semi-functional prototype stages) and individual interviews (in functional prototypes stage) was also positive in regards to the double-chamber syringe’s usability. Nurses emphasized the usefulness of this innovative device, highlighting the potential of the syringe “to assess the patency” and “to prevent drug interaction or contamination”. Furthermore, nurses stated that the syringe “reduces the amount of material wasted” and “reduces the number of catheter manipulations”, while “improving safety for patients and professionals” and “also lowers the risk of infection”. Regarding its ease of use, after manipulating the semi-functional and functional prototypes, nurses indicated the syringe body dimensions are “as close as possible to the syringes on the market”, emphasizing their simplicity in preparation and administration phases (“easier”, “reduce errors in the chambers charging”). Nurses also considered that a different color for each drug chamber facilitates the syringe’s use.
However, during the functional prototype usability assessment, major concerns were also identified. Considering the total scores of the usability questionnaire, the mean values for the concept and semi-functional prototype stages were similar (M = 5.80 points), with a slightly lower value for the functional prototype (M = 5.28 points). The mean values of all usability domains are lower for the functional prototype when compared with the previous stages. This may be explained by major concerns highlighted by the participants when focused on the syringe’s ease of use, mainly in the drug preparation phase (“a bit more hand strength than usual is required”; “you feel slight pressure, making it harder to aspirate”; “having two plungers makes it harder to aspirate the solutions and remove aspirated air from the chamber”; “hand coordination is slightly harder with two plungers”). Despite the difficulties identified, most nurses recognized that “it is a matter of training and getting used to it”. In fact, regarding their ease of learning, the nurses confirmed that written instructions are valuable “to understand which chamber should be filled first” to avoid cross-chamber contamination.
During the different stages, the correlation coefficients between the usability scores and professional experience (measured in months) did not reach any statistical significance in all the three stages of product development (p > 0.05).

4. Discussion

The production of a clinically effective and safe medical device is not sufficient [39], and end-users involvement is important because it enables the early identification of their requirements, design needs, and recommendations [24]. In new devices development, usability assessment ensures that prototypes meet all the requirements of effectiveness, efficiency, safety and satisfaction [38,61] to move forward to clinical studies in real settings [62]. Thus, this study aimed to evaluate the usability of a new device (double-chamber syringe for infusion therapy through vascular access devices) considering the several stages of their development (concept, semi-functional prototype, and functional prototype). Nurses were considered the primary group of end-users that could provide significant inputs during the device’s development stages.
Usability questionnaires play an important role in usability testing, along with other quantitative and qualitative methods [32,35,39]. Although a few usability questionnaires have been developed [49,50] and validated [45,48] for European Portuguese, there is no specific instrument for the evaluation of medical devices’ usability. Facing this, a new usability questionnaire was developed, and a pilot validation study was conducted. The involvement of the relevant stakeholders (academic, technological and industrial partners) along with a comprehensive review on existing scales and measures, ensured the content validity of the questionnaire. The item-total and item-domain correlation coefficients are positive and statistically significant, demonstrating the conceptual relevance of the items. Despite the importance of the quantitative data on usability, qualitative methods also enable the acquisition of more detailed and comprehensive data on the device being developed. Conceptual design solutions are important for product development, but the manipulation of a prototype, even semi-functional, enables the evaluation of significant aspects of the device’s usability features. Functional prototypes allow the end-users to perform a more rigorous evaluation of the medical device, by considering the relevant characteristics of its applications in real-life settings (e.g., hospital ward), which is not possible with semi-functional prototypes. In this study, the usability scores of the functional prototype were lower than in previous stages focused on the double-chamber syringe’s concept and semi-functional prototype. The quantitative data collected converges with the qualitative results gathered during the focus groups and interviews with nurses. Participants that use the functional prototype of the double-chamber syringe in a simulated setting identified significant challenges in its use.
The incorporation of usability testing in medical devices’ development allows the early identification of end-users perceptions of a device’s effectiveness, ease of use, learning and training requirements [35]. This type of qualitative and quantitative data obtained during the product development cycle provides a common language for developers, end-users, and all stakeholders involved. Specifically, usability data will be used to ensure important outcomes for product development, as found in this study with a new medical device, or even highlight the need to redesign the device. A redesign can be defined as designing something again or in a different way, altering a product, but can also involve simple or subtle modifications of the product [59]. In this study, the quantitative and qualitative data collected provided a comprehensive overview of the double-chamber syringe from the nurses’ point of view, informing the research team of current challenges that must be addressed by the technological and industrial partners.
Regarding the usability questionnaire, despite the satisfactory statistical results obtained, future validation studies are warranted with larger and heterogeneous samples, focused on the usability of several medical devices in different stages of product development, with different classifications and clinical applications. Given its current format and scoring system, future validation studies should implement different psychometric approaches to the questionnaire’s item validation, such as the Item Response Theory, since 3-point rating scales are considered more appropriate in the measurement of mental load and mental effort [63,64]. Moreover, future studies should be conducted to determine the usability cut-off-points for medical devices.

5. Conclusions

The development of innovative medical devices plays an important role in disease management and health promotion, providing healthcare professionals with efficient resources that can be used in the provision of safer care. Despite this, there are several risks for medical devices’ manufacturers, which can be associated with a lack of involvement of end-users within the medical devices’ development process. Usability testing is currently a legal requirement for medical devices development before placing it on the European Union market. In this context, quantitative data from usability questionnaires and qualitative information from focus groups/interviews provide a quick and simple usability testing during the development process of new medical devices, ensuring important information for all stakeholders involved in those development processes (designers, engineers, manufacturers).

Author Contributions

Conceptualization, P.P. and L.B.S.; formal analysis, L.B.S.; writing—original draft preparation, L.B.S., I.A.M. and P.S.-C.; writing—review and editing, P.P., L.B.S., I.A.M., P.S.-C., S.C., F.C., A.C. and A.S.-O.; supervision, P.P., A.C. and A.S.-O. All authors have read and agreed to the published version of the manuscript.

Funding

This research and APC was funded by the RESEARCH & TECHNOLOGICAL DEVELOPMENT INCENTIVE SYSTEM (SI I&DT Co-Promotion), grant reference: POCI-01-0247-FEDER-017604.

Acknowledgments

The authors would like to thank the Health Sciences Research Unit: Nursing (UICISA: E) from Nursing School of Coimbra (ESEnfC) for all the support. The authors P.S.-C. (SFRH/BD/136487/2018) and I.A.M. (SFRH/BD/136973/2018) thank the Portuguese National Funding Agency for Science, Research and Technology (FCT) for the financial support granted through the 2018 Ph.D. Scholarship program.

Conflicts of Interest

The authors declare no conflict of interest. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results.

References

  1. Carr, P.J.; Higgins, N.S.; Cooke, M.L.; Mihala, G.; Rickard, C.M. Vascular access specialist teams for device insertion and prevention of failure. Cochrane Database Syst. Rev. 2018, 3, CD011429. [Google Scholar] [CrossRef] [PubMed]
  2. Kaur, P.; Rickard, C.; Rickard, C.; Domer, G.S.; Kaur, P. Dangers of peripheral intravenous catheterization: The forgotten tourniquet and other patient safety considerations. Intechopen 2019. [Google Scholar] [CrossRef]
  3. Alexandrou, E.; Ray-Barruel, G.; Carr, P.J.; Frost, S.; Inwood, S.; Higgins, N.; Lin, F.; Alberto, L.; Mermel, L.; Rickard, C.M. International prevalence of the use of peripheral intravenous catheters. J. Hosp. Med. 2015, 10, 530–533. [Google Scholar] [CrossRef] [PubMed]
  4. Mattox, E.A. Complications of peripheral venous access devices: Prevention, detection, and recovery strategies. Crit. Care Nurse 2017, 37, e1–e14. [Google Scholar] [CrossRef]
  5. Miliani, K.; Taravella, R.; Thillard, D.; Chauvin, V.; Martin, E.; Edouard, S.; Astagneau, P.; CATHEVAL Study Group. Peripheral venous catheter-related adverse events: Evaluation from a multicentre epidemiological study in France (the CATHEVAL project). PLoS ONE 2017, 12, e0168637. [Google Scholar] [CrossRef]
  6. Abolfotouh, M.A.; Salam, M.; Bani-Mustafa, A.; White, D.; Balkhy, H.H. Prospective study of incidence and predictors of peripheral intravenous catheter-induced complications. Ther. Clin. Risk Manag. 2014, 10, 993–1001. [Google Scholar] [CrossRef]
  7. Cicolini, G.; Manzoli, L.; Simonetti, V.; Flacco, M.E.; Comparcini, D.; Capasso, L.; Di Baldassarre, A.; Eltaji Elfarouki, G. Phlebitis risk varies by peripheral venous catheter site and increases after 96 hours: A large multi-centre prospective study. J. Adv. Nurs. 2014, 70, 2539–2549. [Google Scholar] [CrossRef]
  8. Dunda, S.E.; Demir, E.; Mefful, O.J.; Grieb, G.; Bozkurt, A.; Pallua, N. Management, clinical outcomes, and complications of acute cannula-related peripheral vein phlebitis of the upper extremity: A retrospective study. Phlebology 2015, 30, 381–388. [Google Scholar] [CrossRef]
  9. Oliveira, A.S.S.; Parreira, P.M.S.D. Intervenções de enfermagem e flebites decorrentes de cateteres venosos periféricos. Revisão sistemática da literatura. Rev. Enferm. Ref. 2010, serIII, 137–147. [Google Scholar] [CrossRef]
  10. Danski, R.; Tannia, M.; Johann, A.; Vayego, A.; Paulo, S. Artigo Original Complicações relacionadas ao uso do cateter venoso periférico: Ensaio clínico randomizado. Acta Paul Enferm. 2016, 29, 84–92. [Google Scholar] [CrossRef]
  11. Arvaniti, K.; Lathyris, D.; Blot, S.; Apostolidou-Kiouti, F.; Koulenti, D.; Haidich, A.B. Cumulative Evidence of Randomized Controlled and Observational Studies on Catheter-Related Infection Risk of Central Venous Catheter Insertion Site in ICU Patients. Crit. Care Med. 2017, 45, e437–e448. [Google Scholar] [CrossRef] [PubMed]
  12. Beloin, C.; Fernandez-Hidalgo, N.; Lebeaux, D. Understanding biofilm formation in intravascular device-related infections. Intensive Care Med. 2017, 43, 443–446. [Google Scholar] [CrossRef] [PubMed]
  13. Guembe, M.; Pérez-Granda, M.J.; Capdevila, J.A.; Barberán, J.; Pinilla, B.; Martín-Rabadán, P.; Bouza, E.; NUVE Study Group. Nationwide study on peripheral-venous-catheter-associated-bloodstream infections in internal medicine departments. J. Hosp. Infect. 2017, 97, 260–266. [Google Scholar] [CrossRef]
  14. Keogh, S.; Flynn, J.; Marsh, N.; Higgins, N.; Davies, K.; Rickard, C.M. Nursing and midwifery practice for maintenance of vascular access device patency. A cross-sectional survey. Int. J. Nurs. Stud. 2015, 52, 1678–1685. [Google Scholar] [CrossRef] [PubMed]
  15. Gorski, L.A.; Hadaway, L.; Hagle, M.; McGoldrick, M.; Orr, M.; Doellman, D. The 2016 Infusion Therapy Standards of Practice; Wolters Kluwer: South Holland, The Netherlands, 2016. [Google Scholar]
  16. Queensland Government, Department of Health. Guidelines: Peripheral Intravenous Catheter (PIVC); Queensland Government: Queensland, Australia, 2018; pp. 1–28. [Google Scholar]
  17. Royal College of Nursing (Andrea Denton). Standards for Infusion Therapy; R Coll Nurs; Royal College of London: London, UK, 2016; pp. 41–42. [Google Scholar]
  18. Parreira, P.; Sousa, L.B.; Marques, I.A.; Costa, P.; Cortez, S.; Carneiro, F.; Cruz, A.; Salgueiro-Oliveira, A. Development of an innovative double-chamber syringe for intravenous therapeutics and flushing: Nurses’ involvement through a human-centred approach. PLoS ONE 2020, 15, e0235087. [Google Scholar] [CrossRef]
  19. Ministério da Saúde. Decreto-Lei n.o 145/2009 de 17 de Junho (Regras a que Devem Obedecer a Investigação, o Fabrico, a Comercialização, a Entrada em Serviço, a Vigilância e a Publicidade dos Dispositivos Médicos e Respectivos Acessórios); Ministério da Saúde: Lisbon, Portugal, 2009. [Google Scholar]
  20. European Council’s Recent Revision in MedDev 2. 7.1 Guidelines (Revision 4)—Quantifying its Change and its Impact. J. Bioeng. Biomed. Sci. 2017, 7, 1–7. [Google Scholar]
  21. Legislation117. Regulation (EU) 2017/745 of the European Parliament and of the Council of 5 April 2017 on medical devices, amending Directive 2001/83/EC, Regulation (EC) No 178/2002 and Regulation (EC) No 1223/2009 and repealing Council Directives 90/385/EEC and 93/42/EE. Off. J. Eur. Union 2017, 60, 2–175.
  22. Privitera, M.B.; Evans, M.; Southee, D. Human factors in the design of medical devices – Approaches to meeting international standards in the European Union and USA. Appl. Ergon. 2017, 59, 251–263. [Google Scholar] [CrossRef]
  23. Vincent, C.J.; Li, Y.; Blandford, A. Integration of human factors and ergonomics during medical device design and development: It’s all about communication. Appl. Ergon. 2014, 45, 413–419. [Google Scholar] [CrossRef]
  24. Ciurana, J. Designing, prototyping and manufacturing medical devices: An overview. Int. J. Comput. Integr. Manuf. 2014, 27, 901–918. [Google Scholar] [CrossRef]
  25. Tarricone, R.; Torbica, A.; Drummond, M. Challenges in the Assessment of Medical Devices: The MedtecHTA Project. Health Econ. UK 2017, 26, 5–12. [Google Scholar] [CrossRef] [PubMed]
  26. Ciani, O.; Wilcher, B.; van Giessen, A.; Taylor, R.S. Linking the Regulatory and Reimbursement Processes for Medical Devices: The Need for Integrated Assessments. Health Econ. 2017, 26, 13–29. [Google Scholar] [CrossRef] [PubMed]
  27. Fuchs, S.; Olberg, B.; Panteli, D.; Perleth, M.; Busse, R. HTA of medical devices: Challenges and ideas for the future from a European perspective. Health Policy 2017, 121, 215–229. [Google Scholar] [CrossRef] [PubMed]
  28. EUnetHTA Joint Action 2, Work Package 8. HTA Core Model ® version 3.0 (Pdf). EUnetHTA JA; National Institute for Health and Welfare: Helsinki, Finland, 2016; Volume 2, pp. 1–410.
  29. Harte, R.; Glynn, L.; Rodríguez-Molinero, A.; Baker, P.M.; Scharf, T.; Quinlan, L.R.; ÓLaighin, G. A Human-Centered Design Methodology to Enhance the Usability, Human Factors, and User Experience of Connected Health Systems: A Three-Phase Methodology. JMIR Hum. Factors 2017, 4, e8. [Google Scholar] [CrossRef]
  30. ISO 9241-11: 2018. Ergonomics of Human-System Interaction—Part 11: Usability: Definitions and Concepts; International Organization for Standardization: Geneva, Switzerland, 2018. [Google Scholar]
  31. Kortum, P.T.; Bangor, A. Usability Ratings for Everyday Products Measured With the System Usability Scale. Int. J. Hum. Comput. Interact. 2013, 29, 67–76. [Google Scholar] [CrossRef]
  32. Johnson, C.M.; Johnson, T.R.; Zhang, J. A user-centered framework for redesigning health care interfaces. J. Biomed. Inform. 2005, 38, 75–87. [Google Scholar] [CrossRef]
  33. Wiklund, M.; Kendler, J.; Strochlic, A. Usability Testing of Medical Devices, 2nd ed.; CRC Press: Boca Raton, FL, USA, 2016; ISBN 978-1-4665-9588-0. [Google Scholar]
  34. Money, A.G.; Barnett, J.; Kuljis, J.; Craven, M.P.; Martin, J.L.; Young, T. The role of the user within the medical device design and development process: Medical device manufacturers’ perspectives. BMC Med. Inform. Decis. Mak. 2011, 11, 15. [Google Scholar] [CrossRef]
  35. Martin, J.L.; Murphy, E.; Crowe, J.A.; Norris, B.J. Capturing user requirements in medical device development: The role of ergonomics. Physiol. Meas. 2006, 27, R49–R62. [Google Scholar] [CrossRef]
  36. Borsci, S.; Macredie, R.D.; Martin, J.L.; Young, T. How many testers are needed to assure the usability of medical devices? Expert Rev. Med. Devices 2014, 11, 513–525. [Google Scholar] [CrossRef]
  37. Bligård, L.O.; Strömberg, H.; Karlsson, M.A. Developers as Users: Exploring the Experiences of Using a New Theoretical Method for Usability Assessment. Adv. Hum. Comput. Interact. 2017. [Google Scholar] [CrossRef]
  38. Schmettow, M.; Schnittker, R.; Schraagen, J.M. An extended protocol for usability validation of medical devices: Research design and reference model. J. Biomed. Inform. 2017, 69, 99–114. [Google Scholar] [CrossRef] [PubMed]
  39. Martin, J.L.; Norris, B.J.; Murphy, E.; Crowe, J.A. Medical device development: The challenge for ergonomics. Appl. Ergon. 2008, 39, 271–283. [Google Scholar] [CrossRef] [PubMed]
  40. Martin, J.L.; Barnett, J. Integrating the results of user research into medical device development: Insights from a case study. BMC Med. Inform. Decis. Mak. 2012, 12, 74. [Google Scholar] [CrossRef] [PubMed]
  41. ISO 14155. Clinical Investigation of Medical Devices for Human Subjects—Good Clinical Practice; International Standard Organization: Geneva, Switzerland, 2011. [Google Scholar]
  42. ISO 14971. Medical Devices—Application of Risk Management to Medical Devices; International Standard Organization: Geneva, Switzerland, 2012. [Google Scholar]
  43. ISO 62366-2. Medical Devices—Part 2: Guidance on the Application of Usability Engineering to Medical Devices; International Standard Organization: Geneva, Switzerland, 2016. [Google Scholar]
  44. Bangor, A.; Kortum, P.T.; Miller, J.T. An empirical evaluation of the system usability scale. Int. J. Hum. Comput. Interact. 2008, 24, 574–594. [Google Scholar] [CrossRef]
  45. Martins, A.I.; Rosa, A.F.; Queirós, A.; Silva, A.; Rocha, N.P. European Portuguese validation of the System Usability Scale (SUS). Procedia Comput. Sci. 2015, 67, 293–300. [Google Scholar] [CrossRef]
  46. Lewis, J.R. Psychometric Evaluation of the Post-Study System Usability Questionaire: The PSSUQ. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting, Atlanta, GA, USA, 12–16 October 1992; pp. 1259–1263. [Google Scholar]
  47. Lewis, J.R. Psychometric Evaluation of the PSSUQ Using Data from Five Years of Usability Studies. Int. J. Hum. Comput. Interact. 2002, 14, 463–488. [Google Scholar] [CrossRef]
  48. Rosa, A.F.; Martins, A.I.; Costa, V.; Queirós, A.; Silva, A.; Rocha, N.P. Validação para português europeu do Post-Study System Usability Questionnaire (PPSUQ). In Proceedings of the 10th Iberian Conference on Information Systems and Technologies (CISTI), Aveiro, Portugal, 17–20 June 2015. [Google Scholar]
  49. Martins, A.I.; Rosa, A.F.; Queirós, A.; Silva, A.; Rocha, N.P. Definition and validation of the ICF-Usability Scale. Procedia Comput. Sci. 2015, 67, 132–139. [Google Scholar] [CrossRef]
  50. Martins, A.I.; Queirós, A.; Rocha, N.P. Validation of a usability assessment instrument according to the evaluators’ perspetive about users’ performance. Univers. Access Inf. Soc. 2020, 19, 515–525. [Google Scholar] [CrossRef]
  51. Demers, L.; Weiss-Lambrou, R.; Ska, B. The Quebec User Evaluation of Satisfaction with Assistive Technology (QUEST 2.0): An overview and recent progress. Technol. Disabil. 2019, 14, 101–105. [Google Scholar] [CrossRef]
  52. Lund, A.M. Measuring usability with the USE questionnaire. Usability Interface 2001, 8, 3–6. [Google Scholar]
  53. Lewis, J.R. Psychometric evaluation of an after-scenario questionnaire for computer usability studies: The ASQ. ACM SIGCHI Bulletin. 1991, 23, 78–81. [Google Scholar] [CrossRef]
  54. Lewis, J.R. Psychometric evaluation of an after-scenario questionnaire for computer usability studies. ACM SIGCHI Bull. 2007, 23, 78–81. [Google Scholar]
  55. Ghanbary Sartang, A.; Ashnagar, M.; Habibi, E.; Sadeghi, S. Evaluation of Rating Scale Mental Effort (RSME) effectiveness for mental workload assessment in nurses. J. Occup. Health Epidemiol. 2016, 5, 211–217. [Google Scholar] [CrossRef]
  56. Davis, F.D. Perceived Usefulness, Perceived Ease Of Use, And User Acceptance. MIS Q. 1989, 13, 319–339. [Google Scholar] [CrossRef]
  57. Davis, F.D.; Bagozzi, R.P.; Warshaw, P.R. User Acceptance of Computer Technology: A Comparison of Two Theoretical Models. Manag. Sci. 1989, 35. [Google Scholar] [CrossRef]
  58. Venkatesh, V. Determinants of perceived ease of use: Integrating control, intrinsic motivation, and emotion into the technology acceptance model. Inf. Syst. Res. 2000, 11, 342–365. [Google Scholar] [CrossRef]
  59. Santos, I.C.T.; Tavares, J.M.R.S. Additional peculiarities of medical devices that should be considered in their development process. Expert Rev. Med. Devices 2013, 10, 411–420. [Google Scholar] [CrossRef]
  60. Bardin, L. L’analyse de Contenu; Quadrige/PUF: Paris, France, 2007. [Google Scholar]
  61. Yen, P.Y.; Walker, D.M.; Smith, J.M.G.; Zhou, M.P.; Menser, T.L.; McAlearney, A.S. Usability evaluation of a commercial inpatient portal. Int. J. Med. Inf. 2018, 110, 10–18. [Google Scholar] [CrossRef]
  62. Parreira, P.; Sousa, L.B.; Marques, I.A.; Costa, P.; Braga, L.; Cruz, A.; Salgueiro-Oliveira, A. Double-chamber syringe versus classic syringes for peripheral intravenous drug administration and catheter flushing: A study protocol for a randomised controlled trial. Trials 2020, 21, 78. [Google Scholar] [CrossRef]
  63. Krell, M. Evaluating an instrument to measure mental load and mental effort using Item Response Theory. Sci. Educ. Rev. Lett. 2015, 3, 1–6. [Google Scholar]
  64. Krell, M. Evaluating an instrument to measure mental load and mental effort considering different sources of validity evidence. Cogent Educ. 2017, 4, 1–10. [Google Scholar] [CrossRef]
Ijerph 17 08335 g001
Figure 1. Double-chamber syringe (patent submission number 20192000899107 to Portuguese Authority in Industrial Property; international patent application number PCT/IB2020/056219 under the Patent Cooperation Treaty).
Figure 1. Double-chamber syringe (patent submission number 20192000899107 to Portuguese Authority in Industrial Property; international patent application number PCT/IB2020/056219 under the Patent Cooperation Treaty).
Ijerph 17 08335 g001
Table 1. Focus Groups: Users’ characterization.
Table 1. Focus Groups: Users’ characterization.
Concept Stage
(n = 16)		Semi-Functional Prototype Stage
(n = 22)		Functional Prototype Stage
(n = 30)
Sex n (%)
 Male5 (31.3%)7 (31.8%)8 (26.7%)
 Female11 (68.7%)15 (68.2%)22 (73.3%)
Age (years)39.25 ± 10.09637.86 ± 9.08336.57 ± 8.012
 M ± SD (Min.–Max.)25–5525–5526–55
Education n (%)
 Bachelors’ degree3 (18.7%)3 (13.6%)17 (56.7%)
 Post-graduate/Specialty4 (25.0%)6 (27.3%)4 (13.3%)
 Master’s degree9 (56.3%)12 (54.5%)7 (23.3%)
 Ph.D.-1 (4.5%)2 (6.7%)
Professional time (months) 195.56 ± 120.434185.32 ± 106.512 163.87 ± 97.604
 M ± SD (Min.–Max.)25–5536–37224–384
Department n (%)
 Operating room5 (31.3%)6 (27.3%)2 (6.7%)
 General hospital1 (6.2%)1 (4.5%)5 (16.7%)
 Internal medicine-3 (13.6%)3 (10.0%)
 Research unit3 (18.8%)3 (13.6%)2 (6.7%)
 Cancer unit1 (6.2%)2 (9.1%)2 (6.7%)
 Gastroenterology--2 (6.7%)
 Emergency Room--3 (10.0%)
 Orthopaedics2 (12.5%)2 (9.1%)1 (3.3%)
 Rheumatology--2 (6.7%)
 Intensive care unit2 (12.5%)2 (9.1%)-
 Physical medicine/Rehabilitation--2 (6.7%)
 Pneumology1 (6.2%)1 (4.5%)-
 Haematology--1 (3.3%)
 Pain consultation-1 (4.5%)-
 Psychiatric ward--1 (3.3%)
 Burn unit1 (6.2%)1 (4.5%)-
 Urology--1 (3.3%)
 Continued care unit--1 (3.3%)
 Unemployed--1 (3.3%)
Type of healthcare institution n (%)
 Public institutions13 (81.3%)19 (86.4%)25 (83.3%)
 Private institutions--2 (6.7%)
 Other (Teaching/Research)3 (18.7%)3 (13.6%)3 (10.0%)
Time at the current professional unit (months)135.25 ± 124.509124.55 ± 110.403 185.32 ± 106.512
 M ± SD (Min.–Max.)6–3726–3721–384
M—Mean; SD—Standard Deviation; Min.—Minimum; Max.—Maximum.
Table 2. Usefulness items (The use of a double-chamber syringe for intravenous therapeutics).
Table 2. Usefulness items (The use of a double-chamber syringe for intravenous therapeutics).
ItemsItem-Total Correlationα (If Item Excluded)Item-Domain Correlation
1. is useful for my work.0.645 **0.9750.771 **
2. facilitates the performance of my tasks.0.751 **0.9750.890 **
3. helps me to be more effective.0.754 **0.9750.901 **
4. helps me to be more efficient.0.781 **0.9750.897 **
5. achieves everything I would expect it to do.0.763 **0.9750.803 **
6. allows me to complete my tasks.0.728 **0.9750.767 **
7. allows me to complete my tasks easily.0.763 **0.9750.814 **
8. allows me to complete my tasks quickly.0.717 **0.9750.797 **
9. allows me to have better control over my tasks.0.725 **0.9750.835 **
10. helps me to be more productive.0.777 **0.9750.844 **
11. allows me to provide safer care.0.661 **0.9750.751 **
12. answers my needs.0.837 **0.9750.860 **
** p < 0.01.
Table 3. Ease of use items (The double-chamber syringe for intravenous therapeutics).
Table 3. Ease of use items (The double-chamber syringe for intravenous therapeutics).
ItemsItem-Total Correlationα (If Item Excluded)Item-Domain Correlation
13. is easy to use.0.795 **0.9750.861 **
14. is simple to use.0.815 **0.9750.870 **
15. is user-friendly.0.721 **0.9750.802 **
16. requires few steps to accomplish my work.0.670 **0.9750.815 **
17. is flexible to use according to my needs.0.701 **0.9750.663 **
18. does not require physical effort to use it.0.761 **0.9750.837 **
19. does not require mental effort to use it.0.509 **0.9750.653 **
20. allows me to complete tasks in a logical sequence.0.573 **0.9750.702 **
21. is not associated with significant possibilities of error in its use.0.651 **0.9750.717 **
22. allows me to recover from mistakes quickly and easily.0.651 **0.9750.642 **
** p < 0.01.
Table 4. Ease of learning items (Concerning the double-chamber syringe for intravenous therapeutics).
Table 4. Ease of learning items (Concerning the double-chamber syringe for intravenous therapeutics).
ItemsItem-Total Correlationα (If Item Excluded)Item-Domain Correlation
23. I learned to use it quickly.0.633 **0.9750.888 **
24. I learned to use it easily.0.740 **0.9750.853 **
25. I easily remember how to use it.0.611 **0.9750.867 **
26. I quickly became skilful with it.0.651 **0.9750.863 **
27. it is not necessary too much previous knowledge to use it.0.490 **0.9760.766 **
28. there is no need for written instructions to use it.0.335 *0.9770.705 **
** p < 0.01; * p < 0.05.
Table 5. Satisfaction/Intention to use items (Concerning the possibility of using the double-chamber syringe for intravenous therapeutics in the future).
Table 5. Satisfaction/Intention to use items (Concerning the possibility of using the double-chamber syringe for intravenous therapeutics in the future).
ItemsItem-Total Correlationα (If Item Excluded)Item-Domain Correlation
29. I will be satisfied with it.0.898 **0.9740.917 **
30. I would recommend it to colleagues.0.884 **0.9740.914 **
31. it will allow the performance of my tasks.0.875 **0.9740.902 **
32. it will be interesting for the performance of my tasks.0.867 **0.9740.915 **
33. I feel I need to have it in my work.0.608 **0.9740.656 **
34. it will be pleasant to use.0.810 **0.9750.823 **
35. I will feel comfortable in using it.0.851 **0.9750.861 **
36. I will feel confident in using it.0.821 **0.9750.863 **
37. I will feel secure in using it.0.788 **0.9750.835 **
38. the dimensions of the device are adjusted.0.804 **0.9750.814 **
39. the weight of the device is adjusted.0.479 **0.9760.577 **
40. the appearance of the device is adjusted.0.531 **0.9760.560 **
41. I will like to use it frequently.0.798 **0.9750.893 **
42. it will be easy to adjust it during the performance of my work.0.725 **0.9750.802 **
** p < 0.01.
Table 6. Evaluation of the MD: Quantitative scores.
Table 6. Evaluation of the MD: Quantitative scores.
Usability DimensionsConcept
(n = 16)		rSemi-Functional Prototype
(n = 22)		rFunctional Prototype
(n = 30)		r
Global scoreM ± SD5.80 ± 0.534−0.1745.80 ± 0.786−0.2505.28 ± 0.9440.273
Min.–Max.4.81–6.76 3.93–7.00 3.19–7.00
UsefulnessM ± SD6.11 ± 0.5930.0625.95 ± 0.875−0.2835.36 ± 1.1600.309
Min.–Max.4.75–7.00 3.58–7.00 2.00–7.00
Ease of useM ± SD5.69 ± 0.5970.2675.72 ± 0.9280.0825.06 ± 1.0130.183
Min.–Max.4.30–6.40 3.20–7.00 3.20–7.00
Ease of learningM ± SD6.05 ± 0.767−0.3065.96 ± 0.9160.1115.53 ± 1.0220.130
Min.–Max.4.33–7.00 4.00–7.00 2.83–7.00
Satisfaction/Use intentionM ± SD5.93 ± 0.860−0.2015.83 ± 0.826−0.0705.37 ± 1.0820.274
Min.–Max.4.71–7.00 3.57–7.00 3.07–7.00
M—Mean; SD—Standard Deviation; Min.—Minimum; Max.—Maximum; r—correlation coefficients with professional experience (months).
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Parreira, P.; Sousa, L.B.; Marques, I.A.; Santos-Costa, P.; Cortez, S.; Carneiro, F.; Cruz, A.; Salgueiro-Oliveira, A. Usability Assessment of an Innovative Device in Infusion Therapy: A Mix-Method Approach Study. Int. J. Environ. Res. Public Health 2020, 17, 8335. https://doi.org/10.3390/ijerph17228335

AMA Style

Parreira P, Sousa LB, Marques IA, Santos-Costa P, Cortez S, Carneiro F, Cruz A, Salgueiro-Oliveira A. Usability Assessment of an Innovative Device in Infusion Therapy: A Mix-Method Approach Study. International Journal of Environmental Research and Public Health. 2020; 17(22):8335. https://doi.org/10.3390/ijerph17228335

Chicago/Turabian Style

Parreira, Pedro, Liliana B. Sousa, Inês A. Marques, Paulo Santos-Costa, Sara Cortez, Filipa Carneiro, Arménio Cruz, and Anabela Salgueiro-Oliveira. 2020. "Usability Assessment of an Innovative Device in Infusion Therapy: A Mix-Method Approach Study" International Journal of Environmental Research and Public Health 17, no. 22: 8335. https://doi.org/10.3390/ijerph17228335

APA Style

Parreira, P., Sousa, L. B., Marques, I. A., Santos-Costa, P., Cortez, S., Carneiro, F., Cruz, A., & Salgueiro-Oliveira, A. (2020). Usability Assessment of an Innovative Device in Infusion Therapy: A Mix-Method Approach Study. International Journal of Environmental Research and Public Health, 17(22), 8335. https://doi.org/10.3390/ijerph17228335

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop