Design and Validation of a Web-Based Exploratory Survey Investigating Qualified Professionals and Volunteers Using 3D Printing for Upper Limb Prosthetics: A Methodological Paper

Abstract

Background: 3D printing (3DP) workflow has made its entry in the upper limb prostheses (ULP) manufacturing process. Although it represents a valuable change in clinical practice, its implementation is not ubiquitous. Additional data are required to establish recommendations and unanimously accepted guidelines to facilitate clinical application. Objectives: Our study aimed to develop and validate a web-based multilingual survey investigating the sociodemographic and technical profiles and expertise of professionals and volunteers using 3DP for manufacturing ULP. Methods: We followed a multi-stage development and validation process, including item generation, experts’ review, cognitive testing and pre-testing among the population of interest (POI). Validity evidence was accumulated at each stage, with Content Validity and Face Validity measurements. The survey was available in French, English and Spanish and distributed through the REDCAP web-based platform. Results: The validated questionnaire comprised fifty-two primary questions, organized in nine sections. Experts’ evaluations demonstrated appropriate topic coverage and a high degree of relevance throughout the survey: most single item Content Validity Indexes (CVI) ranged from 0.87 to 1 and Average CVIs for survey sections reached between 0.86 to 1. The pre-test among the POI included 42 participants and led to limited questionnaire revisions. The final version of the survey was approved unanimously by all experts. Conclusions: The newly developed web-based survey demonstrated good evidence for validity. This instrument is an acceptable tool to investigate stakeholders using 3DP for manufacturing ULP and to further establish guidelines.

1. Introduction

1.1. Background

Upper limb differences, due to traumatic or congenital etiologies, cause limitations in impaired individuals’ ability to perform Activities of Daily Living (ADL) [1,2]. Furthermore, such impairments can alter pediatric patients’ psychomotor development [1,2,3]. In order to hinder such psycho-social and somatic repercussions, prostheses create the theoretical possibility to replace the missing limb segment [4]. Upper limb prostheses are assistive devices provided to patients lacking an upper extremity segment on one or both limbs [1,2,4]. Prosthetic devices’ purpose is to compensate for a missing segment, either only aesthetically or by restoring some degree of function [4]. Costs of currently available upper limb prostheses are high and can represent a limiting factor for either low-income households or children, as the latter’s growth logically requires frequent replacements [3,5,6]. Moreover, traditional manufacturing processes of prosthetic devices can be cumbersome, time-consuming and require a high amount of raw material which in turn leads to non-negligible waste [7].

Over recent years, digitalization (i.e., 3D scanners, digital workflow) and additive manufacturing (AM), commonly referred as “3D printing”, have been explored and implemented by qualified professionals (prosthetists, engineers, designers etc.) as they offer partial solutions to some of these limitations: reduction of manufacturing time, material consumption and waste, improvement in efficiency and customization, just to name a few [7,8,9,10]. In addition, some non-governmental organizations (NGOs) and volunteering professionals and private individuals, often called “makers”, use 3D printing to produce upper limb prosthetic or gripping-assistive devices at affordable costs [5,6,9,10]. AM technology consists of the production of a three dimensional (3D) physical object from its 3D virtual image, layer by layer through various possible technical processes. These methods vary in complexity, accessibility and affordability [9,11].

Although numerous case reports and small cohort studies discussing the use of 3D printing technologies to manufacture upper limb prostheses and their componentry are available in the literature, no established guidelines exist [6,7,9]. Very few studies were undertaken among professionals (prosthetics, engineers, physicians), NGOs and volunteers to investigate and gather their clinical and technical expertise, preferences and feedback in employing AM to manufacture, partially or entirely, upper limb (UL) prosthetic devices [6,7,9]. For example, Olsen et al. investigated a fully digitized method to produce trans-radial diagnostic sockets and highlighted its pros and cons [7]. The authors reported both participants’ feedback and clinician experts’ perspective [7]. Those experts’ input was essential to provide an appropriate perspective of the digital workflow in comparison to traditional manufacturing methods [7]. Savage et al. conducted a study in nine countries to understand the relationship between the stakeholders collaborating and involved in ecosystems which include 3D printing for the manufacturing of assistive devices [12]. Prior to that work, studies focused on US professionals or makers [12].

Such descriptive studies are essential as they provide information and evidence about relevant technical aspects to produce prosthetic limbs using 3D printing, which paves the way to the establishment of reliable guidelines and recommendations. To the best of our knowledge, no exploratory study has been undertaken to investigate and describe the technical and clinical expertise of qualified professionals (prosthetics, engineers, physicians), NGOs and volunteers in using 3D printing for producing upper limb prostheses, at an international scale.

1.2. Theoretical Background

The relevant research instruments to investigate and gather such information are surveys [13,14,15]. Surveys are established research tools allowing for the capture of individuals’ knowledge, attitudes and behavior pattern at a given time (cross-sectional study) or at multiple occasions (longitudinal study) [13,14,15]. Diverse types of surveys have been used and described: postal-, telephone-, email-, web-, and SMS-based surveys [14,15]. Initial postal survey methodologies are strongly established, still relevant and constitute the foundation for the other categories [14,15,16]. However, each method has its specific advantages and drawbacks [14,16]. The prevalence of Internet- or Web-based survey methodological and field studies has dramatically increased over the past three decades [14,16,17]. The seemingly easy methodology and implementation, and practical benefits have contributed to that increase [14,17].

Indeed, this approach offers the possibility to reach a broader audience and to collect data with rapid turnaround [14,17,18]. Moreover, Internet-based surveys are economically more advantageous than their paper-based counterparts [14,16,18]. Also, such methods can provide the comfort of confidentiality and anonymity for sensitive questions, which cannot be easily obtained in face-to-face interviews [14,16]. Furthermore, multiple online survey host platforms provide user-friendly services for the creation and distribution of surveys [14,16,18,19]. However, despite these appealing pros, Web-based surveys are not devoid of non-negligible limitations [14]. Although Internet access has definitely progressed worldwide, the online population is still considered different from the global population [14,20]. Therefore, generalizability of results must be performed carefully [14,17,18,20]. Hence, sampling, another potential disadvantage of such surveys, is a crucial step in order to provide reliable and faithful data as representativity of the population of interest can be skewed [14,16,17,18,19,20]. Moreover, unlike e-mail surveys, in web-based survey research, information about participants can be limited and not verifiable [14,16]. Finally, Internet-based surveys come with the possibility for a respondent to abandon the study whenever they please, which can impact collected results [14].

Therefore, the quality of collected data and conclusions thereof are highly dependent on the methodology followed to ensure the validity and reliability of questions [14,16,18,20,21,22]. Numerous books and guidelines articles have described the required rigorous stages to develop a valid survey [14,16,17,18,21,22] Validity of a research instrument is considered to be the first and most important element to achieve [14,23]. The most commonly cited definition of validity is the ability for a test to measure what it is intended to measure, with researchers often listing different types of validity [14,17,23]: content validity, face validity, construct validity and criterion validity, for example [23]. However, recently, scale developers and theorists have argued that the validity of an instrument is not to be regarded as a single event supported by specific statistical analysis [23]. Thus, establishing an instrument’s validity should be perceived as an accumulation of evidence obtained through different valid, reliable and documented approaches [23]. These methods correspond to the different types of validity previously mentioned [23]. Furthermore, methodologists advise researchers to determine the approach best suited for their research purpose and design [17,23].

1.3. Objectives

Therefore, this paper aims to describe the successive stages of development of a web-based survey and of its validation which thus consists of the accumulation of validity evidence. This online survey was used to conduct a cross-sectional pilot study among qualified professionals and volunteers who use 3D printing technologies to manufacture upper limb prostheses.

2. Materials and Methods

2.1. Study Goal and Objectives

The goal of our international pilot study was to gather and describe technical and clinical expertise, preferences and feedback from qualified professionals (prosthetists, engineers, physicians, …) and volunteers who use 3D printing technologies to manufacture upper limb prostheses. The chosen research tool was a web-based survey.

From this main goal, objectives and indicators to guide our survey development were derived, which are summarized in

Table 1. Study objectives and indicators.

Objectives	Indicators
To describe the sociodemographic profiles of the target population.	Age Gender Country of residence Education Professional field Income
To explore the rationale for using 3D printing for upper limb prosthetics.	Number of people provided with prostheses Help provided to family members Age of individuals provided with prostheses Past or current use of 3D printing for prostheses
To explore the financial conditions for manufacturing prosthetics with 3D printing.	The frequency of individuals reporting: Manufacturing of prostheses free of charge Manufacturing of prostheses requiring monetary payment or material contribution from recipients Average cost for manufacturing prostheses Support from state or private institutions
To explore possible financial repercussions of using 3D printing for upper limb (UL) prosthetics.	The frequency of individuals reporting: Negative impact on personal finances
To explore possible repercussions on personal life of using 3D printing for upper limb prosthetics.	The frequency of individuals reporting: Negative impact of manufacturing prostheses on their private life Feeling burdened by manufacturing stages Feeling valued for their work
To explore feedback from recipients.	The frequency of individuals reporting: Feedback received by recipients or their family
To explore the level of professionals’ skills in manufacturing UL prosthetics with 3D printing.	The frequency of individuals reporting: Manufacturing stages performed by self Being trained to test prostheses onto recipients Main issues in manufacturing stages
To explore follow-up provided to recipients of UL prosthetics, when manufactured with 3D printing.	The frequency of individuals reporting: Follow-up (frequency, added-value, stakeholders)
To describe technical profile of the target population.	Expertise in 3D printing Expertise in prosthetic field 3D printing processes for manufacturing prostheses Material Software Reparation procedures
To explore collaboration among stakeholders for manufacturing UL prosthetics.	The frequency of individuals reporting: Collaborative relationships with other professionals Professionals in collaboration Type of collaborations
To explore expertise with UL prosthetics componentry, manufacture, assembly and maintenance when manufacturing with 3D printing, regarding:	Type of components for joints Main damaged areas and causes of failure of prostheses Need for specific customization Adjustments to be made Categories of manufactured prostheses Level of produced prostheses Name of prosthesis

. The objectives and indicators were reviewed by a methodologist, expert in survey research.

2.2. Research Tool

As our target audience was spread over a wide geographical area and from diverse socio-economic and professional backgrounds, and due to the nature of our research purpose, an online survey was deemed a relevant, cost efficient and suitable solution. Moreover, professionals and private individuals owning or subcontracting 3D printing equipment need to possess a computer and, consecutively, an Internet connection, for hardware control and updates, customer service and community sharing. Therefore, our population of interest constitutes a suitable and literate target for web-based surveys.

2.3. Distribution

Study data were collected and managed using REDCap (version 13.11.1, Vanderbilt University, Nashville, TN, USA) electronic data capture tools hosted at Cliniques Universitaires Saint-Luc, Brussels, Belgique [24,25]. REDCap (Research Electronic Data Capture) is a secure, web-based software platform designed to support data capture for research studies, providing (1) an intuitive interface for validated data capture; (2) audit trails for tracking data manipulation and export procedures; (3) automated export procedures for seamless data downloads to common statistical packages; and (4) procedures for data integration and interoperability with external sources [24,25].

2.4. Stage 1—Preliminary Questionnaire

2.4.1. Item Generation

Study indicators oriented the development of a pool of items constituted of question items that were both identified from other relevant published validated questionnaires, related works or textbooks, and that were formulated by the research team. Items’ format varied: we opted for both open-ended (OEQ) and close-ended (CEQ) questions, with a majority of the latter. The CEQ comprised multiple-choice questions (MCQ), dichotomous single-choice questions (SCQ) and Likert-like rating scales. Recommendations to develop clear, unambiguous, neutral and not double-barreled items were followed. For MCQ, response options contained all possible alternatives. The purpose of the survey being exploratory and descriptive, no item response options were defined as “correct answers”. A preliminary version of a self-administered questionnaire was constituted by this pool of items.

The entire initial version of the survey can be found in the Supplementary Materials.

2.4.2. Translation

The initial questionnaire was developed in French. Therefore, in order to possibly allow a high rate of individuals to enroll, it was secondarily translated into English (ENG) and Spanish (SPA) by researchers who possessed a good knowledge of these languages and of the area of interest, supported by native speakers. The two translated versions (ENG and SPA) were then each reviewed by two independent translators who had expertise in reviewing and translating biomedical works. If any disagreement between reviewers occurred, translations were discussed among the research team until a consensus was reached. This methodology is based on translation recommendations suggested by Cha et al. [26]. The French version of the questionnaire was also reviewed by an independent translator with expertise in reviewing and translating biomedical works. Although recommendations suggest multiple translation levels (forward and back translations) for research instruments, we followed this methodology due to local resource constraints and timeframe, as reported by Cha et al. [26].

Translation of the questionnaire occurred after four different stages: initial version of questionnaire; secondary to experts’ review; secondary to cognitive testing; secondary to the piloting phase among the audience of interest.

2.5. Stage 2—Validity Evidence

2.5.1. Experts’ Review—Content Validity

The content validity method gathers validity evidence related to items’ relevance as assessed by a panel of experts in the field of interest. Hence, our questionnaire was tested for content validity by a panel of eight experts [23,27]. These experts were qualified prosthetists, bioengineers, orthopedic surgeons and experimented volunteering “makers” of 3D-printed upper limb assistive devices. They were tasked to rate each survey item on its relevance on a 4-point Likert-type ranking question, ranging from “not relevant” (score 1), “somewhat relevant” (score 2), “quite relevant” (score 3), to “highly relevant (score 4) [27,28]. Moreover, experts were asked to share any suggestions for improving or revising items. In addition, experts were to indicate if they would consider the questionnaire complete and if it covered the subject correctly. Finally, experts had to report if there were any additional questions that they expected to find in the survey but were omitted. If so, they could suggest those. Their inclusion in the survey was up to the discretion of the research team.

For each survey item, an item content validity index (I-CVI) was calculated. For each section of the questionnaire, a scale content validity index (S-CVI) was also calculated. The I-CVI value corresponds to the proportion of experts rating an item content as valid (i.e., relevance rated at 3 or 4). It was calculated using the following formula [27,28]:

$$\text{I-CVI}={\displaystyle \frac{\mathrm{n}\mathrm{u}\mathrm{m}\mathrm{b}\mathrm{e}\mathrm{r} \mathrm{o}\mathrm{f} \mathrm{e}\mathrm{x}\mathrm{p}\mathrm{e}\mathrm{r}\mathrm{t}\mathrm{s} \mathrm{r}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{n}\mathrm{g} \mathrm{t}\mathrm{h}\mathrm{e} \mathrm{i}\mathrm{t}\mathrm{e}\mathrm{m} \mathrm{a}\mathrm{t} 3 \mathrm{o}\mathrm{r} 4}{\mathrm{n}\mathrm{u}\mathrm{m}\mathrm{b}\mathrm{e}\mathrm{r} \mathrm{o}\mathrm{f} \mathrm{e}\mathrm{x}\mathrm{p}\mathrm{e}\mathrm{r}\mathrm{t}\mathrm{s}}}$$

Two methods allowed us to calculate the S-CVI. The first one corresponds to the proportion of items of a scale reaching “universal agreement” (UA) among experts, being rated 3 or 4 by all experts (S-CVI UA) [27,28]. If the UA was achieved for a given item, its UA value was 1. If not, the value was 0. The second approach consists of calculating the average of I-CVIs of all individual items (S-CVI Ave) of a scale/section/domain. The S-CVIAve and the S-CVIUA were calculated using the following formula [27,28]:

$$\text{S-CVI} \text{Ave}={\displaystyle \frac{\mathrm{s}\mathrm{u}\mathrm{m} \mathrm{o}\mathrm{f} \text{I-CVI} \mathrm{s}\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{e}\mathrm{s} \mathrm{o}\mathrm{f} \mathrm{a} \mathrm{s}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}}{\mathrm{n}\mathrm{u}\mathrm{m}\mathrm{b}\mathrm{e}\mathrm{r} \mathrm{o}\mathrm{f} \mathrm{i}\mathrm{t}\mathrm{e}\mathrm{m}\mathrm{s} \mathrm{o}\mathrm{f} \mathrm{a} \mathrm{s}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}}}$$

$$\text{S-CVIUA}={\displaystyle \frac{\mathrm{s}\mathrm{u}\mathrm{m} \mathrm{o}\mathrm{f} \mathrm{U}\mathrm{A} \mathrm{s}\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{e}\mathrm{s} \mathrm{o}\mathrm{f} \mathrm{a} \mathrm{s}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}}{\mathrm{n}\mathrm{u}\mathrm{m}\mathrm{b}\mathrm{e}\mathrm{r} \mathrm{o}\mathrm{f} \mathrm{i}\mathrm{t}\mathrm{e}\mathrm{m}\mathrm{s} \mathrm{o}\mathrm{f} \mathrm{a} \mathrm{s}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}}}$$

Content validity indexes are considered good from 0.78 for I-CVI, 0.90 for S-CVIAve and 0.80 for S-CVIUA [27,28]. However, as Polit et al. advised, we primarily considered S-CVIAve over S-CVIUA for evaluating sections’ overall relevance, as the former better “embodies information about the performance of each item through the averaging feature” [27]. Based on calculated indexes and suggested modifications, survey items were either accepted, rejected or modified accordingly.

The revised questionnaire was then processed further through the Face validity phase.

2.5.2. Face Validity

The face validity phase provides validity evidence that any ambiguity or any difficulty of comprehension of concepts included in the survey are detected and corrected appropriately.

• Experts’ evaluation

Alongside assessing items’ relevance, experts were also asked to rate on a Likert-type scale, from 1 (absent) to 4 (highly), the degree of understandability and acceptability of each item for prospective adult respondents. Items considered poorly understandable were clarified. Those rated as not acceptable were either rejected or maintained in the questionnaire after team discussion, based on their relevance for the overall study.

2.

Cognitive testing

The purpose of this cognitive and qualitative approach was to ensure the clarity, the appropriateness and the adequacy of questions, and that they produce the intended data from the target population’s perspective.

Five individuals with similar characteristics to the target population agreed to participate voluntarily in this pre-test phase. They were recruited through a short recruitment campaign on specialized social media (dedicated forums, Facebook pages) and online platforms (LinkedIn), and through e-mails to professional associations. The link to the online survey was shared, accompanied by a short description of this validation stage.

Respondents were asked to complete the entire survey online and to qualitatively document every section of the survey, by reporting any unclear or irrelevant elements, missing elements, and by suggesting corrections. The qualitative data collected from this stage were discussed among researchers and used to refine and finalize the questionnaire structure. The revised and reorganized version of the survey being validated by experts and a limited sample of the target now requires a larger-scale study to confirm its validity and feasibility. Although pre-testing is also part of the validation process, for clarity purposes, it was categorized separately.

2.6. Stage 3—Piloting

A short-term cross-sectional design study was undertaken to evaluate the validity of the new survey. Such a pre-test is essential to confirm that the target population adequately understands the developed question items and proposed response options as intended by our research team, and that it is capable of answering meaningfully.

2.6.1. Sampling

Our first intention was to obtain a sample frame composed of members from professional associations, learned societies and NGOs, and volunteering private individuals. However, due to respective privacy policies and an important absence of replies from contacted organizations, it was not possible to establish a clear evaluation of the population of interest. Besides organizations’ general membership information online, no other information was available to draw population lists and to adequately estimate the sample size. Therefore, the remaining options to attempt to access the population of interest were non-probability sampling methods such as volunteer opt-in and snowball approaches. As a sample size could not be reliably estimated, the required sample size for the pre-test was determined using the method described by Perneger et al. [29]. They recommended threshold sample sizes depending on their statistical power to detect problems, according to their prevalence, in psychometric questionnaires [29]. The authors recommended a pre-test sample size of a minimum of 32 respondents to achieve a reasonably high power of detection, about 80%, for a problem occurring in 5% of the population [29]. Moreover, this represents a minimum value. Therefore, the more respondents participate in the pre-test, the higher the power to detect errors and refine a research instrument [16,29]. Hence, within the context of our study and the limited access to the population of interest, our research team opted for a sample size of minimum thirty-two respondents. However, efforts were made to reach a higher number. The pre-test was run between 1 August 2023 and 31 August 2023. As recruitment for voluntary and anonymous online surveys can be complicated, a large sample size could reduce the prospective pool available for the main online pilot study. Therefore, it was decided that the pilot test would be either concluded if a sample size of 50 respondents was achieved before 31 August or if a minimum of 32 participants participated in the test survey by 31 August.

2.6.2. Recruitment

A presentation message describing the study pre-test’s purpose including the link to the web-based survey was prepared. It was posted on social media (Facebook pages, LinkedIn) and online platforms (forums) dedicated to professionals (prosthetists, engineers, physicians, …) and volunteers using 3D-printing for manufacturing upper limb prostheses. Moreover, e-mails were sent inviting learned societies and professional associations to share the purpose of the study and the link to the survey to their respective members list. Nevertheless, almost none of these official organizations replied to our inquiries. Also, some private dedicated companies were also contacted via e-mails.

As participation was voluntary and anonymous, any individual willing to participate in the pre-test could simply click on the link and complete the survey. The survey was accessible for a period of 30 days. Participants were to give their informed consent before taking the survey. Afterwards, they were asked to specify their age, as being underage (i.e., below 18 years old) in their current country of residence was considered as an exclusion criterion. Other selection criteria comprised: being a current or past manufacturer of upper limb prosthetic devices using 3D printing, being capable of providing informed consent to enroll in the study and having access to the Internet.

2.7. Stage 4—Final Experts’ Review

A final round of experts’ assessment was undertaken after the piloting phase and the implementation of modifications derived from data analysis. The revised questionnaire was submitted to the same panel of experts to evaluate changes according to the same modalities as aforementioned, as in Stage 2.

2.8. Ethical Considerations

On the welcome page of the online survey, participants in the pre-testing phases could read more detailed participant information, and a PDF file containing complete in-depth details on the study and their rights could also be downloaded. Secondly, in order to participate in the study, respondents were required to give their consent by answering “I agree to participate in the study” to a consent question after reading that information. Ethical approval was obtained through the Ethics Committee of Saint-Luc University Hospital, Brussels, Belgium. All survey responses were anonymous and were treated confidentially. There were no real potential ethical issues or risks from participating other than that respondents were required to reflect on personal experience. Prospective respondents were informed on the main page of the survey of their right to withdraw at any time. Participants were also informed that all data, even partial, were recorded. Additionally, due to the international nature of the web-based survey, we ensured to be aligned with data privacy regulations. Hence, only personal data strictly necessary for the purposes and duration of the study were collected (i.e., age, gender, country of residence, level of education, professional field). No data leading to the identification of participants were extracted (e.g., name, IP address, ID number, phone number, e-mail address, face photographs, address, bank details, audio-video recording, etc.). Collected data were stored on a secure database, in an ISO27001 environment [30], only accessible after logging with a password and backups were performed. Finally, no data were transferred out of the research center.

2.9. Data Analysis

Content validity indexes were computed using Microsoft Excel software (version 2507, Microsoft Corporation, Redmond, WA, USA). Values were interpreted using the threshold mentioned previously as reported by Polit et al. [27,28]. Moreover, corrections and suggestions provided by the panel of experts were thoroughly analyzed and followed if relevant to the overall study. All qualitative feedback and suggestions received through the face validity stage were implemented as they improved the clarity and readability of the questionnaire. Data obtained through the pilot phase were thoroughly examined.

The completion rate (CR) was calculated as follows:

$$\mathrm{CR}={\displaystyle \frac{\mathrm{n}\mathrm{u}\mathrm{m}\mathrm{b}\mathrm{e}\mathrm{r} \mathrm{o}\mathrm{f} \mathrm{p}\mathrm{a}\mathrm{r}\mathrm{t}\mathrm{i}\mathrm{c}\mathrm{i}\mathrm{p}\mathrm{a}\mathrm{n}\mathrm{t}\mathrm{s} \mathrm{w}\mathrm{h}\mathrm{o} \mathrm{c}\mathrm{o}\mathrm{m}\mathrm{p}\mathrm{l}\mathrm{e}\mathrm{t}\mathrm{e}\mathrm{d} \mathrm{t}\mathrm{h}\mathrm{e} \mathrm{s}\mathrm{u}\mathrm{r}\mathrm{v}\mathrm{e}\mathrm{y}}{\mathrm{n}\mathrm{u}\mathrm{m}\mathrm{b}\mathrm{e}\mathrm{r} \mathrm{o}\mathrm{f} \mathrm{p}\mathrm{a}\mathrm{r}\mathrm{t}\mathrm{i}\mathrm{c}\mathrm{i}\mathrm{p}\mathrm{a}\mathrm{n}\mathrm{t}\mathrm{s} \mathrm{w}\mathrm{h}\mathrm{o} \mathrm{g}\mathrm{a}\mathrm{v}\mathrm{e} \mathrm{t}\mathrm{h}\mathrm{e}\mathrm{i}\mathrm{r} \mathrm{c}\mathrm{o}\mathrm{n}\mathrm{s}\mathrm{e}\mathrm{n}\mathrm{t}}}$$

Furthermore, descriptive analysis was used to analyze data obtained from Stage 3, the pilot phase. Questionnaire items were analyzed to highlight any specific pattern in respondents’ response behavior. Missing answers were also reviewed to identify any potential pattern and were managed using the pair-wise method [31,32]. Content analysis was applied on qualitative answers when applicable to determine if redundancy appeared. If so, the possibility of modifying response options or question format was discussed among the research team.

3. Results

3.1. Stage 1—Preliminary Questionnaire

Item Generation

Based on defined indicators and literature review, an initial draft of a questionnaire was obtained. A total of 49 primary question items were generated. This early version of the questionnaire was constituted of 20 multiple choice questions (MCQ), 16 dichotomous single-choice questions, 8 open-ended questions (OEQ) and 5 Likert-like rating scale questions. Among these, 31 questions were contingency questions. In most cases, response options of contingency questions would include a checkbox “Other” leading to a secondary conditional question, if chosen. Other conditional questions were to provide deeper understanding of respondents’ knowledge or experience. After analyzing items topics and scope, it appeared that all were independent constructs exploring distinct concepts. The initial version of the questionnaire can be found in the Supplementary Materials.

3.2. Stage 2—Validity Evidence

3.2.1. Content Validity Evidence

The panel of eight experts reviewed the entire preliminary questionnaire and rated the relevance of each item.

Table 2. Content Validity Index (CVI) values and items revision.

Section n°1—Sociodemographic A	EIA a	I-CVI b	UA c
Are you a manufacturer of 3D-printed upper limb prostheses?	8	1	Yes
2. What is your age?	5	0.62	No
3. In what country do you live?	7	0.87	No
4. d What is your gender?	6	0.75	No
5. e What is the highest qualification or level of education you have completed?	8	1	Yes
6. Are you currently employed?	6	0.75	Yes
7. In which professional field do you work?	8	1	No
S-CVIAve f	0.86
S-CVIUA g	0.43
Section n°2 Sociodemographic B—3D Printing Experience	EIA	I-CVI	UA
Questions #8 to #13 h	8	1	Yes
14. d Why did you decide to start making 3D-printed upper limb prostheses?	7	0.87	No
15. d Regarding you helping people who need a 3D- printed upper limb prosthesis, you do it as…	8	1	Yes
S-CVIAve	0.98
S-CVIUA	0.87
Section n°3 Technical Aspects	EIA	I-CVI	UA
Questions #16 to #20 h	8	1	Yes
21. d What type(s) of cables do you use to make your prosthesis functional?	7	0.87	No
22. e What types of attachments do you use to attach the prosthesis to the recipient’s stump?	8	1	Yes
23. e What joint systems do you use to link the different parts of the prosthesis together?	8	1	Yes
24. e What solutions do you usually choose to repair a recipient’s prosthesis?	7	0.87	No
Questions #25 to #29 h	8	1	Yes
S-CVIAve	0.98
S-CVIUA	0.86
Section N°4 Prosthetics	EIA	I-CVI	UA
30. d Which of the following 3D upper limb prostheses have you ever printed?	6	0.75	No
Questions #31 to #39 h	8	1	Yes
S-CVIAve	0.97
S-CVIUA	0.9
Section n°5 Personal Perceptions	EIA	I-CVI	UA
Questions #40 and #41 h	8	1	Yes
S-CVIAve	1
S-CVIUA	1
Section n°6 Manufacturing	EIA	I-CVI	UA
Questions #42to #45 h	8	1	Yes
S-CVIAve	1
S-CVIUA	1
Section n°7 Collaboration	EIA	I-CVI	UA
46. d Who chooses the most appropriate 3D-printed prosthesis design for the prospective recipient?	8	1	Yes
47. Which other professionals collaborate with you to manufacture optimal prostheses for recipients?	8	1	Yes
49. What type of collaboration is involved?	7	0.87	No
50. Does collaboration with other professionals contribute to better-quality 3D-printed prosthetics?	7	0.87	No
53. d At what stage do you involve the future recipient of the prosthesis?	8	1	Yes
55. d Does collaboration with future recipients contribute to better-quality 3D-printed prosthetics?	7	0.87	No
S-CVIAve	0.94
S-CVIUA	0.5
Section n°8 Follow-Up	EIA	I-CVI	UA
Questions #56 to #69 h	8	1	Yes
S-CVIAve	1
S-CVIUA	1
Section n°9 Final Comments	EIA	I-CVI	UA
Questions #70 and #71 h	8	1	Yes
S-CVIAve	1
S-CVIUA	1
Removed Items
Section n°1
Thinking of your household’s total monthly or weekly income, is your household able to make ends meet, that is pay your usual expenses	2	0.25	No
Section n°4
Are most of the recipients of the 3D-printed prostheses you have made members of your family?	5	0.62	No
Section n°5
Does making 3D-printed upper limb prostheses put your financial security at risk?	5	0.62	No
Does making 3D-printed upper limb prostheses have a negative impact on your personal life?	5	0.62	No

^a: EIA: Expert in agreement (relevance rating at 3 or 4). ^b: I-CVI: Item-Content Validity Index. ^c: UA: Universal agreement among experts (i.e., all experts rating item relevance at 3 or 4). ^d: Sign marking contingency question. One or some response options lead to additional conditional questions. ^e: Sign marking questions added to survey following experts’ review. ^f: S-CVIAve: Average of I-CVI of a scale or domain. ^g: S-CVIUA: Average of I-CVI of a scale, including only items that reached Universal agreement among experts (i.e., all experts rating item relevance at 3 or 4). ^h: To facilitate Table readability, all consecutive questions, within a given section, that reached universal agreement among experts, were replaced by their question number and were associated with similar statistical values (i.e., EIA = 7; I-CVI = 1; UA = YES); Detailed report of statistical analysis can be found in Table A1. Bold formatting was used to highlight the table subcategories.

displays ratings and values from statistical analysis. Following experts’ recommendations, the order of questions was revised allowing their subdivision in sections facilitating the computation of CVI values. Moreover, based on experts’ appraisal of items’ relevance, some questions were revised or deleted, and new ones were added to the questionnaire, as illustrated in Table 2 and

Table A1. Content Validity Index (CVI) values and items revision (detailed report).

Section n°1—Sociodemographic A	EIA a	I-CVI b	UA c
Are you a manufacturer of 3D-printed upper limb prostheses?	8	1	1
2. What is your age?	5	0.62	0
3. In what country do you live?	7	0.87	0
4. f What is your gender?	6	0.75	0
5. g What is the highest qualification or level of education you have completed?	8	1	1
6. Are you currently employed?	6	0.75	0
7. In which professional field do you work?	8	1	1
S-CVIAve d	0.86
S-CVIUA e	0.43
Section n°2 Sociodemographic B— 3D Printing Experirence	EIA	I-CVI	UA
8. h Are you an orthotist/prosthetist (qualified or in training) or an orthotist/prosthetist technician (qualified or in training)?	8	1	1
9. h As a manufacturer (maker), when did you make your last 3D-printed upper-limb prosthesis?	8	1	1
10. How long have you been in the 3D-printing field (3D design, 3D printing, post-processing, etc.)?	8	1	1
11. h,f Why do you use 3D-printing?	8	1	1
12. h,f How have you gained experience in prosthetics?	8	1	1
13. How long have you been making 3D-printed upper limb prostheses?	8	1	1
14. f Why did you decide to start making 3D-printed upper limb prostheses?	7	0.87	0
15. f Regarding you helping people who need a 3D-printed upper limb prosthesis, you do it as…	8	1	1
S-CVIAve	0.98
S-CVIUA	0.87
Section n°3 Technical Aspects	EIA	I-CVI	UA
16. f What 3D printing technology do you use to print upper limb prostheses?	8	1	1
17. What 3D printer(s) do you use to make upper limb prostheses?	8	1	1
18. f What material do you use to print most of the prosthesis?	8	1	1
19. What design/3D design software do you use for 3D-printed upper limb prostheses? (e.g., Fusion 360, Freeform)	8	1	1
20. What 3D pre-print slicing software do you use? (e.g., Cura, Slic3r)	8	1	1
21. f What type(s) of cables do you use to make your prosthesis functional?	7	0.87	0
22. g What types of attachments do you use to attach the prosthesis to the recipient’s stump?	8	1	1
23. g What joint systems do you use to link the different parts of the prosthesis together?	8	1	1
24. g What solutions do you usually choose to repair a recipient’s prosthesis?	7	0.87	0
25. g Which part(s) of the prosthesis usually break(s) when the prosthesis is damaged?	8	1	1
26. g What usually causes the 3D-printed prostheses you make to break?	8	1	1
27. f,g What adjustments need to be made to 3D prostheses for recipients to wear them more often?	8	1	1
28. f,g How long does it usually take you to make an upper limb prosthesis?	8	1	1
29. f,g Do you regularly modify the chosen prosthesis to meet a specific need of the recipient (such as holding a musical instrument, bicycle handlebars, car steering wheel, etc.)?	8	1	1
S-CVIAve	0.98
S-CVIUA	0.86
Section n°4 Prosthetics	EIA	I-CVI	UA
30. f Which of the following 3D upper limb prostheses have you ever printed?	6	0.75	0
31. Please indicate the names of other protheses you have already 3D printed.	8	1	1
32. f Do you use 3D printing to make most parts of a prosthesis or just a few parts of it?	8	1	1
33. f If it is only to make specific parts, what part(s) of the prosthesis do you print in 3D?	8	1	1
34. f What type(s) of prostheses do you 3D print most often?	8	1	1
35. Which of the following types of prostheses do you print most often?	8	1	1
36. Approximately how many people have you made a 3D-printed upper limb prosthesis for?	8	1	1
37. What age are the people you have made a 3D-printed upper limb prosthesis for?	8	1	1
38. f Regarding the costs for the recipients…	8	1	1
39. g What is the approximate cost for a 3D upper limb prosthesis? (in your local currency)	8	1	1
S-CVIAve	0.97
S-CVIUA	0.9
Section n°5 Personal Perceptions	EIA	I-CVI	UA
40. Do you feel valued in what you do?	8	1	1
41. h Do you feel that the recipients appreciate your help?	8	1	1
S-CVIAve	1
S-CVIUA	1
Section n°6 Manufacturing	EIA	I-CVI	UA
42. Concerning the different steps in the production of 3D-printed prostheses, which ones do you do yourself?	8	1	1
43. f What production steps do you consider most difficult?	8	1	1
44. f,h During your time as a manufacturer (maker), have you ever received training in fitting prostheses onto a recipient?	8	1	1
45. What are the main issues you face when measuring a future recipient’s limbs?	8	1	1
S-CVIAve	1
S-CVIUA	1
Section n°7 Collaboration	EIA	I-CVI	UA
46. f Who chooses the most appropriate 3D-printed prosthesis design for the prospective recipient?	8	1	1
47. Which other professionals collaborate with you to manufacture optimal prostheses for recipients?	8	1	1
48. f How often do you work with these other professionals?	7	0.87	0
49. What type of collaboration is involved?	7	0.87	0
50. Does collaboration with other professionals contribute to better-quality 3D-printed prosthetics?	7	0.87	0
51. Why does collaboration with other professionals contribute to better-quality 3D-printed prosthetics?	8	1	1
52. Why doesn’t collaboration with other professionals contribute to better-quality 3D-printed prosthetics?	8	1	1
53. f At what stage do you involve the future recipient of the prosthesis?	8	1	1
54. Why do you not involve the future recipient of the prosthesis?	8	1	1
55. f Does collaboration with future recipients contribute to better-quality 3D-printed prosthetics?	7	0.87	0
S-CVIAve	0.94
S-CVIUA	0.5
Section n°8 Follow-Up	EIA	I-CVI	UA
56. f Who ensures that the prosthesis does not harm the recipient?	8	1	1
57. f Who ensures that the prosthesis meets the recipient’s needs?	8	1	1
58. f Is there any recipient follow-up after a 3D-printer upper limb prosthesis has been made?	8	1	1
59. Is this follow-up provided for each recipient of a prosthesis?	8	1	1
60. Why is follow-up not provided for each recipient?	8	1	1
61. How often is follow-up carried out?	8	1	1
62. Why “never”? Please specify.	8	1	1
63. f Who does the follow-up?	8	1	1
64. In your opinion, does follow-up enable recipients to use their prosthesis for longer?	8	1	1
65. g How does follow-up enable recipients to use their prostheses for longer?	8	1	1
66. Why does not the follow-up care allow the recipients to use their prostheses longer?	8	1	1
67. f Do you receive support (money, materials) from any association, the state, or another organization to help you make upper limb prostheses?	8	1	1
68. f In your opinion, how many of the people for whom you made 3D-printed upper limb prostheses were satisfied with their prosthesis?	8	1	1
69. What are the main complaints or remarks from recipients about their prosthesis?	8	1	1
S-CVIAve	1
S-CVIUA	1
Section n°9 Final Comments	EIA	I-CVI	UA
70. In your opinion, what technological improvements should be studied to improve 3D-printed upper limb prostheses?	8	1	1
71. What additional comments or remarks would you like to share regarding the advantages, constraints or conditions of use of 3D-printed upper limb prostheses?	8	1	1
S-CVIAve	1
S-CVIUA	1
Removed Items
Section n°1
Thinking of your household’s total monthly or weekly income, is your household able to make ends meet, that is pay your usual expenses?	2	0.25	0
Section n°4
Are most of the recipients of the 3D-printed prostheses you have made members of your family?	5	0.62	0
Section n°5
Does making 3D-printed upper limb prostheses put your financial security at risk?	5	0.62	0
Does making 3D-printed upper limb prostheses have a negative impact on your personal life?	5	0.62	0

^a: EIA: Expert in agreement (relevance rating at 3 or 4). ^b: I-CVI: Item-Content Validity Index. ^c: UA: Universal agreement among experts (All experts rating item relevance at 3 or 4). ^d: S-CVIAve: Average of I-CVI of a scale or domain. ^e: S-CVIUA: Average of I-CVI of a scale, including only items that reached Universal agreement among experts (i.e., all experts rating item relevance at 3 or 4). ^f: Sign marking contingency question. One or some response options lead to additional conditional questions. ^g: Sign marking questions added to survey following experts’ review. ^h: Sign marking revised question items based on experts’ recommendations. Bold formatting was used to highlight the table subcategories.

.

Through the experts’ evaluation, we observed a high degree of relevance of the suggested items. Indeed, most single items reached an I-CVI between 0.87 and 1. Four items had an I-CVI below the cut-off of 0.78 but were retained due to their relevance to the overall study: Three items had an I-ICV 0.75 and one item at 0.62. These items investigated either sociodemographic aspects (e.g., gender, employment status), and were deemed relevant to analyze respondents’ demographics, or specific models of 3DP prostheses manufactured by respondents. Therefore, although they were ranked as poorly relevant only by non-academic experts, the research team decided not to remove them. Based on poor relevance scores, four items were removed from the questionnaire draft. Additionally, the S-CVIAve of each section consequently indicated high average relevance as their values ranged between 0.86 and 1. Details about CVI scores can be found in Table 2 and Table A1. Furthermore, based on experts’ suggestions and recommendations, eleven new items were added to the survey and five items were required to be reworded. Major modifications are summarized in Table 2 and all revisions are detailed in Appendix A Table A1. Following experts’ assessment, section n°3 was significantly expanded with the addition of eight new questions, see Table A1 for details, and other sections lengths remained relatively stable.

Table 2 summarizes the preliminary questionnaire sections after experts’ assessment and the associated CVI scores. Table A1 displays all preliminary questionnaire items and associated CVI scores in full length.

All experts indicated that the survey covered the topics and constructs of interest.

3.2.2. Face Validity Evidence

Experts’ Evaluation

Experts also assessed the understandability and acceptability of survey items. Most items and response options were deemed understandable and acceptable for prospective respondents. Experts recommended the correction or addition of some response options for twelve questions to provide more clarity. Table S1 illustrates those modifications.

Cognitive Testing—Target Population

As aforementioned, five individuals, from the target population, agreed to participate in this small-scale pre-test. The cognitive testing was performed in French and English. From their completion of the online survey and their qualitative feedback, it appeared that all respondents considered the survey as complete and covering the topics of interest adequately. They did not highlight major mistakes or omissions in the questionnaire. From their responses and suggestions, some minor revisions or additions were carried out on some response options for specific questions. For example, respondents recommended a correction in terminology, using “recipient” instead of “user”. This recommendation seemed relevant to the research team as the denomination “user” can relate to a limiting mechanistic view of an assistive device. Conversely, “recipient” appears as a more dignified term referring to the active and progressive individuals’ journey of receiving, accepting and integrating a prosthesis into all aspects of their lives. Therefore, the term “user” was replaced by “recipient” in questions and responses where applicable.

3.3. Stage 3—Piloting

To finalize the accumulation of validity evidence, a larger-scale pre-test among the target population was carried out. Forty-four anonymous individuals, recruited through specialized online platforms, e-mail invitations and personal contacts, accessed the survey webpage. Forty-two prospective respondents provided their informed consent but only forty participants proceeded with the actual questionnaire, after indicating their age and country of residence. Thirty-one participants reached the end of the survey which corresponds to a completion rate of 74%. Respondents’ characteristics are summarized in

Table 3. Characteristics of participants.

Age (Years), n (%)	N = 40
20–30	6 (15)
31–40	9 (22.5)
41–50	8 (20)
51–60	8 (20)
60+	9 (22.5)
mean (SD)	48 (15)
Country, n (%)	N = 40
Belgium	1 (2.5)
Burkina Faso	1 (2.5)
Canada	2 (5)
Colombia	1 (2.5)
France	12 (30.0)
Israel	1 (2.5)
Nigeria	1 (2.5)
Poland	1 (2.5)
Romania	1 (2.5)
Sweden	1 (2.5)
The Netherlands	4 (10.0)
United Kingdom	3 (7.5)
United States of America	10 (25.0)
Venezuela	1(2.5)
Gender, n (%)	N = 38
Male.	29 (76.3)
Female.	9 (23.7)
Education, n (%)	N = 38
High school.	3 (7.9)
Trade/technical/vocational training.	5 (13.2)
College/University degree.	30 (78.9)
Currently Employed, n (%)	N = 38
Yes.	29 (76.3)
No.	9 (23.7)
Professional Field a, n (%)	N = 29
Professional services.	5 (17.2)
Technical services.	3 (10.3)
Craft and trades.	1 (3.4)
Healthcare.	5 (17.2)
Education and teaching.	5 (17.2)
Arts and entertainment.	2 (6.9)
Public services.	2 (6.9)
Agriculture.	1 (3.4)
Information technology.	2 (6.9)
Other (not specified).	3 (10. 3)
Certified Prosthetist (Technician) or in Training, n (%)	N = 36
Yes.	2 (5.6)
No.	34 (94.4)
As Manufacturer, Last Production of ULP b with 3DP c, n (%)	N = 36
Less than 1 year ago.	25 (69.4)
1 to 3 years ago.	5 (13.9)
More than 3 years ago.	4 (11.1)
Approved by an association (e.g., NGO), awaiting requests.	2 (5.6)
Having Been in the 3D Printing Field, n (%)	N = 35
Less than 1 year.	1 (2.9)
1 to 5 years.	15 (42.9)
6 to 10 years.	17 (48.6)
More than 10 years.	2 (5.7)
Rationale for Being in the 3DP Field, n (%) d	N = 36
It is a hobby.	27 (75)
It is my job.	7 (19.4)
It is a way to help other people.	28 (77.8)
Other (not specified).	3 (8.3)
Experience in Prosthetics, n (%) d	N = 36
It is my professional field.	2 (5.6)
By voluntarily making 3D-printed prostheses to help.	29 (80.6)
It is a hobby.	13 (36.1)
Other (not specified).	1 (2.8)
Having Been Manufacturing ULP Using 3DP, n (%)	N = 36
Less than 1 year.	7 (19.4)
1 to 5 years.	18 (50)
6 to 10 years.	11 (30.6)
Rationale for Manufacturing ULP Using 3DP, n (%) d	N = 36
It is my job.	4 (11.1)
To earn some money.	0
To help.	32 (88.9)
Just to try.	4 (11.1)
To take part in a technology competition.	0
It is a hobby.	11 (30.6)
Other (not specified).	2 (5.6)
Status for Providing ULP Manufactured Using 3DP, n (%) d	N = 36
Independent volunteer.	12 (33.3)
Volunteer, member of an aid association.	24 (66.7)
Employee of an aid association.	0
Employee of a company.	3 (8.3)
Prefer not to answer.	0
Other (not specified).	2 (5.6)

^a: Professional services: Lawyers, accountants, consultants, etc.; Technical services: Engineers, computer specialists, technicians, electricians, etc.; Craft and trades: Carpenters, cabinetmakers, etc.; Healthcare: Orthotists/Prosthetists, doctors, etc.; Education and teaching: Teachers, professors, educators, trainers, etc.; Arts and entertainment: Actors, musicians, etc.; Public services: Firefighters, police officers, etc.; Agriculture: Farmers, breeders, etc.; Information technology: Software developers, etc. ^b: ULP—Upper limb prosthesis. ^c: 3DP—3D printing. ^d: Multiple Choice Questions are associated with a total percentage of cases by question superior to 100 as a single participant could select multiple responses. Bold formatting was used to highlight the table subcategories.

.

The rate of missing answers varies between 1.6% to 20% depending on the survey section and the type of questions. Indeed, open-ended questions’ response rates are commonly lower due to a higher cognitive cost [21,22]. Detailed information about missing answers is displayed in

Table 4. Missing answers, Modifications—Pilot phase.

Section	Number of Respondents	Number of Questions	Missing Answers (n)/Rate f (%)/Type of Questions g	Frequency of ”Other” Responses per Section/R a–NR b	Modification in Questions
Section n°1	N = 41	6	n = 4; 1.6%. Type: OQ c: 4	n = 4 NR: 4	No
Section n°2	N = 36	8	n = 9; 3% Type: CEQ d: 1 OQ: 8	n = 8 NR: 8	No
Section n°3	N = 34	15	n = 59; 11.5% Type: OEQ e: 13 CEQ: 41 OQ: 5	n = 33 R: 28 NR: 5	Yes—6 revised questions
Section n°4	N = 34	9	n = 30; 10% Type: CEQ: 29 OQ: 1	n = 4 R: 3 NR: 1	No
Section n°5	N = 34	2	n = 9; 13% Type: CEQ: 9	N/A	No
Section n°6	N = 34	4	n = 28; 20% Type: OEQ: 17 CEQ: 10 OQ: 1	n = 1 NR: 1	Yes—2 revised questions
Section n°7	N = 34	10	n = 32; 9.4% Type: CEQ: 16 OEQ: 8 OQ: 7	n = 14 R: 7 NR: 7	Yes—3 revised questions
Section n°8	N = 34	16	n = 67; 12% Type: OEQ: 24 CEQ: 41 OQ: 2	n = 25 R: 23 NR: 2	Yes—5 revised questions
Section n°9	N = 31	2	n = 30; 48% Type: OEQ: 2	N/A	Yes—1 revised question

^a: R: Number of responses provided to the secondary question “If Other, please specify”, in each section. ^b: NR: Number of non-responses provided to the secondary question “If Other, please specify”, in each section. ^c: OQ: “Other” response option (which leads to an open-ended question). ^d: OEQ: Open-ended question, different from “Other, please specify” open-ended questions. ^e: CEQ: Close-ended question. ^f: “Rate” refers to the rate (relative value) of missing answers in section distinctively. ^g: “Type of questions” refers to the absolute number of missing answers for each type of questions (OEQ, OQ, CEQ) in each section.

.

From the analysis of these pre-test data, it appeared that few major modifications were required. Most corrections targeted response options of specific questions items. Besides the merging of two questions in section n°9 due to apparent redundancy, no other question was removed. One conditional question was added to clarify the “I do not know” or “Not applicable” responses to a new contingency question. Only one question was required to be reformulated in order to include a larger prospective audience that works with children: “What are the main complaints or remarks from recipients [(or from their parents)] about their prosthesis?”. The majority of other corrections were the addition, the rewording and the removal of response options. These modifications were based on the frequency of similar answers to open-ended questions and on participants’ response behavior to close-ended questions. Finally, items n°18 and n°19 from Section n°3 of the questionnaire (Table A1) presented both a high rate of missing answers and of “Other” options chosen by respondents. After analysis, we concluded these conditions were due to a long list of suggested responses options and a lack of relevant response alternatives. Therefore, it was decided to reduce the number of response options by categorizing answer alternatives and to include responses frequently reported by respondents. Table S2 summarizes all modifications implemented consecutively to this pilot stage.

Additionally, besides Section n°9 being shortened from two questions to one, other sections lengths remained stable.

3.4. Stage 4—Final Experts’ Review

A final round of evaluation by the panel of experts was undertaken to evaluate the relevance, the understandability and the acceptability of all modifications derived from the large-scale pre-test, summarized in Table S2. All experts evaluated each revised survey item and response options as relevant, understandable and acceptable for prospective respondents.

The final version of the online survey is constituted of fifty-two primary questions, and forty-six conditional secondary questions, and can be found in full in the Supplementary Materials.

4. Discussion

The purpose of this work was to report the development and validation stages of a survey designed for a web-based international cross-sectional pilot study. The aim of this study was to attempt the description of socio-demographic and professional profiles and to gather the expertise and knowledge of qualified professionals and volunteer individuals who use 3D printing technologies to manufacture upper limb prosthetic devices.

Our methodology and this paper’s structure were derived from reference works [14,16,18,22], and guidelines for good reporting practice to ensure transparency and quality of this report, such as the CHERRIES checklist from Eysenbach et al. [33], guidelines reported by Bennett et al. [34], and quality criteria published by Andrews et al. [16].

A completed CHERRIES checklist is included in the Supplementary Materials.

4.1. Research Tool

Internet-based surveys present numerous advantages as shown in Section 1 [14]. However, some drawbacks cannot be avoided, especially when using non-probability sampling [14,16]. For example, contrary to random samples, non-random online surveys do not provide information about respondents [14]. They consequently require specific socio-demographic questions to obtain information about participants which lengthen the questionnaire [14,16]. Indeed, in our final survey, twelve questions assessed the socio-demographic, professional and technical profiles of respondents, which is non-negligible. Moreover, as any individual can volunteer to opt into the study, the validity and truthfulness of data cannot be thoroughly verified. Therefore, working with sample frames and the support of governing bodies of professional associations and NGOs can reduce the impact of such constraints and risks of bias [14,16,18]. Those limitations are to be considered in order to draw careful and realistic conclusions.

4.2. Questionnaire Format

Scientific literature presents contradictory evidence concerning the impact of survey length on completion rates of online surveys [14,16,22]. The final version of our survey was composed of fifty-two primary questions and forty-six conditional secondary question items. The majority of these secondary questions aimed to clarify the response option “Other” chosen by respondents.

Although the completion rate of the pilot stage was considered good, it does not ensure similar results at the actual survey research. We maintained the primary open-ended question (OEQ) if responses provided through the pilot phase were considered relevant, diverse and meaningful. Only one primary open-ended question was shifted into a MCQ as the responses provided were relevant but highly redundant.

The general survey design was kept simple (no graphical media) to maintain a low download time and higher accessibility [16,21,22]. Due to the length of the survey, questions were displayed through a Multipage Questionnaire. However, the number of questions were carefully limited to minimize excessive scrolling [14,16,22,33].

4.3. Translation

Although recommendations suggest multiple translation levels (forward and back translations) for research instruments, due to local resources and timeframe, we opted for another valid alternative as aforementioned in Section 2.4.2 [26,35]. Our approach could introduce biases: as researchers initially translated the questionnaire, they could lack the objectivity of native translators and the cultural knowledge of target languages [26,35]. Also, despite being guided by native speakers, researchers could fail to ensure content equivalence and maintain an accessible linguistic level in both target languages [26,35]. Independent forward (i.e., from original language to target language) translations can underline potentially imprecise or ambiguous concepts to convey as discrepancies between translated versions will emerge [26,35,36]. Backward (i.e., from target language to original language) translations can highlight potential misunderstandings and unclear, or inadequate, wordings from initial translations [26,35,36]. Therefore, the well-known Brislin’s approach of back-translation does prevent major translation errors and inaccuracies [26,36]. However, the reality of limited availability of bilingual translators remains a weakness of this method as each translation version and round should be carried out independently [26,36]. Thus, in our methodology, the translated versions of our questionnaire were each reviewed by two independent bilingual translators and accepted by the research team.

During the cognitive testing, no difficulty in understanding question items and response options were reported by participants. At the pilot phase, surveys were completed by French-, English- and Spanish-speakers and no specific response patterns, indicative of translation issues, were detected.

4.4. Response Options

In order to avoid any order effect for answering MCQ, if no logical order was required, response options were not systematically ordered alphabetically [14,22]. As the respondents’ experience with electronic surveys was unknown, no drop-down menu was used [14]. Matrices were used sparingly as literature about its effect on measurement error is unclear [14,21,22]. They were mainly employed for two ranking scale questions (“totally agree” to “totally disagree”) and for one question evaluating a single factor (i.e., recipients’ satisfaction) over time. Open-ended questions were considered with caution in our survey due to their high cognitive cost [18,21,22]. As expected, OEQ had globally a higher rate of missing answers [16]. However, responses obtained from primary OEQ and the pilot phase provided both sufficient similarity and diversity to cover the investigated constructs. Besides questions requiring respondents’ informed consent, age and country of residence, no other survey question required an answer before proceeding through the questionnaire. Choices such as “I do not know”, “not applicable”, or “Prefer not to answer” were included if necessary [16,18].

4.5. Unique Participation

As some prospective respondents could share IP addresses (e.g., coworkers) no specific measures could be taken to ensure that each individual would participate in the study only once (i.e., cookies, passwords) [16,18,21,22]. However, efforts were undertaken during analysis to identify possible similar responses pattern across the dataset. None were clearly identified which does not rule out the possibility of multiple participations [14,18].

4.6. Experts’ Review

Experts’ review provided essential insights and suggestions that led to the reorganization of the questionnaire and the refinement of questions. They recommended specific conditional questions to allow respondents to provide more meaningful responses. Moreover, CVI scores reported in Table 2 and Table A1 demonstrated global high relevance degrees [27,28]. S-CVIAve, as indicated by Polit et al., better reflects the weight of individual I-CVI items in comparison to S-CVIUA scores [27]. S-CVIUA values can be strongly weighed down by the absence of agreement between experts which is not indicative of a lack of relevance [27].

As recommended by Polit et al. [27], we conducted a second experts’ review following all development and validation stages, and consecutive modifications. That final round of evaluation provided a closure step ensuring uniformity to the questionnaire before its distribution [16].

4.7. Piloting

4.7.1. Sampling—Recruitment

Prospective respondents were recruited through non-probability sampling methods as aforementioned due to the inaccessibility to a sample frame of the target audience [14,16,17]. Although this approach allowed us to increase the sample size from the population of interest (POI), its non-random characteristics consequently limit its representativity [14,16,18,20]. Nevertheless, for exploratory pilot research, such as this study, non-probability samples are not uncommon situations as undescribed POI are to be investigated [14,15,16,17,18,22]. However, data analysis and conclusions should be undertaken cautiously as extrapolation to the whole population is limited [16,18,20].

4.7.2. Completion Rate

Completion rate (CR) corresponds to the proportion of participants who completed the survey after starting it, independently of the occurrence of missing answers. Similar studies in the prosthetic field reported lower CR reaching to 28% [37,38], others indicated similar values around 70% [39]. CR can be influenced by different factors such as questionnaire length or the use of incentives [14,16,22]. The CR of the pilot phase, 74%, was considered good. This can be due to the nature of the respondents, who are likely to be individuals more motivated or willing to provide help and support for the prosthetics community [14,18,20]. Therefore, although the sample was limited, the survey was most likely completed by motivated participants. Indeed, the presentation message posted online and e-mailed to organizations referred to how respondents’ contribution would provide added-value insights to improve the prosthetic field, which is referred to as nonmaterial incentives [14]. Additionally, based on variables establishing sociodemographic and technical profiles, no specific difference was observed between respondents who completed the entire survey and those who did not [14,16,18,20,21].

4.8. Limitations of the Study

Besides the limitations and constraints mentioned previously in this paper, other hindrances were noted. For example, the cognitive testing phase settings required participants to type their comments and suggestions into large open-ended response boxes at the bottom of each questionnaire page. Those open-ended formats induce high cognitive demands from participants which could reduce the quality of data [14,18,21,22]. However, prospective respondents were aware of the nature of the task beforehand and provided insightful and relevant comments. Therefore, although participants did not communicate their feedback according to the traditional “out loud” methods, their contributions were deemed relevant and adequate [16].

Moreover, although the pilot phase included participants from various backgrounds, some categories were underrepresented. For example, among non-volunteering qualified professionals, few prosthetists, using 3D printing for manufacturing upper limb professionally, enrolled for the study. Therefore, that could bias the collected data and orient the survey toward being more suitable for the maker community. However, experts were not all ‘makers’, and we attempted to generate and rephrase questions items in order to be neutral and inclusive. Questions for both makers and non-volunteering qualified professionals composed the survey. Nevertheless, caution should be kept when conducting the actual survey study, the recruitment phase and the data analysis to explore if any differences would occur. Future studies investigating the target population using this tool should be endorsed by, or at least conducted in collaboration with, governing bodies of specialized learned societies, professional associations and organizations (e.g., NGOs). This approach would allow easier access to their members for higher coverage and enhanced representativity.

4.9. Strengths of the Study

Our work also presented some methodological strengths. Indeed, this survey was developed through a multi-stage process which ensured a higher level of accuracy, relevance and adequacy. Both experts and the population of interest were involved in this development which allowed us to capture the different viewpoints, feedback and suggestions. Therefore, the final questionnaire investigated multiple aspects of the main research goal, as intended.

Additionally, this survey demonstrated acceptable validity evidence levels for use in a population composed of professionals and volunteering ‘makers’ using 3D-printing for manufacturing upper limb prostheses, through a web-based medium. It cannot be extrapolated to different groups in different settings. Future studies will confirm the validity evidence accumulated through this work by testing the construct validity, via methods such as factor analysis. Likewise, reliability of our instrument should be assessed to determine its homogeneity, and its stability over time, by internal consistency and test-retest studies, respectively.

Finally, as 3D printing is increasingly implemented in manufacturing workflows of upper limb prostheses, our survey can help assess local clinical teams’ and institutions’ skilled personnel, expertise and outcomes. This data, collected through our newly developed survey, can facilitate refinement of the manufacturing process, exchange of practical knowledge within and between institutions, and, most importantly, improvement of the quality of prosthetic care provided to recipients. Therefore, our validated instrument can serve as an indirect tool to follow up and improve clinical applications of 3D printing for ULP manufacturing.

The actual online survey research was conducted between 29 September 2023 and 30 June 2024, results are currently under analysis.

5. Conclusions

We developed and validated a survey to be deployed online to investigate the use of 3D printing by qualified professionals and volunteers to manufacture upper limb prostheses. We reported validity evidence of the questionnaire and applied modifications at each development stage. This survey can serve as a guide for further field studies exploring the implementation of 3D printing in the prosthetic field.