A Systematic Review of Design Workshops for Health Information Technologies

Background: Design workshops offer effective methods in eliciting end-user participation from design inception to completion. Workshops unite stakeholders in the utilization of participatory methods, coalescing in the best possible creative solutions. Objective: This systematic review aimed to identify design approaches whilst providing guidance to health information technology designers/researchers in devising and organizing workshops. Methods: A systematic literature search was conducted in five medical/library databases identifying 568 articles. The initial duplication removal resulted in 562 articles. A criteria-based screening of the title field, abstracts, and pre-full-texts reviews resulted in 72 records for full-text review. The final review resulted in 10 article exclusions. Results: 62 publications were included in the review. These studies focused on consumer facing and clinical health information technologies. The studied technologies involved both clinician and patients and encompassed an array of health conditions. Diverse workshop activities and deliverables were reported. Only seven publications reported workshop evaluation data. Discussion: This systematic review focused on workshops as a design and research activity in the health informatics domain. Our review revealed three themes: (1) There are a variety of ways of conducting design workshops; (2) Workshops are effective design and research approaches; (3) Various levels of workshop details were reported.


Introduction
Participatory approaches are common for designing user-centered health information technologies (HIT) [1][2][3]. Participatory approaches encourage including the tacit (and often invisible) knowledge of the users [4] by involving a wide variety of users to ensure all user needs are addressed in the design [5]. Design workshops can be an effective way of eliciting end-user participation by actively incorporating and translating valuable input from design inception to completion. A design workshop can be defined as a codesign environment opportunity for a team to cohesively disentangle a specified problem by undergoing a series of group exercises to either initiate or finalize a design, or to ameliorate an obstacle on an existing design. Workshops can be rewarding by bringing relevant stakeholders together to utilize powerful tools and techniques, culminating in the best possible creative solutions.
HIT Workshops can be utilized as a design or research activity. As a design activity, workshops often focus on solving a HIT design problem and support collaboration between designers and users. For example, a workshop can be organized to modify an existing HIT app (e.g., food tracking app) previously tailored to a specific population to be inclusive of additional populations. As a research activity, workshops can generate and delineate the necessary knowledge to improve the efficacy of a design process and outcome. For Workshops can be particularly beneficial for the design of HIT targeting diverse end-users (clinicians, caregivers, patients), as the workshop atmosphere can cultivate the conception of a design that meets the needs of all users. As health care becomes increasingly distributed among clinical and daily living settings [6][7][8][9][10], the scope of HIT and its user panel have significantly broadened. Workshops can play a vital role in the redesign of current HIT to match the new scope and expanding user needs.
The benefits of a workshop are contingent on involving the correct selection of participants and utilizing the right tools and techniques for the targeted population. Moreover, conceptual frameworks are useful to keep the scope of workshop activities focused. The aim of this systematic review was to identify the tools, techniques, and approaches available. This information should provide guidance to HIT designers and researchers in devising and organizing design workshops. This guidance could help structure the coordination and delivery of a more efficient workshop targeting user-centered design practices. Furthermore, examining the strengths and weaknesses of previous workshop would serve as a lesson to design more novel workshops. For example, at the time of COVID-19 or a similar pandemic, workshops could be held remotely.

Methods
A systematic literature search was conducted by a health librarian in five databases (Medline, Embase, Cochrane Central Register of Controlled Trials, Cochrane Database of Systematic Reviews, and Web of Science) including the following keywords: Participant design, codesign, cooperative design, user involvement, stakeholder participation, health information technology, medical informatics, software, phone app, mHealth, digital health, games, gaming, gamify, telehealth, telemedicine, software design, universal design, computer-aided design, mobile applications, stakeholder, patients, health provider, physician, nurse, therapist, caregiver, user, and family. (Supplementary Material File S1 describes search strategies).
The search process identified a total of 568 articles, with 562 articles remaining after initial article duplication removal. Figure 1 shows the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) [11] diagram for the search methodology. The diagram outlines the number of records that were identified, included, and excluded.
The second and third authors (C.M.S. and O.F.) conducted concurrent independent reviews of the title field by selecting articles that included participatory design (PD), co-design (CoD), and/or were related to HIT, reducing the number to 364 articles. The concurrent independent reviews continued for the reconciliation of abstracts and pre-full-text reviews (i.e., reviewing full text partially, focusing on only predetermined sections), applying inclusion/exclusion criteria, and resulting in 114 and 72 relevant records, respectively. Studies that met inclusion criteria were journal articles and conference papers explicitly mentioning PD or codesign sessions and workshops utilized in HIT. The first author (M.O.) mediated unreconciled article disputes following each screening cycle. A full-text review of 72 articles was completed by the second author (C.M.S.), followed by the extraction of the following elements from the included articles: year, title, author, journal, type of article, country, purpose, target technology, target disease/medical population or service, guiding framework, phases, participant selection, participant profiles, number of participants, workshop duration, number of workshops, activities and approach, tools and equipment, deliverables/results, evaluation, pre and post PD methods, issues reported/weaknesses, and strengths/benefits. Abstracts without a full paper, posters, and duplicates were likewise excluded. The reconciliation of abstract reviews resulted in 250 excluded records, leaving 114 articles for pre-full-text review. The pre-full-text review resulted in excluding 42 articles, and an additional 10 articles were excluded in the full-text review. Sixty-two articles published from 2006 to 2020 met the inclusion criteria. Publications that were included met the following criteria: (1) written in English, (2) author identified, (3) targeted medical/health-related technology, (4) followed a participatory approach to workshop or session design, (5) full-text available electronically, and (6) included participatory design interchangeable terms such as cocreation, codesign, and user-centered design. Eligible publications included records with varied stakeholders (e.g., clinicians, patients, caregivers), and involved adults and/or children participating in HIT PD. Final full-text inclusion articles ranged in publication date from 2006 to 2020 Additionally, studies that summarized a participatory process without clearly detailing the activities and/or actions were excluded. Studies that were excluded were those that mentioned PD sessions/workshops but did not explicitly describe the implementation process, those that were not focused on a health-related field/topic, and/or those that did not differentiate between PD methods. Abstracts without a full paper, posters, and duplicates were likewise excluded. The reconciliation of abstract reviews resulted in 250 excluded records, leaving 114 articles for pre-full-text review. The pre-full-text review resulted in excluding 42 articles, and an additional 10 articles were excluded in the full-text review. Sixty-two articles published from 2006 to 2020 met the inclusion criteria.
Publications that were included met the following criteria: (1) written in English, (2) author identified, (3) targeted medical/health-related technology, (4) followed a participatory approach to workshop or session design, (5) full-text available electronically, and (6) included participatory design interchangeable terms such as cocreation, codesign, and user-centered design. Eligible publications included records with varied stakeholders (e.g., clinicians, patients, caregivers), and involved adults and/or children participating in HIT PD. Final full-text inclusion articles ranged in publication date from 2006 to 2020. Additionally, studies that summarized a participatory process without clearly detailing the activities and/or actions were excluded.

Results
Sixty-two publications  from the years 2006 to 2020 were included in the review. These publications are a result of 59 unique studies. Included publications were predominantly journals (n = 57), but conference papers were also included [36,55,60,63,71] (n = 5). Studies were conducted in six continents (Europe, 34; Oceania, ten; North America, eight; Asia, four; Africa, one; South America, one). Study location was not reported in four studies. Included studies focused on consumer-facing health technologies (e.g., app or website; n = 39), clinical health information technologies (e.g., electronic health records; n = 8), technologies that involve both clinicians and patients as users (e.g., telehealth technologies; n = 12) and other technologies (n = 3). The studies represented a wide variety of health conditions (e.g., mental health, heart failure, asthma, dementia, kidney transplantation), life span (e.g., youth, adult, elderly), and type of users (patients, caregivers, clinicians). Workshops could be accomplished as a single workshop (n = 10) or as a series (n = 52). The highest number of reported workshops in a single study was 20 [26]. Workshops could include homogeneous (e.g., only clinicians) or heterogenous (e.g., clinicians and patients) groups. The number of participants (in each workshop) varied from 4 [17,19,23] to 47 [52]. The duration of a workshop varied between one hour and over one day.
The included studies focused on a wide variety of target diseases, settings, treatments, or populations. In three of the studies, the target population consisted of clinicians and other stakeholders as users. In four of the studies, the target population focused on the caregivers of patients with various conditions. The remaining 55 studies focused on various conditions. Antibiotic management was the focus in three studies. Each of the following conditions was the focus in two included papers: Type I Diabetes, asthma, adolescent mental health, HIV, chronic illness (general), rheumatoid arthritis, heart disease, and mental health (general).
Many workshop activities and deliverables were reported. These activities include: The grouping of workshop activities is not necessary mutually exclusive. Moreover, some of these activities may overlap. Primary deliverables of these workshops were: • Prototypes/mock-up models (paper-based or wireframes) • Research themes • A list of recommendations/solutions/ideas • Product/technology evaluation Table 1 highlights the abstracted information from each study including authors, publication type, participants, activities, and deliverables. Participant narrative representation of thoughts, brainstorming, presentation on gamification approaches, idea notetaking on sticky notes, whole group discussion, Sketching, presentation of substance images.
Audio recording, prototype sketches, common element identification from data Some workshops were performed as stand-alone design and research activities, while others included workshops as one part of a multiphase design and research process, encompassing other data collection methods such as interviews, focus groups, or observations. Workshops included in a multiphase project utilized preceding activity outcomes to preface the current workshop. Likewise, following an iterative process, workshops provided input to guide subsequent design activities, unless the design workshop was the concluding activity. Some workshops were guided by conceptual or methodological frameworks. Reported conceptual frameworks included behavior change theories [12,29], perspectivist theory [14], democratic dialog theory [14], user experience framework [24], Ottawa decision support framework [26], medical research council framework [30], theoretical framework of social justice [38,39], self-determination theory [44,47,54], IDAS framework [44], nudging theory [44], and cultural historical activity theory [46]. Methodological frameworks included cooperative inquiry [13], appreciative inquiry [18], hermeneutics philosophy [19], ethnography [22], double diamond design process [35], action research [33,48,56,58], PIC-TIVE [55], and cooperative design [61]. Some workshop studies included in this review were guided by custom frameworks [32,40,59]. The dominant data analysis method was qualitative thematic analysis. Other inductive qualitative methods were also reported. Seven articles reported the use of multiple or mixed methods [15,52,57,67,68,72,73]. The majority of the workshops were audio and video recorded.
Although uncommon across studies, some distinct phases within or between workshops were identified. One study [22] organized two workshops and distinguished them as "creative" and "technical" workshops. Another [28] identified the following phases: exploration, ideation, reflection, and implementation. Jeffery et al. [32] identified three phases: priming, designing, and debriefing. Yet another [35] included the following phases: discovering, defining, developing, and delivering. One [59] identified three phases: scoping, participatory design, and prototyping. One study [61] identified four phases: user needs assessment, low fidelity prototyping, high fidelity prototyping, and functional prototyping. One [17] study identified three phases: Discovery; Evaluation; Prototype. Multiple studies used the phases from inception to implementation consistent with a system development life cycle: identification of needs, development/prototyping, and evaluation.
The level of workflow details reported in the included studies varied. Of the 62 studies, 11% (n = 7) described a formal evaluation, 60% (n = 32) a guiding framework, 63% (n = 39) the duration of the workshop, and 85% (n = 53) the participant selection as being critical components when planning a workshop but were not consistently reported.
Only seven [33,40,61,63,66,68,72] publications included data on the evaluation of the workshop. However, only five of them [61,63,66,68,72] reported a deliberate effort for evaluation. These efforts include asking verbal feedback within the whole group and/or a questionnaire. The other two collected feedback informally or extracted workshop evaluations indirectly from qualitative data that was originally collected for the design of focus. In the provided feedback, the evaluation of workflow could be intertwined with the evaluation of the design workshop focus. Table 2 shows a summary of the workshop evaluations.

Authors Evaluation Summary
Jessen et al. 2018 [34] "Overall, we can conclude from the vast variety of user inputs that the workshops were successful in generating new and creative concepts and ideas for mHealth tools . . . The participants and the facilitators alike found the workshops to be both productive and enjoyable. In fact, when getting feedback at the end of the first workshop, all 3 groups wanted to spend more time on the next workshop".
Lee et al. 2018 [40] "Patients, health care professionals, and managers confirmed the relevance and value of the overall concept as well as the organizational setup".
Scandurra & Sjölinder, 2013 [61] "At the concluding workshop the participants described their overall experiences, both with respect to the latest version of the device and with respect to the overall impressions about the project".
Swallow et al. 2016 [63] "The workshop concluded with a general discussion in which the research team summarized the overall findings and gave participants the opportunity to provide any additional feedback about the problems and solutions, as well as their experience of taking part in the participatory design workshop".
van Besouw et al. 2015 [66] "The method used to collect feedback during the trial (an online survey completed by users at the end of each session) also resulted in 'honest' feedback that captured the users' immediate and unrestrained reactions to the session and software".
Warren et al. 2019 [68] "Quantitative evaluation questionnaires were undertaken by 43 participants. Non-responses to individual questions were excluded from analysis. Responses from the evaluation questionnaire indicated that participants found the workshop process used for this project to be enjoyable, useful and interesting. Participants indicated that this workshop stimulated their interest in being involved in future healthcare design work, further emphasizing the potential value of the methods described in healthcare research and development".
Xu et al. 2020 [72] "The workshop evaluation comments indicated that the co-design workshop was successful in creating and generating new ideas and content for smartphone app development".
In summary, this study resulted in three important findings. First, the list of activities in conjunction with Table 1 highlighted that designers and researchers have various approaches to conduct workshops. Second, although the included studies reported rich findings overall, a lack of consistent evaluations prevented the ability to compare the effectiveness and appropriateness of workflow activities for different purposes. Third, workshops could possibly be utilized more frequently and fastidiously if the many workshop details are disseminated. These findings led to three themes as described in the discussion.

Discussion
This systematic review focused on workshops as a design and research activity in the domain of health informatics. This type of participation (i.e., workshops) presupposes the need to give voice to the users, rather than users just serving as a source of information or observation. Our review revealed three themes: (1) There are a variety of ways of conducting design workshops; (2) Workshops are effective design and research approaches; (3) Various levels of workflow details were reported. These themes can provide important insights on HIT development by providing a guidance to researchers and designers to operate and disseminate design workshops in a systematic and effective way.

There Are a Variety of Ways of Conducting Design Workshops
Designers and researchers have numerous workshop activity options as listed in the results section. Sanders et al. [74] identified three classes of workshop activities: (1) making tangible things; (2) acting, enacting, and playing; (3) talking, telling, and explaining. The workshops on the design of health informatics interventions have utilized approaches from all three classes. The selected studies do not provide sufficient details to assess whether any workshop activity is better than another, or if a specific workshop activity is superior (or inferior) in exploring a target technology design, participant characteristics, or other factors. However, we argue that the selected approach should be congruent with (1) the design problem, (2) participant characteristics, and (3) available resources. We also argue that a combination of multisensory eliciting activities (visual, auditory, tactile) within the same workshop will more thoroughly engage the participants, allowing for better absorption of material and understanding. This review is useful in terms of balancing tradeoffs between diverse workshop activities and combining multiple complimentary activities to further strengthen participant acumen and workshop efficaciousness.
Various participants may be more productive using different activities. For example, select participants can be more productive in making tangible things (e.g., prototyping), some participants can be more effective in acting and playing (e.g., role playing); and likewise, other participants can offer more dynamic input in talking and discussion (e.g., brainstorming) activities. Therefore, employing multiple types of workshop activities can best utilize diverse participant characteristics, therefore yielding an overall more productive workshop. Moreover, various participant characteristics such as disability status, developmental level, language barriers, and cultural sensitivities should be accounted for when selecting workshop activities.
Various conceptual and methodological frameworks have been used in organizing workshops. However, the contributions of the highlighted frameworks to the design process were implicit. Although there are frameworks specific to workshops (e.g., [75]) the adoption of these frameworks by workshop designers and researchers is ambiguous. Future studies should focus conceptual and methodological frameworks that link the research questions and design objectives to the workshop activities and deliverables.

Workshops Are Effective Design and Research Approaches
Participatory design suggests involving users throughout the entire design process. Design workshops should be a part of the broader participatory design process, in that they are emblematic of the values of this overall philosophy, creating a space in which designers and users can work together collaboratively to formulate design solutions. The success of a design workshop can be measured by its ability to bring out invisible or tacit knowledge. This study provides designers and researchers a variety of options to facilitate the capture of this hidden knowledge. As the workshops are more systematically evaluated, which activity is more suitable for different purpose and context will be better understood.
None of the included studies cautioned against identified weaknesses or lack of effectiveness and efficiencies of the design workshop. However, we present gaps both in terms of how such workshops have been evaluated and assessing the effectiveness of different types of strategies for different contexts.
Design workshops can improve the design of interventions that affect various subdomains of health informatics. However, these workshops focus on varied aspects of design, not necessarily examining implementation issues. Implementation workshops can complement design workshops and support adoption and sustainability of the informatics interventions that were developed within design workshops.

Various Levels of Workshop Details Were Reported
Replicating successful design workshop practices depend on disseminated details. The papers reported many details but lacked consistency across studies. Moreover, there are some details that were not reported by any studies. For example, all 62 studies included the overall number of participants in a workshop series but did not consistently report on the exact number of participants for each workshop, expertise, and demographic information of participants. Moreover, none of the studies provided full details on the selection of sample; how diversity, equity, and inclusion (DEI) issues (e.g., inclusion of vulnerable groups) were addressed; and any potential sampling bias.
We argue that any study that employs and reports on design workshops should report: the purpose, number of participants, selection of sample, how DEI issues are addressed, workshop duration, workshop agenda, details of conducted activities (e.g., activity identification, selection criteria), guiding framework(s) specific to the workshop (if applicable), expected and actual deliverables, and workshop evaluation process and results.

Recommendations for Conducting a Workshop
Based on our review, we developed the following recommendations for researchers and designers for organizing design workshops:

•
The preparation/planning stage is critical for the success of the project. It is important to be cognizant of differing levels of aptitude. In some cases, participants may benefit from preworkshop technology education (e.g., brochures/pamphlets, emails) to elicit a better understanding of the technology or concepts novel to the participant, and subsequently support workshop participation preparedness.

•
Well defined research/design questions/objectives should be the main drivers of other decisions related to organizing the workshop: participants, technology or intervention being designed, conceptual and methodological framework used, workshop activities employed. There should be a congruence among (1) research/design questions/objectives, (2) sampling, (3) selected activities, (4) technology/intervention that is being designed, (5) guiding framework, and (6) available resources. • Sampling should reflect a wide range of user needs. Vulnerable populations should particularly be considered.

•
An introduction may include an ice breaker/warm up exercise to establish commonality between participants and cultivate a trustful atmosphere with the facilitator. • Workshops could benefit from a facilitator/moderator and a dedicated individual who will document the workshop activities and outcomes by taking notes or audio/video recording. Facilitators should be mindful of potential power imbalances in varied stakeholder groups that can result in a dominating one-sided perspective. • A synergistic creating process can benefit from a relaxed environment, allowing for participants to freely move about and take breaks as needed. Providing coffee, snacks, or meals and encouraging a flexible atmosphere may encourage willingness for continued participation in an often time-intensive proceeding.

•
Utilizing diverse activities will more likely provide better engagement and input, particularly for the heterogenous groups.

•
Workshop conclusions should include a formal evaluation (e.g., an exit questionnaire, brief interviews, providing a visible note taking board to post feedback throughout the workshop) to provide structured feedback when the workshop findings are disseminated. If a formal evaluation is not feasible, the workshop may include debriefing or collective reflection to discuss participant experiences attained from design activities. This exercise dually acts as an informal evaluation of the successes and/or areas in need of improvement and validates the importance of participant contributions to the participant themselves.
These recommendations are the result of critiquing the literature and discussion with the research team. While this is not a comprehensive list of approaches, it provides our recommendations from the 62 studies. Included studies reported on in-person workshops. However, our recommendations potentially hold for conducting workshops remotely using computer mediation due to pandemics (e.g.,  or any situation that makes gathering impossible or impractical. Novel workshop activities specific to or more effective for remote workshops may be needed.

Limitations
A meta-analysis of study results was not possible because of the heterogeneity of the design and study results. We therefore provided a descriptive analysis of publications. We acknowledge that the retrieved studies do not necessarily represent a comprehensive list of all HIT workshops reported in the literature but list studies from the scientific literature returned during our search and that met our inclusion criteria.