Eliciting Co-Creation Best Practices of Virtual Reality Reusable e-Resources

Abstract

Immersive experiential technologies find fertile grounds to grow and support healthcare education. Virtual, Augmented, or Mixed reality (VR/AR/MR) have proven to be impactful in both the educational and the affective state of the healthcare student’s increasing engagement. However, there is a lack of guidance for healthcare stakeholders on developing and integrating virtual reality resources into healthcare training. Thus, the authors applied Bardach’s Eightfold Policy Analysis Framework to critically evaluate existing protocols to determine if they are inconsistent, ineffective, or result in uncertain outcomes, following systematic pathways from concepts to decision-making. Co-creative VR resource development resulted as the preferred method. Best practices for co-creating VR Reusable e-Resources identified co-creation as an effective pathway to the prolific use of immersive media in healthcare education. Co-creation should be considered in conjunction with a training framework to enhance educational quality. Iterative cycles engaging all stakeholders enhance educational quality, while co-creation is central to the quality assurance process both for technical and topical fidelity, and tailoring resources to learners’ needs. Co-creation itself is seen as a bespoke learning modality. This paper provides the first body of evidence for co-creative VR resource development as a valid and strengthening method for healthcare immersive content development. Despite prior research supporting co-creation in immersive resource development, there were no established guidelines for best practices.

1. Introduction

The last century has seen an explosion in all aspects of human knowledge. Medicine has also experienced the same effect with medical data- doubling almost every two years from four decades ago [1]. This trend was exacerbated in the following years with projections in 2011 mentioning medical data doubling the rate of fewer than 3 months. The data is not the same as useful applicable medical knowledge. However, the educational medical content that an aspiring healthcare professional needs to consume before being able to practice safely their craft is enormous. Compounding the effort needed, medical knowledge is critical in nature, and requires exact theoretical background and good tacit knowledge and experience [2]. In that context, the tried-and-true methods of study and practice are constantly challenged.

In that environment, immersive experiential technologies find fertile grounds to grow and support medical education. Virtual patients, chatbots, and Virtual, Augmented, or Mixed reality (VR/AR/MR; under the umbrella term eXtended Reality XR) have proven to impact both the educational and the affective state of the healthcare student’s [2,3,4,5] increasing engagement. Additionally, taking cues from different disciplines such as physics and chemistry educational content has proved effective- with students able to visualize abstract laws in an almost tangible way [6,7,8]. The sensory immediacy of these technologies can produce an intuitive anchoring of the core material to the learner and facilitate a paradigm building based on sound scientific data. This paradigm-building leads to robust, deep topical knowledge and reduces the possibility of establishing or maintaining conceptual errors [9]. In the medical field, there is already a significant body of immersive content. In 2019 the Royal College of Physicians identified Virtual Reality (VR) as a “…powerful educational tool for defined learning objectives…” listing several applications in medical education and surgery [10], p. 181.

Resource-wise, the global virtual reality market size was $3.10 billion in 2019 and is expected to reach $57.55 billion by 2027, while the healthcare market is equally optimistic with its 2018 size being $1.56 billion- expected to reach $30.40 billion by 2026 [11,12]. These numbers demonstrate the potential for immersive content in healthcare education, but they also indicate the significant resource overhead that such content incurs for development, testing, and deployment. A 2019 study identified a cost of $106,387.00 for designing and implementing a VR training exercise for hospital staff regarding evacuation procedures [13]. While these costs are feasible, given the reusability of the VR content, which drastically reduces its per trainee cost over time, the volatility of medical knowledge quickly makes such resources obsolete and reduces their reusability potential. In such an environment, a methodology that would further reduce implementation and design costs, as well as distribute the weight of resource development, would be useful.

1.1. Co-Creation as Method towards Improved Sustainable Development

Co-creation provides dynamic assistance to the product design process for new item improvements [14,15] in the context of co-production. Co-production requires “cooperating with clients” [16,17] or even an involvement in the product/service configuration process [18,19]. Client investment can be expressed as an assistance to the outskirts of a company’s workflow [18], or as a complex, central component, through the use of the company’s knowledge and data sharing and learning [20,21]. Co-creation was also identified by a client association that demonstrated mutual physical, mental, and business practices, as well as access to the various masteries [22]. More specifically, co-generation has been described as a scheme of acts carried out by actors (financial, social, and other) involved in value chain networks [23,24]. It is carried out by coordination [25], exchange [26,27], and the integration of common assets into the value generation process [28]. As clients devote assets through co-creation forms, the primary stakeholder (firm, creator) achieves both the demonstrated satisfaction of the client’s request and the leveraging of the client’s expertise in the expansion of the firm [29,30]. Co-creation also makes it possible for the artistic process to be divided up, but the features of co-production remain within the primary stakeholder [31]. This process helps clients to be completely engaged in the co-production process [32,33] with some studies going so far as to recognize the importance in mutuality, receptivity, and non-hierarchical relations [21,29] as a co-production feature. This kind of detailed understanding of the co-creation process has led research [34] to identify knowledge distribution as one of the key drivers of co-creation effectiveness.

Even though there are publications supporting and technically facilitating the co-design and co-creation of immersive resources (e.g., cf. [35,36]), deciding on a definitive set of best practice guidelines pointing to co-creation as an improved content development methodology is not a proven conclusion.

Health policy often dictates protocols and helps healthcare stakeholders and other providers make evidence-based decisions about important issues. However, if recommended actions are conflicting, ineffective, cost-prohibitive, or result in questionable outcomes, they warrant review [37]. Healthcare policymakers are at the intersection of policy and practice and are naturally positioned to address gaps in healthcare policy and to conduct such research. Health policy guides many decisions that specialists make about healthcare and education. Well-crafted policy has implications for ensuring timely and accurate guidance for stakeholders to deliver effective medical education. Healthcare stakeholders seeking guidance on integrating virtual reality (VR) resources for healthcare training may not find recommendations that align with their daily clinical experiences. This can prompt them to reassess prevailing policy in specific contexts or with unique populations. As policies profoundly impact education outcomes, subsequent patient care, and the overall health of populations, policy analysis is a critical research tool.

1.2. Implementing the Eightfold Policy Analysis Framework towards a Set of Best Practices for Co-Creative VR Content Production

Thus, based on outcomes from the CoViRR project and other sources, this is the first attempt to apply the eightfold policy analysis framework first proposed by Bardach and Patashnick [38]. It is commonly applied in policy and administration research as well as in public health. Bardach’s eightfold policy analysis framework includes the following steps. (1) Defining the problem: This crucial steps provides the rational and a sense of direction on how to assemble the evidence; (2) assembling evidence: identify the relevant background, trends and literature through various resources; (3) constructing alternatives: forming alternatives course of actions or strategies addressing the problem; (4) selecting criteria: explores how alternatives can be measured and evaluated identifying the effectiveness of the policy; (5) projecting outcomes: projecting all the outcomes or impacts that you or other interested parties might reasonably care about considering how realistic or viable each outcome is; (6) confronting trade-offs: considering the trade-offs between and within each policy alternative, in terms of the criteria by which they can be evaluated, and the criteria themselves need to be weighted; (7) decision-making: based on the previous steps deciding on the final policies; and (8) sharing the results of the process: telling your story in the form of a narrative or a set of recommendations. In

Table 1. Eightfold policy analysis framework for VR educational content creation challenge.

Define the problem	There is too little access to quality VR resources for medical learners
Assemble the evidence	Literature research on experiential medical education Commercial entities in immersive health education
Construct alternatives	Maintain the standard development pipeline Educators and technologists co-design resources Crowdsource resource design
	Buy-in services from companies
Select the criteria	Efficiency Educational Efficacy Acceptability from users/learners
Project the outcomes	Assess impact on curriculum Assess speed of resources development Assess quality of resources development Assess cost of resources development Assess the level the resources are tailored to the need of the learners
Confronting the trade-offs	Digital skills overheads Systemic changes in the academic world
Decision Making	Even though co-creative VR resource implementation approaches have some advanced digital literacy overheads they are an effective pathway to the prolific use of immersive media in medical education Co-creative resource development should embed quality assurance processes that maintain high level of technical and topical fidelity, and certifies resources against specific educational use cases, target groups, or even discrete episodes of learning Participatory design methods should be introduced to educators and learners always in conjunction with a training framework in the supporting and enabling technologies that are required for them
Sharing the results of the process	Create a set of recommendations for co-creative resource development pipelines

, the eightfold policy analysis framework is broken down to illustrate how this more comprehensive approach can similarly illuminate decision-making processes for a co-creative approach in VR medical education participatory design.

2. Materials and Methods

2.1. Define the Problem

Immersive technologies which include VR can allow improvement and tracking of performance indices, and acceleration of cognitive functions in healthcare providers [39,40], refined decision-making in emergency scenarios [41], and improve preparedness from procedural and environment familiarization [42,43,44]. Face-to-face high-fidelity scenarios can be transformed into interactive and immersive packages [45], and trainees can interact remotely with staff in real-time using augmented and virtual reality systems [46]. Indeed, most elements of healthcare can be transformed into VR. Techniques, behaviors, theories, protocols, communication, equipment, and other components can be simulated. To aid in this there are several guidelines available to follow that have been created by XR companies to help develop learning resources [34,47,48]. They provide advice for best practices in technology and operations and include case-based steps of the experimental setup to help readers/developers understand how best to test VR with existing training/ educational resources [49]. Some companies and institutions [47] perform the following steps in streamlining the content during development and being fit for purpose:

• Mapping user experience [32,33,34,48]
• Developing the most prioritized resources—prominent level of detail and features may not be needed to reach the training goal [21,29,50]
• Understand user abilities and assert the usage schedule/integration.

However, these recommendations are optimal under certain conditions and typically do not focus on the initial design of learning resources. Therefore, many unsuccessful or problematic results can occur when learning and research institutions attempt to create VR material. The collaboration of VR developer support and internal research groups creating an application for experimental/research purposes can have missing skillsets from the absence of pedagogical design and overlook the inclusion of stakeholders in the creation process. End-users are key in asserting the research priorities, and without their inclusion key issues may be overlooked from the start. Perspectives can be technically orientated but without co-creation, underlying problems may not be addressed. There is a large gap between the needs of healthcare professionals, best practice with inclusion from stakeholders, and steps for utilization of available resources during design and development. A review of current methods of developing virtual reality reusable e-resources is presented in this chapter, including the benefits that co-creation methods could bring. Reviewing the current state of practices allows investigation towards a streamlined approach that can be shared with a global audience for their own resource development.

2.2. Assemble the Evidence

Step 2 instructs to gather current evidence of the landscape of development. The standard pipeline involves in-house development of VR resources using topical experts for consultation and technology experts for design and implementation, with external assistance where required. Institutions can gain financial, intellectual, and research benefits from training their researchers and creating learning resources in-house. Many commonly used software programs are free or low-cost, enabling institutions to create resources in-house for many years with little expenditure. This also helps to understand and keep up to date with XR development changes. There may be a steep learning curve initially, but progress increases once this initial barrier is overcome. Topical experts can provide valuable insights, as they are in close proximity and can validate the content of learning resources. The combination of internal development and topical expert presence is the de facto methodology, as evidenced by recent literature reviews [51].

Nevertheless, this can create lag in project progress as researchers take time to refine their skills. Indeed, there is a need for assistance with complex parts of a system, and the rapidly changing landscape of XR software and components means that sourcing professional consultation will be required. As a realistic compromise that can negate the high price of private companies with the benefits of in-house design, collaboration with sole designers or smaller companies are a growing occurrence. An external consultant can guide researchers with minimal expenditure but be supported when required. There are a growing number of such companies as guidance to institutions. Companies can allow researchers to hire equipment alongside assisting them to develop applications themselves [52,53]. Few guidelines include stakeholders to the degree of closeness required to optimize pedagogy and reusability. Universities have the choice to develop their own content or pay 3rd parties, and many collaborations have occurred [54]. But each application is made differently, with different techniques, user inputs, budgets, goals, and time. A standardized and efficient process is required that can streamline all processes to save cost, time, development effort, and be optimized by, and for, the end user.

2.3. Construct Alternatives

Step 3 of Bardach’s framework then involves the construction of policy alternatives. The process involves the identification, with a critical eye, of possible courses of action in the specific topic or challenge. In this framework, specific mention is made of the “status quo” policy, that is the maintenance of the current policy or standard of practice. This is a necessary methodological precaution against policy changes that would disregard efficient current practices. In the case of VR resource development, a series of approaches have been identified from the literature and common practices. These along with a brief description are outlined in

Table 2. Policy alternatives for immersive resources development.

Policy	Description	References
Maintain the standard development pipeline (status quo)	In-house development of VR resources using topical experts for consultation and technology experts for design and implementation, with external assistance where required	[13,47,55,56,57,58]
Alternatives
Educators and technologists co-design resources	Educators and or learners collaborate to design the educational material. Technologists provide infrastructure for collaborative VR resource design and implementation. (e.g., user-friendly editing environments, new asset creation, etc.)	[35,36]
Crowdsource resource design	Similar to the co-creative approach, the design process is distributed amongst educators. Technical implementation is conducted by interested members of the technologists’ community through social media engagement and smaller, usually community-based or “in-kind” reimbursement.	[59,60,61,62,63,64,65,66,67,68,69,70,71,72,73]
Buy-in services from companies	Similar to status quo. Core difference is that technical implementation is conducted through a different business entity, which elicits design requirements from the educational institution

.

2.4. Policy Alternatives

What follows is a brief outline of their features and some critical “observations from the field” as they appear in the literature and from the CoViRR experience.

2.4.1. Maintain Standard Development Pipeline

This method, being the standard of practice, has the largest coverage in the literature. Such example studies have used or explored this standard production pipeline [47,55,56,57,58]. A recent study has even conducted a cost analysis of VR vs physical presence educational episodes [13]. Results demonstrated that VR costs were 43% greater than the real-world exercise cost. This difference in costs led the study to the conclusion that the per trainee cost would only become effective if this single resource was utilized and maintained for over 3 years, unchanged and with the same demand from the learners.

2.4.2. Stakeholders’ Co-Design of Educational Resources

The co-creation avenue has been proposed as a valid content development and deployment pipeline. The co-creation concept emerged from marketing and more specifically from product design. Value co-creation (VCC), as it was originally termed, was the process of identifying an item’s value offer through client participation rather than the standard statistical surveying avenues [34,48]. This process allows clients to be fully engaged in the co-production process [21,49], with some studies going as far as to identify value in mutualism, receptiveness, and non-hierarchical relations [29] as an element of co-production. This kind of extensive interpretation of the co-creation process has led research [34] to expose as one of the main factors of co-creation efficacy the sharing of knowledge. Literature has proposed content development pipelines and technological innovation approaches to facilitate digital content co-creation from non-technology experts (e.g., educators and learners). A much simpler approach was followed in CoViRR. Utilizing stakeholder engagement in co-design storyboarding and collaborative implementation with co-creative input from the education experts themselves, nine re-usable VR resources have been effectively created at a fraction of the cost of the standard VR development pipeline. Another strength of output comes from ProteinVR [74], who used co-creation and stakeholder inclusion by actively involving users in the software development process through feedback and bug reporting. The open-source nature of the software also allows for contributions from a wide range of stakeholders, including researchers and educators. Furthermore, the 360ViSi [75] ERASMUS+ project is an international collaboration among several universities and companies in Europe. It explores co-creation methodologies [76] using 360° video technology to increase access to low-cost simulation training in health education, and to stimulate the flow and exchange of knowledge between higher education and enterprises.

2.4.3. Crowdsource Resource Design

Initially ascribed by Howe in 2006 [59], crowdsourcing entails an open call of a function from a company or institution to an undefined size network of individuals. Crowdsourcing development has been successful in business and industry yet utilization by educational institutions in outsourcing functions or tasks from education professionals has not had a comparable impact [60]. Technological and internet-based tools allow accelerated idea generation, micro-tasking, and problem-solving. This process has suitability for educational crowdsourcing as a mechanism for collaborative learning in addition to the subsequent crowdsourced content [61]. Medical education has similarities with crowdsourcing as they both have an openness to delivering resources by facilitating collaborative methodologies. Dissemination of materials amongst educators has been shown to be an effective approach to developing, creating, and sharing digital teaching resources [60,61,62,63]. Online content, appraisal of quality, and development of new material in medical education initiatives have increased. Weld et al. [64] suggested that crowdsourcing can solve problems of delivering large-scale virtual education, in addition to modifying an online crowd to be its own support for learners. Therefore, crowd-sourced processes in the design of educational material may accelerate this potential while providing an effect on stakeholders via innovative outputs and collaborative strength.

Examples of this include online crowd utilization during COVID-19 [65] to produce a framework for structured, crowdsourced innovation in healthcare R&D. This took the form of a hackathon, where individuals were convened to crowdsource solutions around a core set of predetermined challenges in a limited amount of time. Furthermore, online crowd recruitment and distribution of educational videos allowed each user to watch and summarize a video segment [66]. Users not only watched the educational videos but also collectively modified the multi-language caption data for all videos, improving their future use for foreign viewers.

Yet, large-scale learning repositories for healthcare are not widely accessible. A 2019 narrative literature review on the role of crowdsourcing to educate health professions suggested no publications involving its role in healthcare curricula design [67], instead a handful of instructional design [68], surgical recruitment [69], and surgical crowdsources assessments [70] projects were noted. This may be due to a lack of guidelines towards crowdsourcing resource design of resources, especially with novel immersive technologies of VR and AR. There are several examples of web-based VR experiments using crowdsourcing [71] however none yet include tasks in the co-creation of educational medical resources. Examples include a ‘VRChat’ [72] application able to implement collaborative VR user studies and future use could allow the co-design of digital resources with VR headset crowds [73]. Additionally, [77] created a cloud-based VR platform that enables crowdsourcing of HRI experiments, and initially tested in a competition where users interacted with a virtual robot through VR devices and the Internet.

2.4.4. Buy-in Services from Companies

There are several hundreds of XR companies creating products for healthcare solutions [78]. The current issue, however, is that the methods used to create XR products are not clear, and a standardized approach has not yet been developed due to the infancy of VR and AR development in healthcare. Stakeholder inclusion is sporadic, and co-creation methods are not widely used. Companies utilizing VR-related technology must successfully navigate a plethora of factors before attaining clients, creating resources, and receiving the financial, research-driven, and social benefits from collaborations. Due to the volatility of research requirements, technology improvements, and availability, each company takes a different approach to collaboration and creation.

However, if collaboration is successful there are many advantages to allowing dependency on external companies in creating products.

• They have the capacity to create custom packages for medical learners specific to the intended learning target.
• They have rapid development and turnaround, with the ability to support during and after resource implementation.
• They have expert insight which may improve the initial plan and output of the learning resource.

There are many disadvantages to sourcing external help in the development for the learning institute. Primarily, the high cost of hiring equipment, staff, processing, and long-term support means for many institutions this option is heavily restricted. Co-creational methods are not the central focus when incorporating private companies as fully developed plans are typically sought to allow immediate resource creation. This means participatory inclusion may be limited/affected during the main development processes. However, there are case examples where companies have successfully collaborated with educational institutions to create effective VR resources. Examples being Nanome [79]—a VR molecular design tool that allows for natural collaboration and interaction with any molecular structure; and Osso [80] VR training and assessment platform that enables healthcare professionals to train on any procedure.

2.5. Select the Criteria

Selecting the criteria is step 4 in Bardach’s framework and explores how the impact of alternatives can be measured and evaluated. This is crucial for determining a policy’s effectiveness. Criteria should be established based on the scope of impact that a policy is aimed to address. It should also be based on established research to ensure the validity that these criteria should carry, subsequently validating the selection process. When exploring the criteria for selecting the most appropriate medical VR resource creation methodology, the evaluation strategy of CoViRR came into focus. There, several axes of efficacy and impact have been identified, which, subsequently, have evolved into the relevant criteria subsequently presented.

• Financial Efficacy. This is an aspect focusing on the economy of resources both human and technological. Such an evaluation utilizes a cost-benefit analysis considering several factors (c.f. the analysis conducted in [13]). These factors are (a) actual costs accrued, (b) audience size and re-usability/repurposing ability of the resources, and (c) the validity of the resource across multiple curricular instances (i.e., the ability for the resource to be used “as is” in several consecutive or not semesters of training.
• Technical efficacy and acceptance, focusing on optimization of the VR experience. This includes usability testing and user experience optimization. Such an effort uses instruments such as SUS [81] and a modified TAM instrument [82]. It must be noted that technical evaluation in the context of CoViRR does not involve VR device evaluation. Thus, its focus stays in the flow and implementation of the resources themselves in multiple VR platforms if this becomes possible.
• Pedagogical efficacy and acceptance, focusing on validating the resources as instruments in specific educational episodes. This includes knowledge retention assessment as well as student engagement and acceptance of the resource. Pedagogical evaluation involves both qualitative and quantitative methods and aims to explore the value of the created resources as both a knowledge transfer vehicle, as well as a summative assessment instrument. Validated methods for such an evaluation includes OSCEs and e-OSCEs [83].
• Stakeholder acceptance/Curriculum feasibility evaluation, focusing on the capacity of the resources to be integrated into medical curricula without extensive institutional or technical overheads. This axis is more of an integrative, meta-evaluation axis, considering learner and educator digital literacy, technology readiness levels of the supporting institutions, as well as financial and administrative considerations. As such, it is best explored using qualitative instruments like semi-structured interviews, heuristics, and focus group sessions [84,85], which can best capture tacit, along with formal, aspects of curriculum integration.

3. Results

3.1. Project the Outcomes

Step 5 in the approach, is, many times, considered the most difficult to explore. The challenge of making realistic projections in each specific criterion selected in the previous step is compounded by the need to assess synergies and cross-pollinating effects between the criteria, as they impact the whole. In that context, this step of the method requires the identification of aspects that would be impacted by the proposed alternatives. In the context of medical education content creation and delivery, we identify the aspects below.

• Assess impact on curriculum. The content creation methodology affects the curriculum both implicitly but also directly. Implicitly, changes in the development and deployment methodologies may produce variations in the quality of the content delivered. For these cf. the third section of this step “Assess quality of resources development”. Direct impact to the curriculum from the development process comes from the tacit training that co-creation provides to the personnel that takes part in the process. It is not uncommon for students and learners to gather novel insights into the subject matter after the relevant co-creation workshops. In that context, co-creation and crowdsourced approaches may lead to a more direct positive curriculum impact.
• Assess speed of resource development. Speed of resource development is the core advantage of the co-creation and crowdsourcing approaches. Distributing the design load and having immediate access to high-quality feedback provides concrete advantages compared to the other methods.
• Assess quality of resource development. Co-creation and crowdsourcing of educational resources require rigorous quality control provisions to maintain the required level of quality for curricular integration. Company buy-in services on the other hand have an administrative overhead similar to that of the current methodology, given the fact that developed resources have to go back and forth between the developer and the client for QA and approval.
• Assess cost of resource development. Together with the speed of resource development, the cost is also an aspect that benefits from co-creation and crowdsourcing. Overheads for iterative design are alleviated since the requirements elicitation comes interactively from the stakeholders themselves without costly pre-production overheads and a subsequent reduction of iterations required for a finished resource.
• Assess the level the resources are tailored to the need of the learners. In that aspect, participatory design methods come ahead even from crowdsourcing because the collaborative creative process inherently adapts resources at the most fundamental level, design. As such, adaptability to the learner’s needs is best served through this methodology.
• Assess the repurposing ability of resources developed in each methodology. Both crowdsourcing and co-creation approaches leverage heavily open access technologies and repurposing architectures. In that context, resources developed through these processes have increased re-purposing capacity than those developed through the standard development pipeline or the buy-in approach from companies.

3.2. Confronting the Trade-Offs

In step 6, confronting trade-offs, we need to consider the trade-offs between and within each policy alternative. The trade-offs need to be considered in terms of the criteria by which they can be evaluated, and the criteria themselves need to be weighted.

• Digital skills overheads. Introducing a co-creative approach or crowdsourcing for VR content development, inherently increases the digital overheads that learners and educators incur in order to be able to participate in the process. However, as overall digital literacy increases and with the advent of visual creative tools, such overheads are diminished.
• Systemic changes in the academic world. Participatory design methods require quite a paradigm shift in the academic world. A resource developed from either a buy-in company or the standard development pipeline carries the quality assurance of an officially recognized entity. Co-created or crowdsourced material is, at times, viewed with suspicion, since the question of quality is not a foregone conclusion. Systemic changes in the academic world, including validating bodies for resource fidelity or at least an institutional auditing system for such resources are useful to alleviate doubts about the validity of such resources.
• Variable fidelity of resources. Co-created resources fidelity can be widely varied addressing the needs of the targeted. While a constant standard of technical fidelity is not always required (low-fi resources targeted at specific use cases can be very effective). In these cases, what becomes crucial is a solid visible and highly transparent QA system embedded in the whole co-creation process to both validate the resource but also certify the created resource against specific use cases and target groups.

4. Discussion

4.1. Decision Making

From the previous analysis, a series of conclusions regarding best practices for the co-creation of VR resources and their use in healthcare education emerge.

• Even though co-creative VR resource implementation approaches have some advanced digital literacy overheads, they are an effective pathway to the prolific use of immersive media in medical education.
• Participatory design methods should be introduced to educators and learners always in conjunction with a training framework in the supporting and enabling technologies that are required for them.
• Co-creative resource development should always consist of rapid iteration cycles with design and development closely engaging all stakeholder teams to maintain educational and not just technical quality.
• Co-creative resource development should embed in every iteration a quality assurance process that (a) maintains a required level of technical and topical fidelity and (b) certifies resources against specific educational use cases, target groups, or even discrete episodes of learning.
• Co-creative resource development should be identified as a bespoke active learning and training activity a bespoke learning modality as participatory knowledge transfer proves to be a valuable educational experience.

4.2. Best Practices for the Co-Creation of VR Resources in Conjunction with Other Guidelines and Frameworks

Experiential learning through scenario-based learning, learning by doing, or other forms, is evident in the literature for many years, with theories such as Kolb’s experiential Learning Cycle [86] theoretically complementing such learning. In order to design and implement such approaches different guidelines have been proposed with no exception to digital learning. For augmented reality, the designers are encouraged to provide appropriate challenges, include a drive by a gamified story, and enhance curiosity by seeing the unseen [87]. Cuendet et al. [88] proposed based on their research experiences five design principles (integration, awareness, empowerment, flexibility, and minimalism) to enhance an AR learning system “work” in a classroom. Furthermore, the “Necessary Nine” Design Principles for Embodied Education in VR focuses on two affordances: the sensation of presence and the embodied affordances of gesture in a three-dimensional learning space [89].

Apart from the design-centred guidelines, a number of studies focuses on co-creation practices. Andersson et al. [90] suggested that co-creation, situated learning, and iterative prototyping result in mind-changing turning points, stating also that an iterative co-creation process is onerous for the participants at the beginning, it is followed by appreciation, in a health context, bringing relevant and meaningful change, which is in line with our suggested best practices. A conceptual model for co-creation in higher education aimed to inform and guide practice for the faculty and administration [91] notes that the “true value of exchange [of knowledge] is the application of the resource by the consumer [learner]”, which points the need for the resources to be tailored to learners needs against specific educational use cases, target groups, or even discrete episodes of learning. A framework for value co-creation in virtual academic learning [92] within its partnership requirements dimension identify, among other, the need for students and teachers to have the necessary knowledge and expertise for co-creation. This is considered essential to achieve the goal of value co-creation, which agrees with the suggested best practices of our work. Including a variety of stakeholders in the co-creation process being aware of the topic of co-creation can be also interpreted as a necessity to include the right stakeholders being aware of the training framework in the supporting and enabling technologies that are required for them, as identified also in the proposed best practices.

The impact of co-creation as a bespoke learning and training activity has been identified in different contexts, enhancing a sense of empowerment, making participants engaged, and improve critical thinking about technology design [93]. Co-creation has been reported to enhance student satisfaction and collaboration on value co-creation [94], while a guided design-based learning approach having student developing AR artifacts proved to be effective both for motivation and learning [95] Thus, co-creation, as noted in the suggested best practices should be identified as bespoke learning modality as participatory knowledge transfer proves to be a valuable educational experience.

5. Conclusions

Sharing the Results of the Process

The last part of the process is the description of the communication channels that will advocate the determined policies to the relevant stakeholders. In the current work, the systematic pathway from concepts to decision-making served as a rigorous formulation of guidelines. Starting from the definition of the problem, clear evidence was present that access to quality VR resources for healthcare learners was limited. Furthermore, a search both in academic and grey literature assembled our evidence base, on which we stood to construct alternatives to the standard development pipeline such as crowdsourcing, buy-in services, and a co-creative, cross-disciplinary design process. To assess these alternatives, we identified as criteria both educational efficacy and resource efficiency, while also considering acceptability from end users/learners. In projecting the outcomes for each alternative, we projected the impact to development speed, quality, and cost of resources, as well as fitness for purpose. Exploring the trade-offs of each alternative we projected the impact on digital skill needs overheads (how much more digitally trained personnel is needed for each alternative), as well as the systemic changes that are needed in the academic environment to sustain each option. Based on this multifaceted analysis we have reached our evidence-based decisions on the presented best practice recommendations and guidelines for VR resource design and implementation. These guidelines are further supported by technical publications, stemming from the same endeavor.

The current work demonstrates the first body of evidence for co-creative VR resource development as a valid workflow for the healthcare immersive content development method. Even though this work is directed to the co-creation of immersive content development for healthcare, it can be further applied to other areas. Future works aim to iterate on this approach and expand upon the capacities of participatory knowledge sharing both as a content creation approach and a bespoke educational framework.