Next Article in Journal
Static Terrestrial Laser Scanning (TLS) for Heritage Building Information Modeling (HBIM): A Systematic Review
Previous Article in Journal
Exploring an Affective and Responsive Virtual Environment to Improve Remote Learning
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Concept Paper

Eliciting Co-Creation Best Practices of Virtual Reality Reusable e-Resources

Panagiotis E. Antoniou
Matthew Pears
Eirini C. Schiza
Fotos Frangoudes
Constantinos S. Pattichis
Heather Wharrad
Panagiotis D. Bamidis
1 and
Stathis Th. Konstantinidis
Medical Physics and Digital Innovation Lab, School of Medicine, Aristotle University of Thessaloniki, 54124 Thessaloniki, Greece
School of Health Sciences, University of Nottingham, Nottingham NG7 2HA, UK
CYENS Center of Excellence, Nicosia 1016, Cyprus
eHealth Laboratory, Department of Computer Science, University of Cyprus, Nicosia 1678, Cyprus
Author to whom correspondence should be addressed.
Virtual Worlds 2023, 2(1), 75-89;
Submission received: 1 February 2023 / Revised: 8 March 2023 / Accepted: 13 March 2023 / Published: 20 March 2023


Immersive experiential technologies find fertile grounds to grow and support healthcare education. Virtual, Augmented, or Mixed reality (VR/AR/MR) have proven to be impactful in both the educational and the affective state of the healthcare student’s increasing engagement. However, there is a lack of guidance for healthcare stakeholders on developing and integrating virtual reality resources into healthcare training. Thus, the authors applied Bardach’s Eightfold Policy Analysis Framework to critically evaluate existing protocols to determine if they are inconsistent, ineffective, or result in uncertain outcomes, following systematic pathways from concepts to decision-making. Co-creative VR resource development resulted as the preferred method. Best practices for co-creating VR Reusable e-Resources identified co-creation as an effective pathway to the prolific use of immersive media in healthcare education. Co-creation should be considered in conjunction with a training framework to enhance educational quality. Iterative cycles engaging all stakeholders enhance educational quality, while co-creation is central to the quality assurance process both for technical and topical fidelity, and tailoring resources to learners’ needs. Co-creation itself is seen as a bespoke learning modality. This paper provides the first body of evidence for co-creative VR resource development as a valid and strengthening method for healthcare immersive content development. Despite prior research supporting co-creation in immersive resource development, there were no established guidelines for best practices.

1. Introduction

The last century has seen an explosion in all aspects of human knowledge. Medicine has also experienced the same effect with medical data- doubling almost every two years from four decades ago [1]. This trend was exacerbated in the following years with projections in 2011 mentioning medical data doubling the rate of fewer than 3 months. The data is not the same as useful applicable medical knowledge. However, the educational medical content that an aspiring healthcare professional needs to consume before being able to practice safely their craft is enormous. Compounding the effort needed, medical knowledge is critical in nature, and requires exact theoretical background and good tacit knowledge and experience [2]. In that context, the tried-and-true methods of study and practice are constantly challenged.
In that environment, immersive experiential technologies find fertile grounds to grow and support medical education. Virtual patients, chatbots, and Virtual, Augmented, or Mixed reality (VR/AR/MR; under the umbrella term eXtended Reality XR) have proven to impact both the educational and the affective state of the healthcare student’s [2,3,4,5] increasing engagement. Additionally, taking cues from different disciplines such as physics and chemistry educational content has proved effective- with students able to visualize abstract laws in an almost tangible way [6,7,8]. The sensory immediacy of these technologies can produce an intuitive anchoring of the core material to the learner and facilitate a paradigm building based on sound scientific data. This paradigm-building leads to robust, deep topical knowledge and reduces the possibility of establishing or maintaining conceptual errors [9]. In the medical field, there is already a significant body of immersive content. In 2019 the Royal College of Physicians identified Virtual Reality (VR) as a “…powerful educational tool for defined learning objectives…” listing several applications in medical education and surgery [10], p. 181.
Resource-wise, the global virtual reality market size was $3.10 billion in 2019 and is expected to reach $57.55 billion by 2027, while the healthcare market is equally optimistic with its 2018 size being $1.56 billion- expected to reach $30.40 billion by 2026 [11,12]. These numbers demonstrate the potential for immersive content in healthcare education, but they also indicate the significant resource overhead that such content incurs for development, testing, and deployment. A 2019 study identified a cost of $106,387.00 for designing and implementing a VR training exercise for hospital staff regarding evacuation procedures [13]. While these costs are feasible, given the reusability of the VR content, which drastically reduces its per trainee cost over time, the volatility of medical knowledge quickly makes such resources obsolete and reduces their reusability potential. In such an environment, a methodology that would further reduce implementation and design costs, as well as distribute the weight of resource development, would be useful.

1.1. Co-Creation as Method towards Improved Sustainable Development

Co-creation provides dynamic assistance to the product design process for new item improvements [14,15] in the context of co-production. Co-production requires “cooperating with clients” [16,17] or even an involvement in the product/service configuration process [18,19]. Client investment can be expressed as an assistance to the outskirts of a company’s workflow [18], or as a complex, central component, through the use of the company’s knowledge and data sharing and learning [20,21]. Co-creation was also identified by a client association that demonstrated mutual physical, mental, and business practices, as well as access to the various masteries [22]. More specifically, co-generation has been described as a scheme of acts carried out by actors (financial, social, and other) involved in value chain networks [23,24]. It is carried out by coordination [25], exchange [26,27], and the integration of common assets into the value generation process [28]. As clients devote assets through co-creation forms, the primary stakeholder (firm, creator) achieves both the demonstrated satisfaction of the client’s request and the leveraging of the client’s expertise in the expansion of the firm [29,30]. Co-creation also makes it possible for the artistic process to be divided up, but the features of co-production remain within the primary stakeholder [31]. This process helps clients to be completely engaged in the co-production process [32,33] with some studies going so far as to recognize the importance in mutuality, receptivity, and non-hierarchical relations [21,29] as a co-production feature. This kind of detailed understanding of the co-creation process has led research [34] to identify knowledge distribution as one of the key drivers of co-creation effectiveness.
Even though there are publications supporting and technically facilitating the co-design and co-creation of immersive resources (e.g., cf. [35,36]), deciding on a definitive set of best practice guidelines pointing to co-creation as an improved content development methodology is not a proven conclusion.
Health policy often dictates protocols and helps healthcare stakeholders and other providers make evidence-based decisions about important issues. However, if recommended actions are conflicting, ineffective, cost-prohibitive, or result in questionable outcomes, they warrant review [37]. Healthcare policymakers are at the intersection of policy and practice and are naturally positioned to address gaps in healthcare policy and to conduct such research. Health policy guides many decisions that specialists make about healthcare and education. Well-crafted policy has implications for ensuring timely and accurate guidance for stakeholders to deliver effective medical education. Healthcare stakeholders seeking guidance on integrating virtual reality (VR) resources for healthcare training may not find recommendations that align with their daily clinical experiences. This can prompt them to reassess prevailing policy in specific contexts or with unique populations. As policies profoundly impact education outcomes, subsequent patient care, and the overall health of populations, policy analysis is a critical research tool.

1.2. Implementing the Eightfold Policy Analysis Framework towards a Set of Best Practices for Co-Creative VR Content Production

Thus, based on outcomes from the CoViRR project and other sources, this is the first attempt to apply the eightfold policy analysis framework first proposed by Bardach and Patashnick [38]. It is commonly applied in policy and administration research as well as in public health. Bardach’s eightfold policy analysis framework includes the following steps. (1) Defining the problem: This crucial steps provides the rational and a sense of direction on how to assemble the evidence; (2) assembling evidence: identify the relevant background, trends and literature through various resources; (3) constructing alternatives: forming alternatives course of actions or strategies addressing the problem; (4) selecting criteria: explores how alternatives can be measured and evaluated identifying the effectiveness of the policy; (5) projecting outcomes: projecting all the outcomes or impacts that you or other interested parties might reasonably care about considering how realistic or viable each outcome is; (6) confronting trade-offs: considering the trade-offs between and within each policy alternative, in terms of the criteria by which they can be evaluated, and the criteria themselves need to be weighted; (7) decision-making: based on the previous steps deciding on the final policies; and (8) sharing the results of the process: telling your story in the form of a narrative or a set of recommendations. In Table 1, the eightfold policy analysis framework is broken down to illustrate how this more comprehensive approach can similarly illuminate decision-making processes for a co-creative approach in VR medical education participatory design.

2. Materials and Methods

2.1. Define the Problem

Immersive technologies which include VR can allow improvement and tracking of performance indices, and acceleration of cognitive functions in healthcare providers [39,40], refined decision-making in emergency scenarios [41], and improve preparedness from procedural and environment familiarization [42,43,44]. Face-to-face high-fidelity scenarios can be transformed into interactive and immersive packages [45], and trainees can interact remotely with staff in real-time using augmented and virtual reality systems [46]. Indeed, most elements of healthcare can be transformed into VR. Techniques, behaviors, theories, protocols, communication, equipment, and other components can be simulated. To aid in this there are several guidelines available to follow that have been created by XR companies to help develop learning resources [34,47,48]. They provide advice for best practices in technology and operations and include case-based steps of the experimental setup to help readers/developers understand how best to test VR with existing training/ educational resources [49]. Some companies and institutions [47] perform the following steps in streamlining the content during development and being fit for purpose:
  • Mapping user experience [32,33,34,48]
  • Developing the most prioritized resources—prominent level of detail and features may not be needed to reach the training goal [21,29,50]
  • Understand user abilities and assert the usage schedule/integration.
However, these recommendations are optimal under certain conditions and typically do not focus on the initial design of learning resources. Therefore, many unsuccessful or problematic results can occur when learning and research institutions attempt to create VR material. The collaboration of VR developer support and internal research groups creating an application for experimental/research purposes can have missing skillsets from the absence of pedagogical design and overlook the inclusion of stakeholders in the creation process. End-users are key in asserting the research priorities, and without their inclusion key issues may be overlooked from the start. Perspectives can be technically orientated but without co-creation, underlying problems may not be addressed. There is a large gap between the needs of healthcare professionals, best practice with inclusion from stakeholders, and steps for utilization of available resources during design and development. A review of current methods of developing virtual reality reusable e-resources is presented in this chapter, including the benefits that co-creation methods could bring. Reviewing the current state of practices allows investigation towards a streamlined approach that can be shared with a global audience for their own resource development.

2.2. Assemble the Evidence

Step 2 instructs to gather current evidence of the landscape of development. The standard pipeline involves in-house development of VR resources using topical experts for consultation and technology experts for design and implementation, with external assistance where required. Institutions can gain financial, intellectual, and research benefits from training their researchers and creating learning resources in-house. Many commonly used software programs are free or low-cost, enabling institutions to create resources in-house for many years with little expenditure. This also helps to understand and keep up to date with XR development changes. There may be a steep learning curve initially, but progress increases once this initial barrier is overcome. Topical experts can provide valuable insights, as they are in close proximity and can validate the content of learning resources. The combination of internal development and topical expert presence is the de facto methodology, as evidenced by recent literature reviews [51].
Nevertheless, this can create lag in project progress as researchers take time to refine their skills. Indeed, there is a need for assistance with complex parts of a system, and the rapidly changing landscape of XR software and components means that sourcing professional consultation will be required. As a realistic compromise that can negate the high price of private companies with the benefits of in-house design, collaboration with sole designers or smaller companies are a growing occurrence. An external consultant can guide researchers with minimal expenditure but be supported when required. There are a growing number of such companies as guidance to institutions. Companies can allow researchers to hire equipment alongside assisting them to develop applications themselves [52,53]. Few guidelines include stakeholders to the degree of closeness required to optimize pedagogy and reusability. Universities have the choice to develop their own content or pay 3rd parties, and many collaborations have occurred [54]. But each application is made differently, with different techniques, user inputs, budgets, goals, and time. A standardized and efficient process is required that can streamline all processes to save cost, time, development effort, and be optimized by, and for, the end user.

2.3. Construct Alternatives

Step 3 of Bardach’s framework then involves the construction of policy alternatives. The process involves the identification, with a critical eye, of possible courses of action in the specific topic or challenge. In this framework, specific mention is made of the “status quo” policy, that is the maintenance of the current policy or standard of practice. This is a necessary methodological precaution against policy changes that would disregard efficient current practices. In the case of VR resource development, a series of approaches have been identified from the literature and common practices. These along with a brief description are outlined in Table 2.

2.4. Policy Alternatives

What follows is a brief outline of their features and some critical “observations from the field” as they appear in the literature and from the CoViRR experience.

2.4.1. Maintain Standard Development Pipeline

This method, being the standard of practice, has the largest coverage in the literature. Such example studies have used or explored this standard production pipeline [47,55,56,57,58]. A recent study has even conducted a cost analysis of VR vs physical presence educational episodes [13]. Results demonstrated that VR costs were 43% greater than the real-world exercise cost. This difference in costs led the study to the conclusion that the per trainee cost would only become effective if this single resource was utilized and maintained for over 3 years, unchanged and with the same demand from the learners.

2.4.2. Stakeholders’ Co-Design of Educational Resources

The co-creation avenue has been proposed as a valid content development and deployment pipeline. The co-creation concept emerged from marketing and more specifically from product design. Value co-creation (VCC), as it was originally termed, was the process of identifying an item’s value offer through client participation rather than the standard statistical surveying avenues [34,48]. This process allows clients to be fully engaged in the co-production process [21,49], with some studies going as far as to identify value in mutualism, receptiveness, and non-hierarchical relations [29] as an element of co-production. This kind of extensive interpretation of the co-creation process has led research [34] to expose as one of the main factors of co-creation efficacy the sharing of knowledge. Literature has proposed content development pipelines and technological innovation approaches to facilitate digital content co-creation from non-technology experts (e.g., educators and learners). A much simpler approach was followed in CoViRR. Utilizing stakeholder engagement in co-design storyboarding and collaborative implementation with co-creative input from the education experts themselves, nine re-usable VR resources have been effectively created at a fraction of the cost of the standard VR development pipeline. Another strength of output comes from ProteinVR [74], who used co-creation and stakeholder inclusion by actively involving users in the software development process through feedback and bug reporting. The open-source nature of the software also allows for contributions from a wide range of stakeholders, including researchers and educators. Furthermore, the 360ViSi [75] ERASMUS+ project is an international collaboration among several universities and companies in Europe. It explores co-creation methodologies [76] using 360° video technology to increase access to low-cost simulation training in health education, and to stimulate the flow and exchange of knowledge between higher education and enterprises.

2.4.3. Crowdsource Resource Design

Initially ascribed by Howe in 2006 [59], crowdsourcing entails an open call of a function from a company or institution to an undefined size network of individuals. Crowdsourcing development has been successful in business and industry yet utilization by educational institutions in outsourcing functions or tasks from education professionals has not had a comparable impact [60]. Technological and internet-based tools allow accelerated idea generation, micro-tasking, and problem-solving. This process has suitability for educational crowdsourcing as a mechanism for collaborative learning in addition to the subsequent crowdsourced content [61]. Medical education has similarities with crowdsourcing as they both have an openness to delivering resources by facilitating collaborative methodologies. Dissemination of materials amongst educators has been shown to be an effective approach to developing, creating, and sharing digital teaching resources [60,61,62,63]. Online content, appraisal of quality, and development of new material in medical education initiatives have increased. Weld et al. [64] suggested that crowdsourcing can solve problems of delivering large-scale virtual education, in addition to modifying an online crowd to be its own support for learners. Therefore, crowd-sourced processes in the design of educational material may accelerate this potential while providing an effect on stakeholders via innovative outputs and collaborative strength.
Examples of this include online crowd utilization during COVID-19 [65] to produce a framework for structured, crowdsourced innovation in healthcare R&D. This took the form of a hackathon, where individuals were convened to crowdsource solutions around a core set of predetermined challenges in a limited amount of time. Furthermore, online crowd recruitment and distribution of educational videos allowed each user to watch and summarize a video segment [66]. Users not only watched the educational videos but also collectively modified the multi-language caption data for all videos, improving their future use for foreign viewers.
Yet, large-scale learning repositories for healthcare are not widely accessible. A 2019 narrative literature review on the role of crowdsourcing to educate health professions suggested no publications involving its role in healthcare curricula design [67], instead a handful of instructional design [68], surgical recruitment [69], and surgical crowdsources assessments [70] projects were noted. This may be due to a lack of guidelines towards crowdsourcing resource design of resources, especially with novel immersive technologies of VR and AR. There are several examples of web-based VR experiments using crowdsourcing [71] however none yet include tasks in the co-creation of educational medical resources. Examples include a ‘VRChat’ [72] application able to implement collaborative VR user studies and future use could allow the co-design of digital resources with VR headset crowds [73]. Additionally, [77] created a cloud-based VR platform that enables crowdsourcing of HRI experiments, and initially tested in a competition where users interacted with a virtual robot through VR devices and the Internet.

2.4.4. Buy-in Services from Companies

There are several hundreds of XR companies creating products for healthcare solutions [78]. The current issue, however, is that the methods used to create XR products are not clear, and a standardized approach has not yet been developed due to the infancy of VR and AR development in healthcare. Stakeholder inclusion is sporadic, and co-creation methods are not widely used. Companies utilizing VR-related technology must successfully navigate a plethora of factors before attaining clients, creating resources, and receiving the financial, research-driven, and social benefits from collaborations. Due to the volatility of research requirements, technology improvements, and availability, each company takes a different approach to collaboration and creation.
However, if collaboration is successful there are many advantages to allowing dependency on external companies in creating products.
  • They have the capacity to create custom packages for medical learners specific to the intended learning target.
  • They have rapid development and turnaround, with the ability to support during and after resource implementation.
  • They have expert insight which may improve the initial plan and output of the learning resource.
There are many disadvantages to sourcing external help in the development for the learning institute. Primarily, the high cost of hiring equipment, staff, processing, and long-term support means for many institutions this option is heavily restricted. Co-creational methods are not the central focus when incorporating private companies as fully developed plans are typically sought to allow immediate resource creation. This means participatory inclusion may be limited/affected during the main development processes. However, there are case examples where companies have successfully collaborated with educational institutions to create effective VR resources. Examples being Nanome [79]—a VR molecular design tool that allows for natural collaboration and interaction with any molecular structure; and Osso [80] VR training and assessment platform that enables healthcare professionals to train on any procedure.

2.5. Select the Criteria

Selecting the criteria is step 4 in Bardach’s framework and explores how the impact of alternatives can be measured and evaluated. This is crucial for determining a policy’s effectiveness. Criteria should be established based on the scope of impact that a policy is aimed to address. It should also be based on established research to ensure the validity that these criteria should carry, subsequently validating the selection process. When exploring the criteria for selecting the most appropriate medical VR resource creation methodology, the evaluation strategy of CoViRR came into focus. There, several axes of efficacy and impact have been identified, which, subsequently, have evolved into the relevant criteria subsequently presented.
  • Financial Efficacy. This is an aspect focusing on the economy of resources both human and technological. Such an evaluation utilizes a cost-benefit analysis considering several factors (c.f. the analysis conducted in [13]). These factors are (a) actual costs accrued, (b) audience size and re-usability/repurposing ability of the resources, and (c) the validity of the resource across multiple curricular instances (i.e., the ability for the resource to be used “as is” in several consecutive or not semesters of training.
  • Technical efficacy and acceptance, focusing on optimization of the VR experience. This includes usability testing and user experience optimization. Such an effort uses instruments such as SUS [81] and a modified TAM instrument [82]. It must be noted that technical evaluation in the context of CoViRR does not involve VR device evaluation. Thus, its focus stays in the flow and implementation of the resources themselves in multiple VR platforms if this becomes possible.
  • Pedagogical efficacy and acceptance, focusing on validating the resources as instruments in specific educational episodes. This includes knowledge retention assessment as well as student engagement and acceptance of the resource. Pedagogical evaluation involves both qualitative and quantitative methods and aims to explore the value of the created resources as both a knowledge transfer vehicle, as well as a summative assessment instrument. Validated methods for such an evaluation includes OSCEs and e-OSCEs [83].
  • Stakeholder acceptance/Curriculum feasibility evaluation, focusing on the capacity of the resources to be integrated into medical curricula without extensive institutional or technical overheads. This axis is more of an integrative, meta-evaluation axis, considering learner and educator digital literacy, technology readiness levels of the supporting institutions, as well as financial and administrative considerations. As such, it is best explored using qualitative instruments like semi-structured interviews, heuristics, and focus group sessions [84,85], which can best capture tacit, along with formal, aspects of curriculum integration.

3. Results

3.1. Project the Outcomes

Step 5 in the approach, is, many times, considered the most difficult to explore. The challenge of making realistic projections in each specific criterion selected in the previous step is compounded by the need to assess synergies and cross-pollinating effects between the criteria, as they impact the whole. In that context, this step of the method requires the identification of aspects that would be impacted by the proposed alternatives. In the context of medical education content creation and delivery, we identify the aspects below.
  • Assess impact on curriculum. The content creation methodology affects the curriculum both implicitly but also directly. Implicitly, changes in the development and deployment methodologies may produce variations in the quality of the content delivered. For these cf. the third section of this step “Assess quality of resources development”. Direct impact to the curriculum from the development process comes from the tacit training that co-creation provides to the personnel that takes part in the process. It is not uncommon for students and learners to gather novel insights into the subject matter after the relevant co-creation workshops. In that context, co-creation and crowdsourced approaches may lead to a more direct positive curriculum impact.
  • Assess speed of resource development. Speed of resource development is the core advantage of the co-creation and crowdsourcing approaches. Distributing the design load and having immediate access to high-quality feedback provides concrete advantages compared to the other methods.
  • Assess quality of resource development. Co-creation and crowdsourcing of educational resources require rigorous quality control provisions to maintain the required level of quality for curricular integration. Company buy-in services on the other hand have an administrative overhead similar to that of the current methodology, given the fact that developed resources have to go back and forth between the developer and the client for QA and approval.
  • Assess cost of resource development. Together with the speed of resource development, the cost is also an aspect that benefits from co-creation and crowdsourcing. Overheads for iterative design are alleviated since the requirements elicitation comes interactively from the stakeholders themselves without costly pre-production overheads and a subsequent reduction of iterations required for a finished resource.
  • Assess the level the resources are tailored to the need of the learners. In that aspect, participatory design methods come ahead even from crowdsourcing because the collaborative creative process inherently adapts resources at the most fundamental level, design. As such, adaptability to the learner’s needs is best served through this methodology.
  • Assess the repurposing ability of resources developed in each methodology. Both crowdsourcing and co-creation approaches leverage heavily open access technologies and repurposing architectures. In that context, resources developed through these processes have increased re-purposing capacity than those developed through the standard development pipeline or the buy-in approach from companies.

3.2. Confronting the Trade-Offs

In step 6, confronting trade-offs, we need to consider the trade-offs between and within each policy alternative. The trade-offs need to be considered in terms of the criteria by which they can be evaluated, and the criteria themselves need to be weighted.
  • Digital skills overheads. Introducing a co-creative approach or crowdsourcing for VR content development, inherently increases the digital overheads that learners and educators incur in order to be able to participate in the process. However, as overall digital literacy increases and with the advent of visual creative tools, such overheads are diminished.
  • Systemic changes in the academic world. Participatory design methods require quite a paradigm shift in the academic world. A resource developed from either a buy-in company or the standard development pipeline carries the quality assurance of an officially recognized entity. Co-created or crowdsourced material is, at times, viewed with suspicion, since the question of quality is not a foregone conclusion. Systemic changes in the academic world, including validating bodies for resource fidelity or at least an institutional auditing system for such resources are useful to alleviate doubts about the validity of such resources.
  • Variable fidelity of resources. Co-created resources fidelity can be widely varied addressing the needs of the targeted. While a constant standard of technical fidelity is not always required (low-fi resources targeted at specific use cases can be very effective). In these cases, what becomes crucial is a solid visible and highly transparent QA system embedded in the whole co-creation process to both validate the resource but also certify the created resource against specific use cases and target groups.

4. Discussion

4.1. Decision Making

From the previous analysis, a series of conclusions regarding best practices for the co-creation of VR resources and their use in healthcare education emerge.
  • Even though co-creative VR resource implementation approaches have some advanced digital literacy overheads, they are an effective pathway to the prolific use of immersive media in medical education.
  • Participatory design methods should be introduced to educators and learners always in conjunction with a training framework in the supporting and enabling technologies that are required for them.
  • Co-creative resource development should always consist of rapid iteration cycles with design and development closely engaging all stakeholder teams to maintain educational and not just technical quality.
  • Co-creative resource development should embed in every iteration a quality assurance process that (a) maintains a required level of technical and topical fidelity and (b) certifies resources against specific educational use cases, target groups, or even discrete episodes of learning.
  • Co-creative resource development should be identified as a bespoke active learning and training activity a bespoke learning modality as participatory knowledge transfer proves to be a valuable educational experience.

4.2. Best Practices for the Co-Creation of VR Resources in Conjunction with Other Guidelines and Frameworks

Experiential learning through scenario-based learning, learning by doing, or other forms, is evident in the literature for many years, with theories such as Kolb’s experiential Learning Cycle [86] theoretically complementing such learning. In order to design and implement such approaches different guidelines have been proposed with no exception to digital learning. For augmented reality, the designers are encouraged to provide appropriate challenges, include a drive by a gamified story, and enhance curiosity by seeing the unseen [87]. Cuendet et al. [88] proposed based on their research experiences five design principles (integration, awareness, empowerment, flexibility, and minimalism) to enhance an AR learning system “work” in a classroom. Furthermore, the “Necessary Nine” Design Principles for Embodied Education in VR focuses on two affordances: the sensation of presence and the embodied affordances of gesture in a three-dimensional learning space [89].
Apart from the design-centred guidelines, a number of studies focuses on co-creation practices. Andersson et al. [90] suggested that co-creation, situated learning, and iterative prototyping result in mind-changing turning points, stating also that an iterative co-creation process is onerous for the participants at the beginning, it is followed by appreciation, in a health context, bringing relevant and meaningful change, which is in line with our suggested best practices. A conceptual model for co-creation in higher education aimed to inform and guide practice for the faculty and administration [91] notes that the “true value of exchange [of knowledge] is the application of the resource by the consumer [learner]”, which points the need for the resources to be tailored to learners needs against specific educational use cases, target groups, or even discrete episodes of learning. A framework for value co-creation in virtual academic learning [92] within its partnership requirements dimension identify, among other, the need for students and teachers to have the necessary knowledge and expertise for co-creation. This is considered essential to achieve the goal of value co-creation, which agrees with the suggested best practices of our work. Including a variety of stakeholders in the co-creation process being aware of the topic of co-creation can be also interpreted as a necessity to include the right stakeholders being aware of the training framework in the supporting and enabling technologies that are required for them, as identified also in the proposed best practices.
The impact of co-creation as a bespoke learning and training activity has been identified in different contexts, enhancing a sense of empowerment, making participants engaged, and improve critical thinking about technology design [93]. Co-creation has been reported to enhance student satisfaction and collaboration on value co-creation [94], while a guided design-based learning approach having student developing AR artifacts proved to be effective both for motivation and learning [95] Thus, co-creation, as noted in the suggested best practices should be identified as bespoke learning modality as participatory knowledge transfer proves to be a valuable educational experience.

5. Conclusions

Sharing the Results of the Process

The last part of the process is the description of the communication channels that will advocate the determined policies to the relevant stakeholders. In the current work, the systematic pathway from concepts to decision-making served as a rigorous formulation of guidelines. Starting from the definition of the problem, clear evidence was present that access to quality VR resources for healthcare learners was limited. Furthermore, a search both in academic and grey literature assembled our evidence base, on which we stood to construct alternatives to the standard development pipeline such as crowdsourcing, buy-in services, and a co-creative, cross-disciplinary design process. To assess these alternatives, we identified as criteria both educational efficacy and resource efficiency, while also considering acceptability from end users/learners. In projecting the outcomes for each alternative, we projected the impact to development speed, quality, and cost of resources, as well as fitness for purpose. Exploring the trade-offs of each alternative we projected the impact on digital skill needs overheads (how much more digitally trained personnel is needed for each alternative), as well as the systemic changes that are needed in the academic environment to sustain each option. Based on this multifaceted analysis we have reached our evidence-based decisions on the presented best practice recommendations and guidelines for VR resource design and implementation. These guidelines are further supported by technical publications, stemming from the same endeavor.
The current work demonstrates the first body of evidence for co-creative VR resource development as a valid workflow for the healthcare immersive content development method. Even though this work is directed to the co-creation of immersive content development for healthcare, it can be further applied to other areas. Future works aim to iterate on this approach and expand upon the capacities of participatory knowledge sharing both as a content creation approach and a bespoke educational framework.

Author Contributions

Conceptualization, S.T.K., C.S.P., H.W. and P.D.B.; methodology, P.E.A., M.P. and S.T.K.; validation, P.E.A., M.P., E.C.S. and F.F.; formal analysis, P.E.A., M.P., E.C.S. and F.F.; investigation, P.E.A., M.P., E.C.S. and F.F.; resources, All authors; writing—original draft preparation, P.E.A., M.P. and E.C.S.; writing—review and editing, P.E.A., S.T.K. and M.P.; visualization, P.E.A., S.T.K. and M.P.; supervision, S.T.K.; project administration, P.E.A. and S.T.K.; funding acquisition, S.T.K., H.W., P.D.B. and C.S.P. All authors have read and agreed to the published version of the manuscript.


This work supported by the ERASMUS+ Strategic Partnership in Higher Education “Co-creation of Virtual Reality reusable e-Resources for European Healthcare Education” (CoViRR) ( (accessed on 10 March 2023)) (2018-1-UK01-KA203-048215) project of the European Union.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

Not applicable.

Conflicts of Interest

The authors declare no conflict of interest.


  1. Antoniou, P.E.; Athanasopoulou, C.A.; Dafli, E.; Bamidis, P.D. Exploring design requirements for repurposing dental virtual patients from the web to second life: A focus group study. J. Med. Internet Res. 2014, 16, e151. [Google Scholar] [CrossRef] [Green Version]
  2. Antoniou, P.E.; Arfaras, G.; Pandria, N.; Athanasiou, A.; Ntakakis, G.; Babatsikos, E.; Nigdelis, V.; Bamidis, P. Biosensor Real-Time Affective Analytics in Virtual and Mixed Reality Medical Education Serious Games: Cohort Study. JMIR Serious Games 2020, 8, e17823. [Google Scholar] [CrossRef] [PubMed]
  3. Antoniou, P.; Arfaras, G.; Pandria, N.; Ntakakis, G.; Bambatsikos, E.; Athanasiou, A. Real-Time Affective Measurements in Medical Education, Using Virtual and Mixed Reality. In Brain Function Assessment in Learning. BFAL 2020; Lecture Notes in Computer, Science; Frasson, C., Bamidis, P., Vlamos, P., Eds.; Springer: Cham, Switzerland, 2020; Volume 12462. [Google Scholar] [CrossRef]
  4. Dolianiti, F.; Tsoupouroglou, I.; Antoniou, P.; Konstantinidis, S.; Anastasiades, S.; Bamidis, P. Chatbots in Healthcare Curricula: The Case of a Conversational Virtual Patient. In Brain Function Assessment in Learning. BFAL 2020; Lecture Notes in Computer, Science; Frasson, C., Bamidis, P., Vlamos, P., Eds.; Springer: Cham, Switzerland, 2020; Volume 12462. [Google Scholar] [CrossRef]
  5. Kyriakidou, M.R.; Antoniou, P.; Arfaras, G.; Bamidis, P. The Role of Medical Error and the Emotions it Induces in Learning—A Study Using Virtual Patients. In Brain Function Assessment in Learning. BFAL 2020; Lecture Notes in Computer, Science; Frasson, C., Bamidis, P., Vlamos, P., Eds.; Springer: Cham, Switzerland, 2020; Volume 12462. [Google Scholar] [CrossRef]
  6. Klopfer, E.; Squire, K. Environmental Detectives—The development of an augmented reality platform for environmental simulations. Educ. Technol. Res. Dev. 2008, 56, 203–228. [Google Scholar] [CrossRef]
  7. Dunleavy, M.; DeDe, C.; Mitchell, R. Affordances and Limitations of Immersive Participatory Augmented Reality Simulations for Teaching and Learning. J. Sci. Educ. Technol. 2009, 18, 7–22. [Google Scholar] [CrossRef]
  8. Wu, H.-K.; Lee, S.W.Y.; Chang, H.-Y.; Liang, J.-C. Current status, opportunities and challenges of augmented reality in education. Comput. Educ. 2013, 62, 41–49. [Google Scholar] [CrossRef]
  9. Olympiou, G.; Zacharia, Z.C. Blending physical and virtual manipulatives: An effort to improve students’ conceptual understanding through science laboratory experimentation. Sci. Educ. 2012, 96, 21–47. [Google Scholar] [CrossRef]
  10. Pottle, J. Virtual reality and the transformation of medical education. Future Healthc. J. 2019, 6, 181–185. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  11. “Virtual Reality Market Share” Fortune Business Insights VR Market Trends Online. Available online: (accessed on 15 February 2021).
  12. “Virtual Reality Healthcare Market Size” Fortune Business Insights VR Health Online. Available online: (accessed on 15 February 2021).
  13. Farra, S.L.; Gneuhs, M.; Hodgson, E.; Kawosa, B.; Miller, E.T.; Simon, A.; Timm, N.; Hausfeld, J. Comparative Cost of Virtual Reality Training and Live Exercises for Training Hospital Workers for Evacuation. CIN Comput. Inform. Nurs. 2019, 37, 446–454. [Google Scholar] [CrossRef] [PubMed]
  14. Chien, S.-H.; Chen, J.-J. Supplier involvement and customer involvement effect on new product development success in the financial service industry. Serv. Ind. J. 2010, 30, 185–201. [Google Scholar] [CrossRef]
  15. Droge, C.; Stanko, M.A.; Pollitte, W.A. Lead Users and Early Adopters on the Web: The Role of New Technology Product Blogs. J. Prod. Innov. Manag. 2010, 27, 66–82. [Google Scholar] [CrossRef]
  16. Hu, Y.; McLoughlin, D. Creating new market for industrial services in nascent fields. J. Serv. Mark. 2012, 26, 322–331. [Google Scholar] [CrossRef]
  17. Nuttavuthisit, K. If you can’t beat them, let them join: The development of strategies to foster consumers’ co-creative practices. Bus. Horiz. 2010, 53, 315–324. [Google Scholar] [CrossRef]
  18. Auh, S.; Bell, S.J.; McLeod, C.S.; Shih, E. Co-production and customer loyalty in financial services. J. Retail. 2007, 83, 359–370. [Google Scholar] [CrossRef]
  19. Lemke, F.; Clark, M.; Wilson, H. Customer experience quality: An exploration in business and consumer contexts using repertory grid technique. J. Acad. Sci. 2011, 39, 846–869. [Google Scholar] [CrossRef] [Green Version]
  20. Boselli, R.; Cesarini, M.; Mezzanzanica, M. Customer knowledge and service development, the web 2.0 role in co-production. In Proceedings of the World Academy of Science, Engineering, and Technology, Parigi, France, 30 July 2008. [Google Scholar]
  21. Ordanini, A.; Pasini, P. Service co-production and value co-creation: The case for a service-oriented architecture (SOA). Eur. Manag. J. 2008, 26, 289–297. [Google Scholar] [CrossRef]
  22. Ertimur, B.; Venkatesh, A. Opportunism in Co-Production: Implications for Value Co-Creation. Australas. Mark. J. 2010, 18, 256–263. [Google Scholar] [CrossRef]
  23. Vargo, S.L.; Lusch, R.F. Service-dominant logic: Continuing the evolution. J. Acad. Mark. Sci. 2008, 36, 1–10. [Google Scholar] [CrossRef]
  24. Achrol, R.; Kotler, P. Frontiers of the marketing paradigm in the third millennium. J. Acad. Mark. Sci. 2012, 40, 35–52. [Google Scholar] [CrossRef]
  25. Lusch, R.F.; Vargo, S.L.; O’Brien, M. Competing through service: Insights from service-dominant logic. J. Retail. 2007, 83, 5–18. [Google Scholar] [CrossRef]
  26. Aarikka-Stenroos, L.; Jaakkola, E. Value co-creation in knowledge intensive business services: A dyadic perspective on the joint problem solving process. Ind. Mark. Manag. 2012, 41, 15–26. [Google Scholar] [CrossRef]
  27. Grönroos, C. Conceptualising value co-creation: A journey to the 1970s and back to the future. J. Mark. Manag. 2012, 28, 1520–1534. [Google Scholar] [CrossRef]
  28. Ballantyne, D.; Varey, R.J. The service-dominant logic and the future of marketing. J. Acad. Mark. Sci. 2008, 36, 11–14. [Google Scholar] [CrossRef]
  29. Arvidsson, A. Ethics and value in customer co-production. Mark. Theory 2011, 11, 261–278. [Google Scholar] [CrossRef]
  30. Chen, J.S.; Tsou, H.T.; Ching, R.K.H. Co-production and its effects on service innovation. Ind. Mark. Manag. 2011, 40, 1331–1346. [Google Scholar] [CrossRef]
  31. Vargo, S.L.; Lusch, R.F. Evolving to a newdominant logic for marketing. J. Mark. 2004, 68, 1–17. [Google Scholar] [CrossRef] [Green Version]
  32. Krishna, A.; Morrin, M. Does Touch Affect Taste? The Perceptual Transfer of Product Container Haptic Cues. J. Consum. Res. 2008, 34, 807–818. [Google Scholar] [CrossRef] [Green Version]
  33. Troye, S.V.; Supphellen, M. Consumer participation in coproduction: “I made it myself” effects on consumers’ sensory perceptions and evaluations of outcome and input product. J. Mark. 2012, 76, 33–46. [Google Scholar] [CrossRef]
  34. Ranjan, K.R.; Read, S. Value co-creation: Concept and measurement. J. Acad. Mark. Sci. 2014, 44, 290–315. [Google Scholar] [CrossRef]
  35. Antoniou, P.E.; Bamidis, P.D. Devising a Co-creative digital content development pipeline for Experiential Healthcare Education. CC-TEL/TACKLE@EC-TEL 2018, 2190, 1–10. [Google Scholar]
  36. Antoniou, P.E.; Ioannidis, L.; Bamidis, P.D. OSCase: A data scheme for transfer of Web based Virtual Patients to OpenSim. In Ambient Intelligence and Smart Environments; Prauveneers, D., Ed.; IOS Press: Amsterdam, The Netherlands, 2015; pp. 228–235. [Google Scholar] [CrossRef]
  37. Clancy, C.M.; Glied, S.A.; Lurie, N. From Research to Health Policy Impact. Health Serv. Res. 2012, 47, 337–343. [Google Scholar] [CrossRef] [Green Version]
  38. Bardach, E.; Patashnik, E.M. A Practical Guide for Policy Analysis: The Eightfold Path to More Effective Problem Solving; CQ Press: Washington, DC, USA, 2019. [Google Scholar]
  39. MacCrossan, M.; Hamilton, A.; Kerrins, J.; Tallentire, V. 15 Non-technical skills: Developing a behavioural marker system to assess medical student performance within high fidelity simulation. E-Posters 2016, 2, A26. [Google Scholar] [CrossRef]
  40. Frederiksen, J.G.; Sørensen, S.M.D.; Konge, L.; Svendsen, M.B.S.; Nobel-Jørgensen, M.; Bjerrum, F.; Andersen, S.A.W. Cognitive load and performance in immersive virtual reality versus conventional virtual reality simulation training of laparoscopic surgery: A randomized trial. Surg. Endosc. 2020, 34, 1244–1252. [Google Scholar] [CrossRef]
  41. Cha, M.; Han, S.; Lee, J.; Choi, B. A virtual reality based fire training simulator integrated with fire dynamics data. Fire Saf. J. 2012, 50, 12–24. [Google Scholar] [CrossRef]
  42. Sugimoto, M.; Shiga, Y.; Abe, M.; Kameyama, S.; Azuma, T. Immersive Surgical Navigation Using Spatial Interactive Virtual Reality AND Holographic Augmented Reality. Nihon Geka Gakkai Zasshi 2016, 117, 387–394. [Google Scholar] [PubMed]
  43. Pulijala, Y.; Ma, M.; Pears, M.; Peebles, D.; Ayoub, A. An innovative virtual reality training tool for orthognathic surgery. Int. J. Oral Maxillofac. Surg. 2018, 47, 1199–1205. [Google Scholar] [CrossRef] [Green Version]
  44. Judd, T.; Aquilina, A.L.; Hunter, I. Virtual Reality Medical Training: A Non-Inferiority Randomised Controlled Trial of VR vs Face-to-Face Training. J. Medic. Educ. Train. 2020, 4, 44. [Google Scholar]
  45. Orts-Escolano, S.; Rhemann, C.; Fanello, S.; Chang, W.; Kowdle, A.; Degtyarev, Y.; Kim, D.; Davidson, P.L.; Sameh Khamis, S.; Dou, M.; et al. Holoportation: Virtual 3d teleportation in real-time. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology, Tokyo, Japan, 16–19 October 2016; pp. 741–754. [Google Scholar]
  46. Biyani, C.; Hanchanale, V.; Rajpal, S.; Jain, S.; Garthwaite, M.; Cartledge, J.; Somani, B.; Cornford, P.; Gowda, B.; Koenig, P.; et al. First urology simulation boot camp in the United Kingdom. Afr. J. Urol. 2017, 23, 258–267. [Google Scholar] [CrossRef] [Green Version]
  47. Pulijala, Y.; Ma, M.; Pears, M.; Peebles, D.; Ayoub, A. Effectiveness of Immersive Virtual Reality in Surgical Training—A Randomized Control Trial. J. Oral Maxillofac. Surg. 2018, 76, 1065–1072. [Google Scholar] [CrossRef] [Green Version]
  48. The Future of Competition the Summary in Brief, 2004. Available online: (accessed on 18 April 2021).
  49. Pears, M.; Henderson, J.; Antoniou, P.E.; Ntakakis, G.; Nikolaidou, M.; Bamidis, P.D.; Schiza, E.; Pattichis, C.S.; Frangoudes, F.; Gkougkoudi, E.; et al. Feasibility and Acceptance of Virtual Reality Reusable e- Resources Embedded in Healthcare Curricula. In Proceedings of the 2022 International Conference on Interactive Media, Smart Systems and Emerging Technologies (IMET), Limassol, Cyprus, 4–7 October 2022. [Google Scholar] [CrossRef]
  50. Crowdsourcing. Available online: (accessed on 18 April 2021).
  51. Barteit, S.; Lanfermann, L.; Bärnighausen, T.; Neuhann, F.; Beiersmann, C. Augmented, Mixed, and Virtual Reality-Based Head-Mounted Devices for Medical Education: Systematic Review. JMIR Serious Games 2021, 9, e29080. [Google Scholar] [CrossRef]
  52. Harmony Studios UK—Augmented Reality and Virtual Reality Developers. Available online: (accessed on 30 August 2021).
  53. Welcome—Dave Lynch 2021. Available online: (accessed on 30 August 2021).
  54. Virtual Clinical Education|Faculty Resources|CHIPS|UTHSC. Available online: (accessed on 25 May 2021).
  55. Dascal, J.; Reid, M.; IsHak, W.W.; Spiegel, B.; Recacho, J.; Rosen, B.; Danovitch, I. Virtual Reality and Medical Inpatients: A Systematic Review of Randomized, Controlled Trials. Innov. Clin. Neurosci. 2017, 14, 14–21. [Google Scholar]
  56. Lemheney, A.; Bond, W.; Padon, J.; LeClair, M.; Miller, J.; Susko, M. Developing virtual reality simulations for office-based medical emergencies. J. Virtual Worlds Res. Ser. Online 2016, 9, 1–18. [Google Scholar]
  57. Ulrich, D.; Farra, S.; Smith, S.; Hodgson, E. The Student Experience Using Virtual Reality Simulation to Teach Decontamination. Clin. Simul. Nurs. 2014, 10, 546–553. [Google Scholar] [CrossRef]
  58. Farra, S.; Miller, E.; Timm, N.; Schafer, J. Improved Training for Disasters Using 3-D Virtual Reality Simulation. West. J. Nurs. Res. 2013, 35, 655–671. [Google Scholar] [CrossRef] [PubMed]
  59. Hewe, J. The Rise of Crowdsourcing. Available online: (accessed on 18 April 2021).
  60. Foulger, T.S. The 21st-Century Teacher Educator and Crowdsourcing. J. Digit. Learn. Teach. Educ. 2014, 30, 110. [Google Scholar] [CrossRef]
  61. Suhonjić, A.Z.; Despotović-Zrakić, M.; Labus, A.; Bogdanović, Z.; Barać, D. Fostering students’ participation in creating educational content through crowdsourcing. Interact. Learn. Environ. 2019, 27, 72–85. [Google Scholar] [CrossRef]
  62. Blackwell, K.A.; Travis, M.J.; Arbuckle, M.R.; Ross, D.A. Crowdsourcing medical education. Med. Educ. 2016, 50, 576. [Google Scholar] [CrossRef] [Green Version]
  63. Thoma, B.; Mohindra, R.; Artz, J.D.; Chan, T.M. CJEM and the changing landscape of medical education and knowledge translation. Can. J. Emerg. Med. 2015, 17, 184–187. [Google Scholar] [CrossRef] [Green Version]
  64. Weld, D.S.; Adar, E.; Chilton, L.B.; Hoffmann, R.; Horvitz, E.; Koch, M.; Landay, J.A.; Lin, C.H.; Mausam, M. Personalized Online Education—A Crowdsourcing Challenge. Available online: (accessed on 18 April 2021).
  65. Ramadi, K.B.; Nguyen, F.T. Rapid crowdsourced innovation for COVID-19 response and economic growth. NPJ Digit. Med. 2021, 4, 18. [Google Scholar] [CrossRef]
  66. Culbertson, G.; Shen, S.; Andersen, E.; Jung, M. Have your Cake and Eat it Too: Foreign Language Learning with a Crowdsourced Video Captioning System. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing, Portland, OR, USA, 25 February–1 March 2017; pp. 286–296. [Google Scholar] [CrossRef]
  67. John-Matthews, J.S.; Newton, P.; Grant, A.; Robinson, L. Crowdsourcing in health professions education: What radiography educators can learn from other disciplines. Radiography 2019, 25, 164–169. [Google Scholar] [CrossRef] [Green Version]
  68. Penciner, R. Crowdsourcing: An instructional method at an emergency medicine continuing education course. Can. J. Emerg. Med. 2015, 17, 433–436. [Google Scholar] [CrossRef] [Green Version]
  69. Chen, C.; White, L.; Kowalewski, T.; Aggarwal, R.; Lintott, C.; Comstock, B.; Kuksenok, K.; Aragon, C.; Holst, D.; Lendvay, T. Crowd-Sourced Assessment of Technical Skills: A novel method to evaluate surgical performance. J. Surg. Res. 2013, 187, 65–71. [Google Scholar] [CrossRef] [PubMed]
  70. Dai, J.C.; Lendvay, T.S.; Sorensen, M.D. Crowdsourcing in Surgical Skills Acquisition: A Developing Technology in Surgical Education. J. Grad. Med. Educ. 2017, 9, 697–705. [Google Scholar] [CrossRef] [Green Version]
  71. Ma, X.; Cackett, M.; Park, L.; Chien, E.; Naaman, M. Web-Based VR Experiments Powered by the Crowd. In Proceedings of the 2018 World Wide Web Conference, Lyon, France, 23–27 April 2018; pp. 33–43. [Google Scholar] [CrossRef] [Green Version]
  72. VRChat. Available online: (accessed on 29 March 2021).
  73. Saffo, D.; Yildirim, C.; Di Bartolomeo, S.; Dunne, C. Crowdsourcing Virtual Reality Experiments using VRChat. In Proceedings of the Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems, Honolulu, HI, USA, 25–30 April 2020; pp. 1–8. [Google Scholar] [CrossRef]
  74. Cassidy, K.C.; Šefčík, J.; Raghav, Y.; Chang, A.; Durrant, J.D. ProteinVR: Web-based molecular visualization in virtual reality. PLoS Comput. Biol. 2020, 16, e1007747. [Google Scholar] [CrossRef]
  75. About—360visi. Available online: (accessed on 6 March 2023).
  76. Taylor, M.; Wharrad, H.; Konstantinidis, S. Immerse Yourself in ASPIRE–Adding Persuasive Technology Methodology to the ASPIRE Framework. Lect. Notes Netw. Syst. 2022, 390, 1106–1117. [Google Scholar] [CrossRef]
  77. Inamura, T.; Mizuchi, Y.; Yamada, H. VR platform enabling crowdsourcing of embodied HRI experiments—Case study of online robot competition. Adv. Robot. 2021, 35, 697–703. [Google Scholar] [CrossRef]
  78. Pears, M.; Yiasemidou, M.; Ismail, M.A.; Veneziano, D.; Biyani, C.S. Role of immersive technologies in healthcare education during the COVID-19 epidemic. Scott. Med. J. 2020, 65, 112–119. [Google Scholar] [CrossRef] [PubMed]
  79. Nanome Virtual Reality for Academic Researchers, Educators, and Students. Available online: (accessed on 6 March 2023).
  80. About|Osso VR Is Transforming Training and Assessment for Healthcare Professionals Everywhere. Available online: (accessed on 6 March 2023).
  81. Brooke, J. SUS-A quick and dirty usability scale. In Usability Evaluation in Industry, 1st ed.; Jordan, P.W., Thomas, B., McClelland, I.L., Weerdmeester, B., Eds.; CRC Press: London, UK, 1996; ISBN 9780429157011. [Google Scholar]
  82. Szajna, B. Empirical Evaluation of the Revised Technology Acceptance Model. Manag. Sci. 1996, 42, 85–92. [Google Scholar] [CrossRef]
  83. Dafli, E.L.; Bamidis, P.D.; Pappas, C.; Dombros, N. Design and Development of a Pilot on Line Electronic OSCE Station for Use in Medical Education. IFMBE Proc. 2010, 29, 975–978. [Google Scholar] [CrossRef]
  84. Murtza, R.; Monroe, S.; Youmans, R.J. Heuristic Evaluation for Virtual Reality Systems. Proc. Hum. Factors Ergon. Soc. Annu. Meet. 2017, 61, 2067–2071. [Google Scholar] [CrossRef]
  85. Calder, B.J. Focus Groups and the Nature of Qualitative Marketing Research. J. Mark. Res. 1977, 14, 353. [Google Scholar] [CrossRef]
  86. Vince, R. Behind and Beyond Kolb’s Learning Cycle. J. Manag. Educ. 1998, 22, 304–319. [Google Scholar] [CrossRef]
  87. Dunleavy, M. Design Principles for Augmented Reality Learning. Techtrends 2013, 58, 28–34. [Google Scholar] [CrossRef]
  88. Cuendet, S.; Bonnard, Q.; Do-Lenh, S.; Dillenbourg, P. Designing augmented reality for the classroom. Comput. Educ. 2013, 68, 557–569. [Google Scholar] [CrossRef] [Green Version]
  89. Johnson-Glenberg, M.C. Immersive VR and Education: Embodied Design Principles That Include Gesture and Hand Controls. Front. Robot. AI 2018, 5, 81. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  90. Andersson, E.A.; Frödin, M.; Dellenborg, L.; Wallin, L.; Hök, J.; Gillespie, B.M.; Wikström, E. Iterative co-creation for improved hand hygiene and aseptic techniques in the operating room: Experiences from the safe hands study. BMC Health Serv. Res. 2018, 18, 2. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  91. Dollinger, M.; Lodge, J.; Coates, H. Co-creation in higher education: Towards a conceptual model. J. Mark. High. Educ. 2018, 28, 210–231. [Google Scholar] [CrossRef]
  92. Ranjbarfard, M.; Heidari, M.; Sureshjani, M.H. Offering a framework for value co-creation in virtual academic learning environments. Interact. Technol. Smart Educ. 2018, 15, 2–27. [Google Scholar] [CrossRef]
  93. Radu, I.; Yuan, J.; Huang, X.; Schneider, B. Charting opportunities and guidelines for augmented reality in makerspaces through prototyping and co-design research. Comput. Educ. X Real. 2023, 2, 100008. [Google Scholar] [CrossRef]
  94. Ribes-Giner, G.; Perello-Marín, M.R.; Díaz, O.P. Co-creation Impacts on Student Behavior. Procedia-Soc. Behav. Sci. 2016, 228, 72–77. [Google Scholar] [CrossRef] [Green Version]
  95. Buchner, J.; Kerres, M. Students as Designers of Augmented Reality: Impact on Learning and Motivation in Computer Science. Multimodal Technol. Interact. 2021, 5, 41. [Google Scholar] [CrossRef]
Table 1. Eightfold policy analysis framework for VR educational content creation challenge.
Table 1. Eightfold policy analysis framework for VR educational content creation challenge.
Define the problem
  • There is too little access to quality VR resources for medical learners
Assemble the evidence
  • Literature research on experiential medical education
  • Commercial entities in immersive health education
Construct alternatives
  • Maintain the standard development pipeline
  • Educators and technologists co-design resources
  • Crowdsource resource design
  • Buy-in services from companies
Select the criteria
  • Efficiency
  • Educational Efficacy
  • Acceptability from users/learners
Project the outcomes
  • Assess impact on curriculum
  • Assess speed of resources development
  • Assess quality of resources development
  • Assess cost of resources development
  • Assess the level the resources are tailored to the need of the learners
Confronting the trade-offs
  • Digital skills overheads
  • Systemic changes in the academic world
Decision Making
  • Even though co-creative VR resource implementation approaches have some advanced digital literacy overheads they are an effective pathway to the prolific use of immersive media in medical education
  • Co-creative resource development should embed quality assurance processes that maintain high level of technical and topical fidelity, and certifies resources against specific educational use cases, target groups, or even discrete episodes of learning
  • Participatory design methods should be introduced to educators and learners always in conjunction with a training framework in the supporting and enabling technologies that are required for them
Sharing the results of the process
  • Create a set of recommendations for co-creative resource development pipelines
Table 2. Policy alternatives for immersive resources development.
Table 2. Policy alternatives for immersive resources development.
Maintain the standard development pipeline (status quo) In-house development of VR resources using topical experts for consultation and technology experts for design and implementation, with external assistance where required[13,47,55,56,57,58]
Educators and technologists co-design resourcesEducators and or learners collaborate to design the educational material. Technologists provide infrastructure for collaborative VR resource design and implementation. (e.g., user-friendly editing environments, new asset creation, etc.)[35,36]
Crowdsource resource designSimilar to the co-creative approach, the design process is distributed amongst educators. Technical implementation is conducted by interested members of the technologists’ community through social media engagement and smaller, usually community-based or “in-kind” reimbursement.[59,60,61,62,63,64,65,66,67,68,69,70,71,72,73]
Buy-in services from companiesSimilar to status quo. Core difference is that technical implementation is conducted through a different business entity, which elicits design requirements from the educational institution
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Antoniou, P.E.; Pears, M.; Schiza, E.C.; Frangoudes, F.; Pattichis, C.S.; Wharrad, H.; Bamidis, P.D.; Konstantinidis, S.T. Eliciting Co-Creation Best Practices of Virtual Reality Reusable e-Resources. Virtual Worlds 2023, 2, 75-89.

AMA Style

Antoniou PE, Pears M, Schiza EC, Frangoudes F, Pattichis CS, Wharrad H, Bamidis PD, Konstantinidis ST. Eliciting Co-Creation Best Practices of Virtual Reality Reusable e-Resources. Virtual Worlds. 2023; 2(1):75-89.

Chicago/Turabian Style

Antoniou, Panagiotis E., Matthew Pears, Eirini C. Schiza, Fotos Frangoudes, Constantinos S. Pattichis, Heather Wharrad, Panagiotis D. Bamidis, and Stathis Th. Konstantinidis. 2023. "Eliciting Co-Creation Best Practices of Virtual Reality Reusable e-Resources" Virtual Worlds 2, no. 1: 75-89.

Article Metrics

Back to TopTop