Pedagogy before Technology: A Design-Based Research Approach to Enhancing Skills Development in Paramedic Science Using Mixed Reality

Abstract

In health sciences education, there is growing evidence that simulation improves learners’ safety, competence, and skills, especially when compared to traditional didactic methods or no simulation training. However, this approach to simulation becomes difficult when students are studying at a distance, leading to the need to develop simulations that suit this pedagogical problem and the logistics of this intervention method. This paper describes the use of a design-based research (DBR) methodology, combined with a new model for putting ‘pedagogy before technology’ when approaching these types of education problems, to develop a mixed reality education solution. This combined model is used to analyse a classroom learning problem in paramedic health sciences with respect to student evidence, assisting the educational designer to identify a solution, and subsequently develop a technology-based mixed reality simulation via a mobile phone application and three-dimensional (3D) printed tools to provide an analogue approximation for an on-campus simulation experience. The developed intervention was tested with students and refined through a repeat of the process, showing that a DBR process, supported by a model that puts ‘pedagogy before technology’, can produce over several iterations a much-improved simulation that results in a simulation that satisfies student pedagogical needs.

1. Introduction

In health education, there is growing evidence that simulation improves learners’ safety [1], competence, and skills [2], especially when compared to traditional didactic methods and/or no simulation training [3]. Of significant importance to the health profession is airways management [4], where inadequate skill and poor judgment can lead to patient complications and death [5]. Airways management simulation education and hands-on training builds essential skills [4,6] and changes attitudes and behaviour for all health professionals. This is especially true for trainee paramedics studying high-priority invasive skills, such as direct laryngoscopy with foreign body removal in pre-emergency care [7], where students require confidence and experience to execute skills correctly [8].

However, this approach to simulation becomes difficult when students are studying at a distance [9]. The specific context of this study is an introductory paramedic science class studied by students at a distance. In this class, learning is considered to be an active process influenced by prerequisites of the learner [10] and the class requires numerous “hands-on” exercises to learn the skills required to be a paramedic. Yet, for many distance students, these skills cannot be provided until they attend residential school, often at the end of a semester of study in a location remote to the university, devoid of the tools needed for skills practice.

Hence, a project was funded by a faculty collaboration between two Australian universities to look at how these pedagogical problems could be solved. The aim of this project was to answer the research question: “How does the use of mixed reality, especially three-dimensional (3D) printing, augmented virtuality, and augmented reality simulation, affect skills development in paramedic science?” The result was that mixed reality could be used to assist with skills development for students studying paramedic science at a distance. Results of the student response to this work are available in [11,12] and show a statistically significant improvement for students that trained with the simulation prior to coming to the residential school.

However, within these papers, little attention is given to the process followed to develop the simulation over several iterations, and the data that was collected to inform this process. Hence, this paper describes the design-based research process that was used to develop the simulation, and presents the data that was used to inform this process including the pedagogical and design choices. It is anticipated that this discussion will provide a framework and process for future mixed reality researchers to follow in designing mixed reality education interventions using design-based research (DBR).

2. Literature Review

Across the education technology discipline, there is often a misconception that technology means that one must always apply the latest technology to enhance learner attitudes [13]. Educators are more interested in illustrating technology use rather than thinking about the underlying pedagogical problem and how technology is best used to improve and solve this problem: or thinking pedagogy before technology. Often, this research relies on learner perceptions based on ill-informed qualitative “happy sheets” promising a silver bullet rather than on the quantifiable learning outcomes [14].

Recently, a new technology continuum of mixed reality (MR) has emerged on the education technology researcher’s radar. The technology appears to offer many solutions to the problems encountered in the classroom, as detailed in [15] and [16]. However, as per above, to date most of the research on the use of MR for education has been one off pilot studies highlighting increasing evidence that education is moving towards the applied use of mixed reality (MR) for application in the classroom [17] without embedding the technology use in an accepted and tested research method with statistically significant changes to the learning in the classroom. The reason for this is that MR is a quickly evolving area of research and development, often driven by a particular application or simulation that captures stakeholder interest before receding back into the background as it is replaced by something else with a different set of characteristics [18]: essentially a gimmick.

As it currently stands, the most commonly used definition of MR is presented by Milgram and Kishino [19], and is situated within the reality-virtuality continuum. This allows for the construction of different environments along the continuum from physical environments to virtual reality (VR) (see Figure 1). The continuum starts with the real physical environment on the left-hand side before moving through augmented reality (AR), where the real world is augmented by the virtual, and augmented virtuality (AV), where the virtual world is augmented by the real, to arrive at virtual reality or virtual environments on the right-hand side of the continuum.

As noted previously [18], research work in MR implements a variety of different software and hardware solutions to place MR in the classroom. In recent implementations, these research efforts often involve the use of commercially available equipment; unlike the situation described by Azuma [15] twenty years ago (as well as in the follow-up paper [16]), where many of the hardware and software solutions were customised tools developed by the military or by research institutions, these days most solutions could be considered a combination of commercially available off-the-shelf equipment with specialised software to suit the education classroom.

Research efforts also cover a wide variety of different uses, from studies that use immersive 360-degree video [20], to work that uses smart technology to augment physical devices [18], to other work that uses a combination of 3D printing (3DP) and AR/VR [20,21,22], all interchanging the MR continuum themes. An issue therefore emerges that this set of themes, and the constant interchange between them, provides little informed information to provide educational researchers on how these themes differ and what characteristics they have [23].

To help situate MR as a design method to enhance pedagogy, we must first distil out which of these devices are used in the educational technology research space, as well as what features are desirable characteristics of those devices and the software supporting them to produce a prototype for educational technology research. To facilitate this, the authors examined literature of MR articles produced in the 23 years since the Milgram continuum was proposed. Each of these papers was selected as an example of a paper that: firstly, cites the Milgram and Kishino paper [17]; secondly, discusses AR, VR, or MR; thirdly, is from education; and finally, includes the term “review”.

Below is a summary of the relevant data from this collection of review articles, collected through Google Scholar, EBSCOHost, and ProQuest using a combination of the search terms: ‘augmented reality’, ‘virtual reality’, ‘mixed reality’ AND ‘education’, and ‘learning’ AND ‘review’. These review papers were ultimately selected and analysed to provide a base for the experimental design presented, with paper citations listed in the left-hand column of

Table 1. An Analysis of Review Papers on AR/virtual reality (VR)/MR.

Research Paper	Focus (AR/VR/MR)
Wu, Lee, Chang, and Liang (2013) [24]	AR
Billinghurst, Clark, and Lee (2015) [25]	AR
Dunleavy and Dede (2014) [26]	AR
Cheng and Tsai (2013) [27]	AR
Bower et al (2014) [28]	AR
Bacca, Baldiris, Fabregat, and Graf (2014) [29]	AR
Phon, Ali, and Halim (2014) [30]	AR
Lukosch, Billinghurst, Alem, and Kiyokawa (2015) [31]	AR
FitzGerald, Ferguson, Adams, Gaved, Mor, and Thomas (2013) [32]	AR
Nincarean, Alia, Halim, and Rahman (2013) [33]	AR
Kipper and Rampolla (2012) [34]	AR
Craig (2013) [35]	AR
Lee (2012) [36]	AR
Yuen, Yaoyuneyong, and Johnson (2011) [37]	AR
Zhou, Duh, and Billinghurst (2008) [38]	AR
Hussien and Natterdal (2015) [39]	VR
Ott and Friena (2015) [40]	VR
Merchant et al (2014) [41]	VR
Mikropoulas and Natsis (2011) [42]	VR
Hughes, Stapleton, Hughes, and Smith (2005) [43]	MR
Bower, Dalgarno, Kennedy, Lee, and Kenney (2015) [44]	MR
Bower and Sturman (2015) [45]	MR
Lindgren and Johnson-Glenberg (2013) [46]	MR

. Although not an exhaustive literature survey, this analysis of review papers gives a good indication of the state of the industry on which future work can be built.

Looking at these papers, it is clear that the MR narrative currently in the research community is wide, with many different approaches using the technology in different ways. In the commercial world for MR, there also appears to be significant variation in use, with different organisations taking different approaches to the problem, developing different hardware and software solutions and trying different things, but all under the general heading of MR. This makes it hard for educators to filter through this space for the best technology to use for their classes, and also makes it harder to connect the technology with the underlying pedagogical problem. For this reason, it is worth taking some time to unpack the mix of different MR implementations and identify the themes and characteristics that they exemplify to assist classroom practice.

Interestingly, whilst all these implementations fit broadly under the Milgram continuum, they represent vastly different implementations of what MR is, perhaps due to significantly different goals. Specifically, when one looks at the examples given, it is clear that there are certainly two main approaches, each mapping to the different extremes of the existing continuum.

The first looks at using presentation technology to augment the world through a headset or mobile phone that can provide a digital overlay onto real reality. HoloLens, Google Tango, and the AR headset-based head research efforts (such as those presented by [24]) are all examples of this type of approach.

The other removes the world entirely, immersing the user in a different world and removing them from the existing one. Whilst at first this would appear to be the epitome of the extreme right-hand side of the Milgram continuum, it is clear from recent efforts that some inclusion of the real world is required, with equipment developed by the company HTC using room sensors to allow the user to move around, Oculus including touch controllers to allow real-world hands to enter the virtual world, and systems, such as the MEteor simulation described by [46], that allow a user’s body to enter into the virtual world with them. This correlates with conclusions made by [40], that, rather than being totally virtual, this style of MR, sitting at the right end of the continuum, requires a combination of both the physical and the virtual depending on the situation.

Geolocation also appears to be an important part of many MR implementations. As noted, many MR implementations claim that geolocation provides them with an AR approach, as it allows them to have an understanding of the real world. In the research literature, geolocation is also often noted as one of the different types of MR system, and is often contrasted with marker-based systems as a markerless alternative. For instance, [33] identify this as a significant differentiator between MR implementations, and [34] also notes that many systems use geolocation as part of their approach.

Another important consideration when looking at these examples is the ability of the device to contextualise the world, as identified by [26]. Whether this is through geolocation, depth-sensing cameras, or image marker / Quick Response (QR) code technology, the ability of the MR hardware to understand the world around it provides an important distinction within the continuum, as identified by [29]. Regardless of whether an approach is more augmented or virtual, without an understanding of the world around them, devices are by necessity limited in how real they can appear to be.

Finally, in recent times, collaboration (multi user communication) also appears to be an important consideration for this technology, as noted by [30,31]. Whilst not strictly part of the AR experience, the ability of MR to allow individuals to work together when interacting has been addressed in the literature (see [34,44] for example) and in the commercial space (see demonstrations by Facebook/Oculus in 2016, [47]). The focus appears to be on building a system where the presence of the other users can be felt, something that is well within the capabilities of most modern AR systems.

Additionally, there is an emerging combined category, where many of these approaches are welded together, providing an augmentation of the space where the physical objects have value of their own, but this can be augmented using a physical object to add extra value to the world [48]. In this case, the MR device not only understands the physical world around it, but can also manipulate and represent objects within that world. This can be done with real objects, 3D printed objects, or specialised Internet of Things (IoT) devices that interact with the MR equipment. In a way, this aspect of the technology allows developers to augment the AR, adding an extra layer to the existing model.

3. Theoretical Framework

In addition to identifying the right technology to use for a mixed reality intervention, a particular challenge in developing mixed reality education solutions is that the intervention must be tested with stakeholders, including students, and must be cycled on iteratively to ensure that the best final design is developed to address student needs. Given the breadth of different solutions to mixed reality education problems exposed in the literature, it is clear that a methodology needs to be selected that can a) allow an iterative approach of testing and refinement; and b) progress the overarching theory and narrative on how mixed reality can be used effectively in the classroom. In particular, it is important that the theoretical framework selected allow for the data collected to be fed back into the growing literature on mixed reality education and make a contribution to how further interaction can be structured in the future: it is in this way that the work transcends a single case study and becomes useful as a study to push forward work in the area as a whole.

To this end, a design-based research (DBR) methodology [49] was selected for the project. Building on the work of Kelly [50] that discussed the use of a conceptual structure to elevate the work from a set of methods to a methodology, this work takes the concepts highlighted by the literature review and incorporates them as prompts in the analysis stage of the DBR process. Using the Sandoval [51] method of embodied conjecture, this then allows these concepts to be embodied in the design solutions developed, and theoretical knowledge about mixed reality education in general to be extracted at the conclusion of each step.

In addition, to address the need for an iterative approach, a model was needed that provided structure for the analysis and design as well as a feedback mechanism. Specifically, the four steps of the DBR methodology detailed by Reeves [52] were followed through the analysis of the problem and design of the simulation solution (as detailed in the section below on experimental design), followed by the iterative implementation of that solution into the classroom by a discipline expert practitioner positioned to evaluate the effectiveness of the solution who provided detailed feedback on the design in the first loop pilot study [11]. This then results in a loop back to design refinement and further iterative testing (see Figure 2).

Finally, as part of this approach, a further model was developed to position the DBR approach within the education context of a classroom. In particular, this model sought to enhance the steps of the DBR approach by connecting them to classroom practice for discipline experts. This model was based on the common perception that often a technology is selected prior to a problem being identified, and then the technology is retrofit onto the problem, or a problem is found that fits the technology [13]. To avoid this happening, a practical model was developed for building technology solutions for the classroom.

Looking to put ‘pedagogy before technology’, this model encourages educators to think about the pedagogical issue in their classroom first, then think about a solution to that problem (free of technology limitations), and finally look at how technology can help. Similar to the overarching DBR framework, this model is iterative and encourages returning to step 1 after testing has been carried out to refine and reflect. Figure 3 gives an overview of these steps.

To provide context for working through this model, the authors of this paper have distilled out a set of guidelines for educational designers, based on our experience as educators, to work through the model step-by-step, developing a final solution that uses technology and can be fed back into the DBR process (Figure 4).

4. Experimental Design

Based on the literature review of mixed reality and the application of the DBR process (results below), a simulation was built. As outlined in the theoretical framework section, initial considerations for how this simulation was to be built were based on the embodied conjectures about mixed reality education extracted from the comprehensive literature review of existing mixed reality initiatives in this area. These conjectures then informed the analysis of the problem to develop a tool that met the needs of the discipline educator whilst still fitting within the existing framework of mixed reality and education and the technology possibilities within the area. Through this process, it was anticipated that a prototype could be built to assess the current theoretical assumptions made in mixed reality education and, through research results, contribute to the literature on how future interventions might be developed.

Please note that much of this simulation design has been reported in previous papers [11,12], so this section in this paper will focus more on the differences between the two loops of the DBR process and how they were informed by user feedback and the hybrid DBR and pedagogy before technology model.

As the aim of the simulation was to provide distance paramedic students with a cost-effective haptic, visual, and auditory feedback mechanic to assist in learning and practising the airway skills in line with the previous simulation fidelity research [53,54,55], the project used 3DP representations of the laryngoscope, Macintosh blade (size 4), and Magill forceps, 3D modelled on a 1:1 scale (Figure 5). Tools were split into pieces to fit onto smaller consumer-level 3DP equipment.

These physical representations allow for a user’s perception to be primed before they carry out the simulation through haptic and visual connection to the tools [55], which was identified as important in the literature review [56]. The design of these tools remained reasonably consistent over the two loops of the DBR process [57], but the markers attached to the tools were refined over the two experiments based on participant feedback, going from smaller cubic markers in the first pilot study (see images in [11]) to a larger flat marker design (Figure 6). 3D models of the tools can be downloaded from [58].

To enable the 3D printed objects to be seen in the simulation, each tool had an AR marker attached for tracking purposes. The AR markers used on the original pilot simulation [11] were six-sided, 3 cm cubes, which allowed the tools to be tracked from any orientation as long as one of the sides was visible to the device’s camera. However, as the experiment evolved, these markers were changed to flat, 6 cm square markers [12], which improved tracking and allowed a label for each tool to be added to the marker. This type of tracking utilised Vuforia’s Image Target marker (www.vuforia.com), situated in Unity (www.unity3d.com). The abstract pattern used for the background of the markers was generated by the Augmented Reality Marker Generator by Brovision (www.brosvision.com) to ensure markers were unique and not misinterpreted by the Vuforia application programming interface (API).

The simulation proposed in the study requires the use of both hands to practise and execute the skills. To allow the learner’s hands to be free throughout the simulation, a smart phone needs to be mounted in front of his/her face. This also allows for the correct view point: looking down the throat of the airway manikin. This design also evolved over the experimental timeline, beginning with a hat-mounted device in the first pilot study [11] before moving to a more authentic hands-free stereoscopic experience that allowed the user to hold the instruments and visualise the simulation environment in VR, providing depth to the simulation scene (Figure 6) and adding a combination of the VR and AR components identified as important in the literature review. This was achieved using a low-cost ColorCross universal mobile phone VR headset (based on the Google Cardboard design).

For the entire run of the trials, the simulation application was available for selected students to download (see [59] for a video demonstration) from both the Apple App Store for iPhone 5+ or the Google Play Store for Android Version 4.4+. However, the version of the app changed between loops of the DBR process based on student feedback. In particular, as well as adjusting for the new marker sizes and the use of the ColorCross headset, a tutorial was also added to the simulation in the second iteration of the testing to familiarise students with the tools before performing the airways management task. A link to download the AR markers and see more project information was also added, as it became apparent that users outside of the paramedics trial were downloading the app and therefore needed access to print the tools and the markers.

Another significant difference in the design of the simulation between loops of the DBR process was in the presentation of the AR and VR environment. In the original simulation, the outside world was always visible, even when doing the skill, as is the case with true AR as identified in the literature reivew. However, for the second loop, based on user feedback, the user is asked to present the tools, but once this connection is established, the AR camera is removed, giving the user a wholly virtual experience but with the physical tools represented by their virtual counterparts (AV). This transition allows the user to maintain a connection to the physical objects as the simulation enters the AV experience, removing the visible world from the user’s perspective, concentrating and focusing the user on the task at hand, and reducing cognitive load [56,57] and working capacity [60].

The user is then guided through the process of laryngoscopy with foreign object removal (Figure 7). This process itself remained relatively consistent amongst the two loops of the DBR trial, but for the second trial a section was added to give the user a tutorial in using the instruments and making sure they could use the instruments before moving on. This resolved issues in transitioning the user from AR to AV by connecting them with their virtual arm and guiding them in moving the AV object in the VR presented environment. The user was able to complete the tutorial before they moved on, or if they were simply practicing the skill with familiarity (they had completed the tutorial) they could access the direct skill which was the same as previous.

In the development of this section of the app, the focus on the simulation to be functionally task aligned [50] meant that only the pertinent information relating to the key learning outcomes was included. For the developed simulation, this refers to only the key indications from the Australian, Queensland Ambulance Service (QAS) (2016) Airways Management Guide [61]. If at any time the user incorrectly positions the instruments, it presents a red guide. The user has a few seconds to correct the mistake before the simulation restarts showing the real world again (via the camera) so that he/she can re-orientate his/her view of the tools before continuing.

One significant difference between the first and the second loop was in the tracking of the instruments. For the first loop of the DBR process, both instruments were tracked at all times, and so were required to be held within view of the camera. However, observation noted that often students completing the first steps (which do not use the forceps) were dropping these tools out of view and therefore dropping out of the simulation prematurely. Hence, for the second iteration, only the scope was tracked for the first part of the simulation, with the forceps only being tracked once they are required as part of the skill.

5. Materials and Methods

To recruit participants for the study, an invitation was sent out to students studying the paramedics skills course and a list of potential candidates was developed, including location details for each candidate and their mobile digital device ownership. Ethics approval was obtained from the hosting institution to issue this invitation and conduct the related study. All 2nd year distance paramedic students were invited to participate across the two rounds of the intervention (N = 159). The list of students that replied was then checked and edited to ensure only students with compatible phones (able to install the app) were included in the study, and then a stratified sample of approximately 30 candidates from each round was selected from this list in a blind fashion without the participation of the discipline expert. These students were then shipped a kit consisting of the 3D printed tools, a ColorCross headset or hat mount, and instructions for downloading and installing the app to a smartphone. The number of students selected to receive the tools was limited by the availability of 3D printed components and headsets. These students were given instructions on how to use the tools and encouraged to practice for 1 week prior to the residential school scheduled for late in the term.

Two loops of the study were conducted. Results of the first loop were presented in [11], with follow-up data from the second loop presented in [12]. Over the two loops, there was a total pool of 159 students, of which 137 students participated. From this pool, 55 students received the simulation for testing prior to coming to the residential school. Once they arrived at the residential school, all participants in the study (including both those who received the tools beforehand and those who did not) were pre-tested using airway mannequins by qualified on-road paramedics, with advance care paramedic level 2 or above on four separate key performance indicators across the two selected airways skills, before any face-to-face skills training took place. The performance indicators included whether or not they:

• placed the laryngoscope in the right side before performing lateral sweep, (simulation step 3),
• elevated the laryngoscope without levering of the teeth (simulation step 4),
• adequately visualised the obstruction and safely removed it (simulation step 5), and/or
• removed the laryngoscope without damaging structures (simulation step 6).

Participants either passed or failed each key performance indicator and were awarded a score of 1 or 0, respectively. Performances on all tasks were combined and assessed collectively, with each participant receiving a score ranging from 0 (failed all four key performance indicators) to 4 (passed all four key performance indicators). This data is reported on in [11,12], with a statistically significant difference in performance noted in [12].

Traditional skills training was then conducted as part of the residential school, with all students given access to the 3DP tools and simulation throughout the period to comply with ethics. At the end of the residential school, all students were surveyed on their experience with the tools, and both quantitative and qualitative data was collected from students, as well as data to inform the DBR process. In addition to data from students, data from observation by paramedic staff and educators was also collated. This data was solidified in observational feedback and staff memos and was used to inform the revise and reflect the final step of the DBR process. This data from the qualitative interview of students and from staff is reported below.

6. Results

The initial data that was used to inform this DBR process was collated from student surveys of the unit before the intervention was introduced. In line with the DBR process (Figure 2), as well as the pedagogy before technology model (Figure 3), pedagogical problems were sought in the classroom that could be solved by technology. An analysis of the student surveys presented provided the following student feedback about the problem (

Table 2. Data Supporting “What is the problem?” step of the Pedagogy before Technology model, as per Figure 3.

Feedback on the Problem
“this course is a ‘skills’ learning course … there should be a way for us to actually get more time doing skills”
“distance students at a severe disadvantage … I am missing out … you can read about the skills but it is impossible to get feedback and to know if you’re doing it right”
“no substitution for experience”
“any software or equipment to at least go through the motions of doing the skills?”

).

Based on this identified pedagogical problem, solutions were discussed. As per the pedagogy before technology model, the discipline expert was asked to consider the solution without regard to the technology, and then the mixed reality team was able to suggest technology solutions that would help to solve this problem.

Table 3. Data Supporting “How can I solve the problem?” step of the Pedagogy before Technology model, as per Figure 3.

Feedback on the Solution.
“we need every student to have a portable airways mannequin and cheap tools to practice the skills, but cost is prohibitive … residential schools work … but not held early enough for students”
“students would need both hands free, so the phone would need to be mounted in their eyeline”

is an extract from notes of conversations over video conference on the possible parameters of a solution to the pedagogical problem.

The other members of the team (educational technologist and technology expert) then discussed these solutions and identified suitable technology that could meet the needs of the solution, and was technically and cost viable for the trials. Technology was selected to ensure it met the parameters of the problem identified.

Table 4. Data Supporting “How can technology help?” step of the Pedagogy before Technology model, as per Figure 3.

Feedback on the Use of Technology
“tools could be provided to students by 3D printing them (cost/set) ~ $1 AUD + postage”
“we can provide a hat and 3D printed hat mount to allow students to have hands free” “A simulation app can then be constructed using a free game engine [Unity3d] that could observe and monitor [learner] skills”

is data from the discussion about the use of technology.

Based on this discussion, an initial solution was then developed, and feedback was taken from both the staff and the students.

Table 5. Data Supporting “Revise and reflect” step of the Pedagogy before Technology model, as per Figure 3.

Feedback on the First Intervention
[Learners] were very excited … [practice] the skills at home … [learner] more involved in the course and less isolated
[Observers] found most [learners] struggled with the setup of the equipment and progression through the required steps … especially when introducing the Magill forceps and removing the foreign body
[Learners] commented “my hands seem to pass by the [simulated airways manikin]” … “spent too much time focusing on the markers and not on the [simulated airways manikin] … resulting in frustration when the simulation would present red boxed and restart
Many [learners] commented that they did not get around to using the [simulation] highlighting time struggles and being “extra” work [Head Paramedic] on reflection should have encouraged more frequent use of the simulation with a reminder and linkage to the learning tasks

is a sample of feedback that informed the reflection and refinement stage of the DBR process.

The system was then redeveloped and further testing performed, resulting in the student results published in [12]. As part of this, the pedagogy before technology model continued to be followed.

Final results from the second trial showed that students enjoyed using the tool, and a general acceptance of the tools, especially by those students who spent more time with them in the pre-residential school simulation training, with scores for question 6 on the final survey, “The 3D distance skill development tools complemented the practical sessions well”, and question 7, “I valued the addition of the 3D distance skill development tools to the course” being especially favourable, with over 75% of students agreeing with these statements. The responses to question 13, “I would like to see more courses using multiple forms of media (i.e., 3D printed tools and smartphone simulation applications)” were also promising for future work, with 100% of students who used the tools agreeing or strongly agreeing with this statement.

7. Discussion

These results provide a lot of detail in answer to the research question: How does the use of mixed reality, especially 3D printing, augmented virtuality, and augmented reality simulation, affect skills development in paramedic science?”

Overall, feedback from the quantitative data shows that the use of mixed reality for skills development can have a positive impact for students in paramedic science. In unpacking the question, however, it can be seen that the way that this was achieved was by executing a process that involved understanding the problem related to skills development, constructing a solution to address that problem, and the use of appropriate technology to scaffold that solution. This occurred over multiple iterations of the DBR cycle, as per Reeves [52], where the understanding of the problem, the solution, and the use of technology was refined to produce an outcome that provided the most positive impact for the skills development task.

Specifically, the data collected from students and staff about the problem being faced was invaluable in making sure that pedagogy was put first, in line with the ‘pedagogy before technology’ model, identifying the major pedagogical problem students faced. Comments such as “there should be a way for us to actually get more time doing skills” and “any software or equipment to at least go through the motions of doing the skills” helped shape the team’s construction of the pedagogical problem and combined with the existing embodied conjectures [51] to include a mechanism for students to practice, as well as a model of the real tools, in line with [26,28,48].

Within the solution development, the use of the pedagogy before technology technique of removing the technology from the discussion in the first instance also produced useful results. A review of the transcripts from the conversations with the discipline expert show that they were able to imagine an idealistic solution that addressed the pedagogical problem, free of constraints relating to technology, as noted in comments, such as “we need every student to have a portable airways mannequin”, which address this idealistic view and are in line with the requirement for a comprehensive solution as identified by [53,54,55].

Finally, in developing the technology, the mixed reality team was also able to map these solutions to technical advancements in the way the mixed reality simulation was constructed. For instance, feedback from the discipline expert about the technology needing be hands-free led to the use of a head mount during the trial, which then progressed to being a stereoscopic headset, and comments about losing track of the forceps also resulted in a change to the system as the trials progressed, congruent with [56].

Throughout the trials, the DBR process also provided much assistance in the development of the experimental intervention by providing valuable feedback for reflection from students and staff. For instance, the DBR process led specifically to the change in marker design based on feedback from students that they “spent too much time focusing on the markers”, as well as the use of a combined AR and AV simulation framework for the second version of the app, in line with experiments involving cognitive load as per [9]. It also led to the introduction of a tutorial to help students become familiar with the system based on comments relating to the “set-up of the system” (as per [44]), and the use of a stereoscopic interface in later trials to help with student immersion in the simulation.

In terms of results, these changes, inspired and driven by the DBR process, resulted in improvements in student outcomes, allowing a statistically significant difference is skill acquisition for students that used the app in the second trial of the system (see [12]). Supported by the pedagogy before technology model that allowed the educator to work through the problem effectively, these are also the areas where the DBR process allows the research to give back to the mixed reality education community, by providing an advancement of the theory on the use of AR versus VR and the inclusion of scaffolding and material to help with cognitive load, that can support more refined and successful innovations in the area in the future, in line with the vision put forward by Kelly that the method should provide a methodological advancement of theory as well as practice [50].

8. Conclusions

This paper has presented the results of a two-loop, design-based research process for the development of a mixed reality simulation that uses 3D printed tools and augmented reality to improve skills development in students studying paramedic science at a distance. Through the use of these interventions, work has shown that student outcomes have improved and they have developed better skills in laryngoscopy by using the simulation. More importantly, through the use of the design-based research process, it has been shown that a process of incremental reflection and refinement can be useful in developing this type of simulation. Specifically, student and staff feedback can be used to refine the application to more closely address the pedagogical problem. Finally, through the use of pedagogy before technology, a design can be developed that presents the most appropriate solution for student improvement, focusing not on the technology but on the success of the student. Future work will continue to investigate the parameters of this work and how the mixed reality simulation can be adjusted to maximize usability and skill development. Through this work and the continued use of DBR as a methodology for not only incremental improvement, but also verification and establishment of theory, a greater understanding of the use and development of innovative technology tools and games simulation in the education space will be obtained, providing a foundation for future work in the area.