Design and Usability Testing of an Augmented Reality (AR) Environment in Pharmacy Education—Presenting a Pilot Study on Comparison between AR Smart Glasses and a Mobile Device in a Laboratory Course

Round 1

Reviewer 1 Report (New Reviewer)

Comments for the Review of Manuscript entitled “Design and usability testing of an augmented reality (AR) environment in pharmacy education – Presenting a pilot study on comparison between AR smart glasses and a mobile device in a laboratory course”

 

 

1.              General Comments

1.1.  This manuscript described a pilot study on applying AR educational technology in pharmacy laboratory setting.

1.2.  The aim of this project was to develop an AR environment suitable for teaching pharmacy and laboratory skills in a course focusing on microbiology and asepsis. 

 

2.              Specific Comments

2.1.  In the introduction, it mentioned that “Study programs now place emphasis on clinically-orientated practice and less on compounding-based practice.”  Then, it followed by the description of the challenges in physical laboratories.  It may be better for the authors to further describe the laboratory training the pharmacy education is as important as the clinical training.  Therefore, it is important to address the needs of laboratory training using technology during the changing time.

2.2.  The authors also mentioned about the ICT competency in the introduction.  Yet, the authors did not describe how ICT competency is important in the changing work environments.  In addition, how using AR environment could facilitate ICT competency in pharmacy education.  It is not clear how the use of AR, VR and MR impact of the students’ skills in the real pharmacy working environment.  Please elaborate.

2.3.  Please elaborate on the class size and the year-2 pharmacy programs.

2.4.  Are the TSE and ANX surveys validated previously, or you developed them for this project?  Please clarify

2.5.  Were there any components of your course that could not be feasible to use AR technology please?

2.6.  Any students who were not suitable to use the AR technology in your pilot study?

2.7.  Have you noticed any differences between the assessment of the quality of lab work between the AR glasses and the mobile device?

2.8.  In terms of the lab preparation time, did you notice any differences for both teachers and students?

2.9.  The use of AR glasses was in 2019 and the AR mobile device was in 2020.  The impact of COVID-19 pandemic may be higher in 2020.  Did you notice any differences?

2.10.               The interactive AR environment was developed with the University of Helsinki and the company.  I think this is great.  How often do you the AR environment should be evaluated and assessed?  Do you think this is sustainable?  Did you have another support team for your project at your institution?

 

 

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 2 Report (New Reviewer)

The study shows how various devices of modern technology can be applied in professional education. The authors managed to embrace many inside and outside controversies related to the topic.  The topic and the investigation is innovative  and worth disseminating.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 3 Report (New Reviewer)

It is a good work that presents a pilot study on the use of AR with smart glasses and mobile devices to support students in their work in a pharmacy laboratory course.

The paper, however, could be improved by clarifying two issues that, in my view, are not clearly stated:
1) in the introduction section: what exactly is the specific problem(s) being addressed and the research question(s). Also establish the relationship between the research question, the research objective (described in the last paragraph) and the problems, presented above, to be addressed (lack of individualized learning, lack of clear purpose, ICT competency, ...?).

2) It is then stated that the TSE and ANX questionnaires are directed only towards technology, why only tech? Wasn't it also to address the lack of individualized learning and clarity of purpose of the lab?

There are also two minor issues that the authors may wish to consider:
1) It would also be interesting for the reader to know how long it would take for the laboratory to work without AR.
2) I have not received Supplementary Material Video S2 (line 211). If it is not going to be provided, this mention should be removed.

Congratulations to the authors for the work and I hope they can continue exploring (and publishing) how to improve this use of AR to support students in their lab work.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 4 Report (New Reviewer)

 

The paper discusses the design of an AR environment for teaching lab skills in pharmacy education and presents a pilot study (N=36) on a comparison between AR smart glasses and a mobile device.

The paper is well written with very good use of English. It contains an adequate review of the literature on the subject of AR in education with an emphasis on pharmacy education.

The pilot study is well though out and executed and the results are presented and discussed appropriately.

The authors should pay some attention to the Discussion (Section 4) since it is the only section of the paper where there exist some minor language problems (missing words or incomplete sentences).

Overall, it is an easy to read and follow paper. Since the emphasis of the paper is more on the comparison of the tools used for implementing the AR environment regarding user experience, technological anxiety and self-efficacy, and less on the novelty of using an AR environment in pharmacy education, one could rightfully claim that the novelty of the paper is quite low. The authors should report whether other similar studies in other areas of education come to similar conclusions.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Round 2

Reviewer 1 Report (New Reviewer)

Comments for the Review of Manuscript entitled “Design and usability testing of an augmented reality (AR) environment in pharmacy education – Presenting a pilot study on comparison between AR smart glasses and a mobile device in a laboratory course”

1.                  General Comments

1.1.   This manuscript described a pilot study on applying AR educational technology in pharmacy laboratory setting.

1.2.   The aim of this project was to develop an AR environment suitable for teaching pharmacy and laboratory skills in a course focusing on microbiology and asepsis.

 

2.                  Specific Comments

2.1.   I am a bit confused with the revised manuscript.  Although the authors tried to respond to my previous comments, they have introduced new issues.  It was mentioned that lab training and knowledge of compounding is elemental in securing patient safety.  Please explain and elaborate. 

2.2.   How content knowledge in compounding enables high quality patient counselling in practice?

2.3.   Please elaborate the correlation of the graduates in different pharmaceutical tasks could relate to your current study design of using AR technology in pharmacy lab training.

2.4.   It is not clear how the use of AR, VR and MR impact of the students’ skills in the real pharmacy working environment.  Please elaborate.

 

2.5.   The current project only involves approximately 1/5 of the entire class.  Please evaluate the selection process of these students in the current pilot study.

 

 

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

This manuscript is a resubmission of an earlier submission. The following is a list of the peer review reports and author responses from that submission.

Round 1

Reviewer 1 Report

Design and usability testing of an augmented reality (AR) environment in pharmacy education – A comparison between AR smart glasses and a mobile device in a laboratory course

The work reports results of the design and usability tests of an augmented reality app to create an environment in pharmacy education to get pros and cons of using AR smart glasses and a mobile device in a laboratory course.

·       The sample is too small, and not balanced. It is mainly unbalanced in the use of technologies (just 30% using glasses). This could be immediately connected with one of the disadvantages of using glasses for AR.

·       For comparing objectives, it is not possible to use such different samples, unbalanced and too small.

·       Shouldn´t have all the students use both?

·       The instructional design of the app was not evaluated in order to be used as a good tool to test the comparison of both hardware technologies: glasses and mobiles.

·       Both technologies have pros and cons and some are evident, and these aspects were not clearly stated. They come like a mix. Just an example: glasses allow free hands for manipulating objects and functions on an interface. Is this important in the example? In fact, I do not understand exactly if the training to manipulating objects is importing or if it is only important to know the protocols, the procedures, etc. And if so, the simple manipulation of a button to get a link would be enough. The available video seems to point to the need of using both hands!!!!!

·       There is a very poor description of the lab work. What is AR adding to what??? A procedure guidance? Which kind of experimental manipulation? Or just questions and guidance? The video is not clear…for me.

·       Is the app creating a lab environment to get the user more immersed? Or is it allowing a sequence of procedures with links to question and feedback? Perspectives and totally distinct!

·       The AR and VR technologies, immersive technologies based on the ability to add virtual contents in both physical and virtual space, are very visual medium. Therefore, I do not understand how this work does not show us images. Only one is used for a pictorial flow chart!  

·       The statement “However, at the university level, AR technology is more often used in courses of specific disciplines, with the emphasis on communication, creative media, computer science, and information technology” is true in some sense BUT it is mandatory to know that those technologies are powerful in training activities. And this is valid not only in industry but also in health and teaching. The authors should find good examples for the bibliography.

·       Although, “AR as one of the most promising technologies [22], and together with VR, they “have the potential to be a standard tool in education””. This is an exaggerated perspective, not only due to their limitations but also and mainly, because the “real thing” will always the best solution. These technologies are, in fact, very powerful and could be a fantastic complement! And students found it, as reported by the authors “Students saw the AR environment as a useful supplementary tool, but they did not wish AR to completely replace traditional laboratory teaching and instruction”. Should be avoided such initial comment.

·       The AR environment also provided think-aloud questions (Table 2), which encouraged the students to reflect on their work with the knowledge learned previously during the course - Are the think-aloud questions used to give to the observers’ (teachers’????) insights into the students’ cognitive processes???? If so, why not explained in the methodology of the work?

·       Table 2 repeats 8 times “Think aloud”. Why not to use “Think aloud Question” on the top of the table, and avoid it on each question?

·       “Mobile devices were noted for their capability to give useful guidance that was beneficial to the students during work.” Which type of guidance? This is important once the open-ended questions were about guidance. I would like to have seen the video for mobiles…

·       “Students also pointed out the easiness and fast responsiveness of using QR codes” – for what are they using the QR code? A video was referred but I do not find it clear. To get the right question?

·       “usability problems have been recognized as one of the biggest barriers to using AR in educational settings [42].”. What about light problems?

·       “the fact that glasses are hands-free, and safer physical movement in the room without the need to look up” – do not forget that their field of view is another problem for many applications. Authors can argue that it is not for this specific case. I would like to be well informed about the particular case.

·       “However, for the first time, an AR environment was designed and introduced for teaching laboratory skills in pharmacy education at the university level.” In fact, I did not realize clearly which lab skills were trained. “. Can we consider a chemical lab so different from a pharmacy lab? I am not sure about other AR apps in Pharmacy but this argument should be not used because there many reported cases for chemical labs!

·       I would understand clearly the number of times students have used the AR. And for how long?

·       The instructional design is an important factor to inform about the real value of the tool. Or, if not done, then the knowledge gain by using the tool…if there is not any of these, why should we compare techniques?

 

The publication “Evaluating an online augmented reality puzzle for DC circuits: Students' feedback and conceptual knowledge gain” [related with a study of basic electrical circuits] based on a sample of about 450 students focus different aspects of this study. It also has many bibliography from other areas not only electrical. And consider the knowledge gain evaluation. Could be of interest to look at it and others.

As it was done in this referred work, if the methodology you have used is well defined, then it can be repeated again, and the sample can be increased. Results need to have some confidence

As being only another tool for teaching/learning with different advantages but always considered as a complementary tool, this referred study considered motivational factors (interest and perceived value), the students positive attitude when using the AR technology in training even if not familiar with it. The study revealed that the perceived value is the most important driver of attitude toward the use of AR, followed by interest. The analysis of knowledge gain has shown that the use of the “AR DC circuit puzzle” associated with the pre‐ and post-test shown the methodology in conceptual questions related to this topic as a acceptable one.

The work also had limitations, too …of course

Author Response

Please see the attachment.

Author Response File: Author Response.docx

Reviewer 2 Report

REPORT

Design and usability testing of an augmented reality (AR) environment in pharmacy education – A comparison between AR smart glasses and a mobile device in a laboratory course

BRIEF SUMMARY:

The aim: to develop an AR environment for teaching pharmacy and laboratory skills, using smart glasses or a mobile device to facilitate learning and to give instant feedback. 

The objective: to compare smart glasses against a mobile device for support of student performance in the laboratory in terms of user experience, technological anxiety, and self-efficacy.

“This is an important field of study given the need for virtual labs to replace hands-on labs in an environment of growing student numbers, increasing costs of labs, and the need to minimise student contact in a world of Covid”

GENERAL COMMENTS:

·         Overall, a very interesting paper with many appropriate references.

·         A more specific comparison of student results from this study with other research would greatly improve the manuscript and improve the Discussion

·         Initially, it was quite difficult to work out the method and quite difficult to understand how to replicate all parts of the study.  Possibly more information accompanying Figure 1 (see below) would help.

·         Including a cost comparison of smart glasses with mobile devices.  How affordable are numbers of smart glasses for use in many countries.  A statement is made that mobile devices are very cost effective but does not include information on smart glasses

·         The limitation of the small number of students using the smart glasses is noted

SPECIFIC COMMENTS:

Clarity of information in figure: Line 164:  Figure 1

The top two sections/lines of the figure are so small that they are unreadable, therefore do not provide any useful information. Text in the boxes needs to be enlarged so that it is readable

More detailed values of the results: Line 287: Table 4

The Likert scale results are presented as a mean of five levels of response.  Instead of average values to each response it would be very interesting to see the number of extreme responses – the extent to which students ‘strongly agree/disagree’

Learning versus preferences:

These results are important and merit more discussion for clarity. (Link the differences in the conclusion at line 369, or elsewhere.)

Line 18: ‘mobile devices promoted the learning process more than the smart glasses’

Line 19 and line 298: ‘self-efficacy results for mobile device use were higher’

Line 369: ‘preference between smart glasses and mobile devices were measured with both being accepted with great positivity but neither being clearly preferred over the other’

 

Comments for author File: Comments.docx

Author Response

Please see the attachment.

Author Response File: Author Response.docx

Reviewer 3 Report

The study discussed the development of an AR environment suitable tool for teaching pharmacy laboratory course. Addressing this technology in pharmacy education especially fundamental and basic courses is considered novel. I appreciate the effort authors made in this study. Also, the limitation section is well defined.

Some points need to be improved:

Introduction: The introduction provides a good, generalized background of the topic.

Lines 77-80: I don’t think it adequate to discuss the cost of the smart glasses in this study.

Lines 120-121: the aim of the study should be more specific regarding the pharmacy course involved in the study. 

Materials and Methods:

Study design should be clearly explained. Course name and description should be precisely explained without further detailed discussion about the content. This could be moved to discussion section if needed.

Design of the AR environment: who was responsible about the material and data that were fed into the AR application? Was it the same course content, were any modification made?

It seems that the comparison was not the aim of the study from the beginning. Number of participants in each group is different and sample size is relatively small.

participants section needs to be rewritten starting with the total number in the whole study.

Pre-&Post forms: Clear rationale for the design of survey questions will be useful. Was it validated and how?

Result

Table 3: User experience statement. Please give a rationale of each point in this table. What does it measure? The first two statements contradict each other.

Conclusion:

Lines 470-472.”For the first time, an AR environment was designed 470 and introduced for teaching laboratory skills in pharmacy education at the university 471 level” please make sure this statement is correct?

Reference style need to be consistent. For example, page numbers are cited in two different forms (pp. 89-100 and 19–27. Without pp)