Augmented and Virtual Reality in Dermatology—Where Do We Stand and What Comes Next?

Abstract

As the skin is an accessible organ and many dermatological diagnostics still rely on the visual examination and palpation of the lesions, dermatology could be dramatically impacted by augmented and virtual reality technologies. If the emergence of such tools raised enormous interest in the dermatological community, we must admit that augmented and virtual reality have not experienced the same breakthrough in dermatology as they have in surgery. In this article, we investigate the status of such technologies in dermatology and review their current use in education, diagnostics, and dermatologic surgery; additionally, we try to predict how it might evolve in the near future.

1. Introduction

Augmented reality (AR) is a component of virtual reality (VR) or virtual environments (VE) where the user is completely immersed into a synthetic environment [1]. While in VR, the user does not see the real world. Instead, AR allows us to superimpose virtual objects onto real ones. The number of AR/VR applications has undeniably exploded over the last decades. While both techniques have been extensively used in various medical fields, such as in plastic surgery [2], little investigation has been performed so far in dermatology [3,4,5]. This is surprising as (i) the skin is an accessible organ of the body particularly suited for AR applications, (ii) dermatology is a medical field where visualization is essential, and (iii) the sector is worth billions of dollars in the US alone, leading to numerous business opportunities. The main applications of AR/VR in dermatology can be categorized into three main groups: education [6,7,8], dermatologic surgery [9,10] and diagnostics [11]. In a paper published in early 2000, Gladstone and coworkers identified the twenty-first century dermatological VR applications [10]. Detailed in their visionary manuscript, they predicted that dermatologists will be soon able to train the night prior to performing complex Mohs surgery using patient specific skin and skeletal data. Students will not only be able to visualize a lesion in 3D but will also be capable of physically touching it using haptic feedback systems. Thanks to such tools, telemedicine could be revolutionized. Finally, VR technology will enable “moving” inside a lesion and observing it at different angles to improve diagnostics. In this following article, we review the status of AR and VR and give insight into potential future developments.

2. Education

Improving dermatologic teaching is a challenge, as visual memory is difficult to maintain without constant exposure. As a result, it has been reported that the diagnostic accuracy of common skin diseases is poor by primary care physicians [12]. The use of 3D silicon-made phantoms greatly improves diagnostics skills and knowledge retention in comparison to standard 2D images [13]. Therefore, it is expected that AR should exhibit similar effects [8]. Nonetheless, recent studies have demonstrated ambiguous results with minor differences in knowledge gain and slightly better knowledge retention for the group using AR-based tools in comparison to textbooks [7]. This outcome could potentially be improved by coupling AR with haptic feedback, as is the case in most surgical training tools.

A potential application that could prove to be beneficial in training the next generation of medical experts would include realistic-looking 3D models displaying skin diseases and irregularities (see Figure 1). These models would contain high resolution clinical textures that are anonymized on newly generated 3D models with anatomical proportions, all of which could be run on smartphones/tablets or VR goggles. The environmental conditions (like lighting conditions) could be adapted for a more realistic experience. In addition, such models could be expanded by using synthetic clinical images, to reduce data scarcity and poor training regarding rare dermatological diseases and certain Fitzpatrick skin types. For example, incidence of melanoma is relatively rare among black populations, where survival rates lag behind the rates for non-Hispanic white populations in the US [14]. Such issues have recently been highlighted in the press [15]. The goal of the DermGAN project is to improve diversity in clinical skin images [16]. In this project, a model generates skin images that exhibit the characteristics of a given pre-specified skin condition, location, and underlying skin color that could be recreated into a 3D visualization.

Aside from health professionals, some studies have investigated AR/VR as an educational tool for patients or the public to increase their awareness of skin disease prevention. The group Elke Hacker in Australia has developed a virtual reality (VR) game containing preventative skin cancer messages for youth and young adults [17]. This was based on the fact that despite successful public campaigns in the 1980’s, high rates of sunburn continue to be reported among the young population.

In another application of VR in dermatology, we created the VR app Virtual Derm, which places students in a virtual practice for dermatology [18]. The app runs on standard Android smartphones, making it accessible to most students. A total of 100 patient vignettes can be solved by students, making it a unique learning opportunity (see Figure 2).

3. Dermatologic Surgery

Dermatologic surgery is the practice of dermatology that specializes in surgical procedures to repair or improve the function and appearance of the skin [19]. While plastic surgery greatly benefited from AR/VR technology [20], few applications have been reported in dermatologic surgery. Aside from surgical procedure training [21] (not discussed in the paragraph above dedicated to education), AR and VR have been implemented in Mohs surgery to reduce patient anxiety [22] or as a surgical assistance tool [23].

For patients, VR has been used to reduce pre- and intraoperative anxiety. The tools employed in this study were unspecific VR sequences which could be used to reduce anxiety under any circumstance. It has been shown that almost all anxiety-related parameters could be improved in the context of surgery [22]. More specifically, VR could be used as tool for patient education, as many patients do not have a clear understanding what a surgical procedure entails, such as using a flap for the reconstruction of a surgical defect. However, this has not been investigated and may also induce anxiety in some patients where too much detail about a surgical procedure is given.

For physicians, VR has been used as a training tool in many surgical specialties and has great potential in dermatologic surgery. The more realistic appearance of a VR image, combined with haptic sensations, can be used to train dermatologic surgical interventions in the same way as in other surgical fields, notably plastic surgery. AR has been successfully used by one group to improve the communication between surgeons and pathologists in cases of Mohs surgery, where the pathology slides were not necessarily read by the surgeon themselves [23]. In addition, AR has great potential for other aspects of Mohs surgery and dermatologic surgery. After acquiring the theoretical background and training with artificial models, a surgical procedure is best learnt by performing the procedure on a real patient. In most training programs, this is performed under the direct supervision of a senior faculty member who is present at the operating table. However, due to shortcomings in staff resources and planning, this is not always possible and junior surgeons sometimes find themselves performing an operation that is above their level of confidence alone. AR could allow a senior faculty member to follow an operation remotely and give very precise and practical instructions to the junior surgeon at the table, with AR images superimposed on the surgical site to guide the surgery. In larger institutions, this would also allow senior staff resources to be used more efficiently, as one experienced surgeon could follow several surgeries at the same time.

4. Diagnostics

Artificial intelligence (AI) has been productively utilized for several decades in dermatology, particularly for skin cancer detection. It has been reported that convolutional neural networks (CNNs) achieve dermatologist-level classification of cutaneous lesions using both dermoscopic and non-dermoscopic images [24]. AI has proven very useful when combined with VR reconstruction of the body surface. In a prominent example, the Canfield^® Vectra WB360 device combines 2D photographs of 46 stereo-vision pods, resulting in a full-body VR reconstruction that can be turned and manipulated at will. The machine further analyzes all photographs with ML-based segmentation and classification, looking for naevi and other cutaneous lesions. All naevi can be analyzed by artificial intelligence for their probability of being malignant. The information on these lesions is saved, with the position of each lesion on the body map, and upon repeated patient use of the Vectra device, changes are tracked over time. By utilizing this technology, newly developing melanomas are actively being found. We are currently performing a study (Melanoma Detection in Switzerland With VECTRA–MELVEC, ClinicalTrials.gov Identifier: NCT04605822) which compares dermatologists’ diagnostic accuracy with and without support by artificial intelligence. Patients receive their VR reconstruction and their results to peruse within the comfort of their home. To date, no other VR approach has been made available in dermatology and has generated much enthusiastic feedback from those in the field.

AI-based smartphone applications, or “apps”, are being successfully developed as practical tools to improve diagnostics [25,26,27,28], despite some limitations in terms of accuracy for melanoma screening [27]. In a recent study [25], a mobile augmented reality approach was used to support real-time diagnostics for melanoma lesions using deep learning. Parameters such as lesion diameter, color, or asymmetry are displayed in real time in the camera view (see Figure 3). The preliminary evaluation of the app was very encouraging [25].

The use of AR/VR in dermatology could strongly benefit from haptic feedback. Haptic feedback, or “haptics”, refers to manual interaction with the environment, and could be used to relay force and tactile information to the user [29]. Indeed, sensory information such as skin temperature, roughness or stiffness would provide valuable additional information to support diagnostic evaluation [30]. In this direction, Kim and co-workers developed a method to convert a single image to a 3D haptic surface and render the generated haptic surface in real-time [31,32].

5. Methodology

For this review we performed a literature search using the Web of Science Core Collection and MEDLINE^® databases. The descriptors used were “virtual environments”, “virtual reality”, “augmented reality”, “dermatology”, and “dermatological science”. We excluded the descriptor “skin” on purpose as we were not interested in haptics systems, but wanted to focus on the applications of augmented and virtual reality in dermatology only. Using these descriptors, the search was performed among “title”, “abstract”, and “author keywords”. We considered all publication years and document types. We did not exclude any “research areas”. We did not exclude any MeSH Qualifiers or MeSH Headings. We restricted our search to documents in the English language only. With such parameters, 64 articles were found, with 35 documents published in the last 5 years and 51 published in the research area “Dermatology” (full search report available as Supplementary document). We manually went through the articles, selecting relevant ones via the title and abstract.

6. Limitations

In this paragraph we shortly review the limitations of augmented and virtual reality in dermatology, focusing on applications, and without investigating potential adverse health effects on the user resulting from the use of AR/VR.

The use of AR/VR in dermatologic teaching suffers from similar limitations as in general medical education. Those have been recently extensively studied, potentially due to the restriction associated with COVID-19 pandemic [33,34,35]. Parsons and MacCallum investigated the affordances and limitations of augmented reality in medical education [33]. Selecting five potential affordances (A1: reducing negative impacts (risk, cost), A2: visualizing the otherwise invisible, A3: developing practical skills in a spatial context, A4: device portability across locations, A5: situated learning in context), the authors explored how these affordances were highlighted in a systematic literature review. They pointed out the versatility of the use of AR/VR in medical education, but suggested that proper selection of the affordances when designing a new system for medical education may provide benefits [33].

In dermatologic surgery and diagnostics, augmented and virtual reality technologies exhibit several limitations that are shared with the applications of AR/VR in general surgery [20]. If technological limitations such as image quality, battery life, usability (e.g., size, cables) can hamper the method, they most likely will decrease with technological evolution. As with all patient electronic data, confidentiality should be guaranteed and all legal requirements should be met [20].

7. Conclusions

AR and VR have brought significant advances for dermatology in education—both for medical students and for patients and general public—in dermatological surgery, particularly for complex surgical operations such as Mohs surgery, and in diagnosis, particularly for naevi screening. Even though the technology has not revolutionized standard practice and is still rarely used in the field, the dramatic increase of the research performed during the last 5 years allows for optimism. There is more to come as the technology progresses and is democratized. However, we believe that VR will be replaced successively with AR approaches that allow users to directly interact with the clinical environment via dedicated haptics.