Beyond Human Vision: Unlocking the Potential of Augmented Reality for Spectral Imaging

Abstract

Spectral imaging is a broad term that refers to the use of a spectroscopy technique to analyze sample surfaces, collecting and representing spatially referenced signals. Depending on the technique utilized, it allows the user to reveal features and properties of objects that are invisible to the human eye, such as chemical or molecular composition. However, the interpretability and interaction with the results are often limited to screen visualization of two-dimensional representations. To surpass such limitations, augmented reality emerges as a promising technology, assisted by recent developments in the integration of spectral imaging datasets onto three-dimensional models. Building on this context, this work explores the integration of spectral imaging with augmented reality, aiming to create an immersive toolset to increase the interpretability and interactivity of the results of spectral imaging analysis. The procedure follows a two-step approach, starting from the integration of spectral maps onto a three-dimensional models, and proceeding with the development of an interactive interface to allow immersive visualization and interaction with the results. The approach and tool developed present the opportunity for a user-centric extension of reality, enabling more intuitive and comprehensive analyses with the potential to drive advancements in various research domains.

1. Introduction

In their own yet intertwined ways, spectral imaging and augmented reality may be considered as enhancers of human perception with a wide range of applications across scientific and industrial fields, particularly in domains such as mineral exploration, cultural heritage diagnostics, and artifact conservation. On the one hand, spectral imaging captures information from a scene beyond classical RGB imaging, mapping it onto a two-dimensional (2D) image. By leveraging the potential of the spectroscopy technique, it may provide information regarding the spatial distribution of chemical elements or sample composition, revealing hidden features that are invisible to the human eye. On the other hand, Augmented Reality (AR) is a technology that enhances the perception of reality by projecting digital information on the physical world via mixed-reality displays or projection solutions. Recent technological advancements have unlocked the potential for new immersive and interactive systems that hold promise as tools for harnessing the full range of human sensory perception to interact with digital data [1]. Thus, while spectral imaging and AR have different purposes, they are connected in the way they provide a user-centric extension of reality and enhance human capabilities [2,3]. Within this context, it also makes sense to explore synergistic pathways for both technologies, in particular, exploring how AR can enhance the visualization and interpretation of spectral imaging datasets.

Conceptually, the combination of spectral imaging with AR presents a unique opportunity to assist and enhance the capabilities of the individual tools in a bidirectional manner, i.e., (i) AR assisting existing spectral imaging and (ii) spectral imaging enhancing AR common applications.

From the perspective of spectral imaging, spectroscopy techniques may offer in situ, sensitive, and rapid spatial maps of chemical analyses, with minimal sample preparation and utilizing micro and non-destructive analytical methods [4,5]. Although spectral imaging results are typically presented as 2D images, a recent approach [6] suggested a methodology for integrating 2D surface maps into 3D models of the sample, which may be constructed using various scanning solutions, such as photogrammetry, LIDAR, or laser scanning [7,8,9,10]. The results demonstrate the possibility of integrating both models in a seamless manner, allowing an external user with no access to the physical sample to locate the data better, look from different perspectives, and explore scientific visualization tools to interact with the spectral dataset in non-trivial manners. For instance, viewing spectral data in 3D allows to see how chemical elements are distributed across the surface texture of a sample, making it easier to correlate specific elements or signal variations with physical characteristics, such as surface roughness or material texture. This type of immersive visualization supports more intuitive pattern recognition and material interpretation compared to static 2D interfaces, where spectral maps are typically analyzed on flat screens without spatial context or physical scale. While such tools are well-established, they often fragment the user attention and require cognitive effort to mentally reconstruct the 3D geometry or correlate spectral data with real-world locations.

In this sense, bridging these 3D digital models to real environments via AR offers a pathway for a new data analysis environment, providing contextual interpretation with unique interaction and rescaling capabilities as explored in other areas [3]. In addition, while most of the work on the topic of spectral imaging focuses on the acquisition of images at the surface of a sample, specific techniques or applications allow access to volumetric information up to a certain degree [11,12,13,14]. In this case, AR can help visualize cross-sectional maps with subsurface 3D reconstructions in real time, holding interesting potential for applications in subjects such as artwork restoration, where understanding complex stratigraphy or volumetric characteristics may be critical.

From the perspective of AR, the key idea is to superimpose digital information into the real world, contrasting with virtual reality which immerses the user in a fully synthetic 3D space, often disconnecting them from their physical surroundings. Thus, AR maintains a connection with the physical space, allowing for a more contextual and interactive experience with digital models and data [15], supporting contextual awareness, in situ annotation, and collaborative analysis within shared physical environments. These characteristics are particularly relevant in domains such as cultural heritage diagnostics and the mining industry, where users may need to compare physical samples, access physical documentation, or interact collaboratively in shared spaces. In this context, spectral imaging holds the potential to add novel multiple layers of information, superimposing them both onto these digital models or even in real-world objects, allowing humans to see beyond human vision and enable deeper analysis and better insights [16,17].

Indeed, some pioneering results in the literature already hint towards these directions, either by utilizing projection methods [18] or interactive headsets, with the latter significantly enhancing the experience via additional embedded sensors. For example, ref. [17] explores the combination of a fiber optic Raman probe and an augmented reality localization strategy and interface to create a visualization of a chemical image of samples, overlaid yet not anchored to the physical object. Regarding hyperspectral imaging, interesting approaches include medical applications such as the classification of tissues during in vivo surgical procedures [19] and a toolset for guided brain tumor phantom resection which utilizes an HoloLens AR headset [16], to mention a few illustrative examples. In the geological sciences domain, ref. [20] presents a toolset to analyze rock surfaces and mineral types, which may be promising to assist in situ operations, but again missing the contextual interpretation of 3D models. In general, although these approaches demonstrate the potential of combining AR and spectral imaging, the analytical potential of the solution is a characteristic often overlooked. To the best of our knowledge, no toolset fuses interactive 3D digital models with spectral imaging, adding the capabilities to arbitrarily scale the samples to enhance details, access other perspectives, or interactively explore spectral information.

In this manuscript, we describe a full-stack methodology and pipeline to integrate spectral imaging datasets onto 3D object digital models, deploying it as a user-friendly interface that provides an interactive environment to explore the object and dataset characteristics. For that purpose, a Unity software solution for spectral analysis will be developed for the Microsoft Hololens 2 device, taking advantage of the Mixed Reality Feature Toolkit needed to add interaction such as zoom adjustment and rotation, in addition to more in-depth spectrum analysis tools. The results presented demonstrate the potential of the fusion of AR and spectral imaging to open new avenues for research and innovation, enabling us to perceive and interact with our world in ways previously unattainable and enhancing human vision capabilities.

2. Method

The main objective of this manuscript is to introduce a versatile and user-friendly solution that allows the analysis of a spectral imaging dataset in an Augmented Reality (AR) environment. More specifically, we effectively capitalize on the interpretability and interactive capabilities of the latter, utilizing 3D models of the analyzed samples.

Following the same rationale, for the purpose of this work, we will focus on headset wearable AR devices, specifically, the Hololens 2 Microsoft model, a robust low-latency and markerless tracking device. Indeed, although AR has advanced significantly in recent years, offering various solutions ranging from headsets to heads-up displays and projection methods [21,22,23], wearable devices are still the most suitable in providing user-centric interactive environments.

In generic terms, the solution presented here consists of a three-step process that will be further detailed in the sections below. The first step concerns data acquisition, i.e., obtaining the spectral imaging dataset and the 3D model of the object. The second step focuses on asset management and consists of a processing pipeline to pre-process spectral imaging datasets and construct spectral maps, as well as their integration with the 3D models via mesh alignment. Finally, the third step focuses on the development and deployment of AR software solution as a user-friendly interface for the effective exploration of the acquired spectral data.

2.1. Step 1: Data Acquisition

As depicted in Figure 1, the first step is to acquire the necessary datasets to construct the 3D digital model of the object and integrate it with spectroscopy information.

For the acquisition of the 3D model of the sample, we utilized a photogrammetry technique [7,24] as previously explored in a recent work [6]. Note that although we opted for this method due to the lower cost of the solution, there are other techniques such as laser-based scanning [25,26] that can be utilized if better resolution models are needed, in particular, if the object is small (e.g., below centimeter scale). For the photogrammetry technique, the 3D reconstruction of the sample was generated by capturing a video using a smartphone, which was then loaded into the Polycam platform to generate a realistic three-dimensional model of the sample and the respective texture.

Regarding the spectral imaging dataset, any spectral imaging technique could be explored, as long it is utilized to scan a sample surface, resulting in a dataset of $N_x\times N_y\times N_{\lambda }$, where $N_x$ and $N_y$ relate to the transverse spatial dimensions, whereas $N_{\lambda }$ is associated with the spectral part (i.e., wavelength or wavenumber). For the present case, we utilized a Laser-Induced Breakdown Spectroscopy technique [6] to scan a sample surface point by point, obtaining a spectral signal for each point that contains emission lines to be related with the elements present in the sample [27].

2.2. Step 2: Asset Management

Building on the datasets acquired in the first step, step 2 comprises the necessary tasks to analyze the spectral dataset, construct the relevant images, and merge them with the 3D digital model of the sample. This step is performed in the form of Python scripts and streamlined user-centric actions.

First, having the spectral dataset acquired, it is necessary to analyze it and construct the relevant maps to be merged with the 3D digital model. While this process could be performed in the headset itself and incorporated into the software solution, we opted to perform it previously for efficiency purposes. Indeed, the analysis of spectral data typically requires a significant amount of pre-processing such as baseline removal [28] or filtering [29], which, in addition to the fact that spectral imaging datasets are typically large in memory size (in the order of GB of data), require a lot of processing power for efficient operation.

For this work, and having utilized LIBS, we applied a baseline removal procedure using Asymmetrical Least Squares Smoothing (ALSS) [28], and the main spectral lines corresponding to each characteristic spectral signature were extracted. Next, we select a set of $N_i$ relevant spectral lines and construct the spectral maps by integrating them in the $\lambda$ dimension around the selected line and applying a spatial Gaussian filter. All of these steps are performed in a single Python script, outputting a series of $N_i$ images for the relevant features selected (see Figure 2). An additional image with a mean spectrum is also saved to a file to be displayed in the final solution as further described in step 3.

Subsequently, and in order to merge with the 3D digital model, we need to superimpose the spectroscopic data in the corresponding surface of the sample. This is performed using a mesh matching algorithm previously presented in the literature [6], which consists of determining the following three transformations: translation, rotation, and scaling between two point clouds, one in the 3D model and the other one in the 2D spectral image, minimizing the mean square error between the clouds.

Mathematically speaking, these selected points, $\overrightarrow{p_i}$ and $\overrightarrow{q_i}$, corresponding to the same spatial region in both the 3D model and the 2D image, can be represented by two matrices (Equation (1)) that contain the set of points intended to be aligned as follows:

$$\mathbf{P}=\left(\begin{array}{ccc}{x}_1& y_1& z_1\\ x_2& y_2& z_2\\ ⋮& ⋮& ⋮\\ x_n& y_n& z_n\end{array}\right)\mathrm{and}\mathbf{Q}=\left(\begin{array}{ccc}{i}_1& j_1& k_1\\ i_2& j_2& k_2\\ ⋮& ⋮& ⋮\\ i_n& j_n& k_n\end{array}\right)$$

(1)

According to ref. [30], this minimization problem can be solved by computing the cross-covariance matrix, ${\mathsf{\Sigma }}_{PQ}$, for the point patterns $\mathbf{P}$ and $\mathbf{Q}$ as

$${\mathsf{\Sigma }}_{PQ}={\displaystyle \frac{1}{n}}\sum _{i=1}^n{(\overrightarrow{q_i}-\overrightarrow{{\mu }_q})}^T(\overrightarrow{p_i}-\overrightarrow{{\mu }_p})$$

(2)

By decomposing the matrix ${\mathsf{\Sigma }}_{PQ}$ through Singular Value Decomposition (SVD), i.e., ${\mathsf{\Sigma }}_{PQ}={\mathbf{UDV}}^T$, it is possible to achieve the optimal transformation parameters that minimize the mean squared error given by the following:

$$\begin{array}{cc}\hfill & Rotation:\mathbf{R}={\mathbf{UCV}}^T\hfill \\ \hfill & Scaling:s={\displaystyle \frac{1}{{\sigma }_P^2}}tr\left(\mathbf{DC}\right)\hfill \\ \hfill & Translation:\overrightarrow{t}=\overrightarrow{{\mu }_Q}-s\mathbf{R}\overrightarrow{{\mu }_P}\hfill \end{array}$$

(3)

In this description, $\overrightarrow{{\mu }_P}$ and $\overrightarrow{{\mu }_Q}$ correspond to the position of the centroids, ${\sigma }_P^2$ and ${\sigma }_Q^2$ correspond to the variances of the matrices, and C is a parameter that ensures that the rotation matrix $\mathbf{R}$ retains the correct orientation, even when the alignment involves a reflection, and this is given by

$$\mathbf{C}=\left\{\begin{array}{cc}\mathbf{I}\hfill & ,\mathrm{if}\det \left(\mathbf{U}\right)\det \left(\mathbf{V}\right)=1,\hfill \\ \mathrm{diag}(1,1,\dots ,1,-1)\hfill & ,\mathrm{if}\det \left(\mathbf{U}\right)\det \left(\mathbf{V}\right)=-1.\hfill \end{array}\right.$$

(4)

This algorithm was implemented in Python (v3.12.7) and executed autonomously using ParaView software (v5.12) [31], requiring only the selection of specified points by the user. Although this method provides flexibility and control, it may limit reproducibility and increase the time required for setup in more complex samples.

Finally, some final additional tweaks are applied to the scene for visualization purposes (e.g., adjust the colorbar, brightness, and textures), before importing it to the software solution. Some results are depicted in Figure 3, showing a reliable alignment between the spectral data and the 3D model, highlighting the effectiveness of the proposed method.

2.3. Step 3: Development and Deployment of Unity Software Solution

In order to interact with the digital models in a meaningful manner in the context of spectral imaging, the simple upload of the generated models to a third-party 3D model visualizer on the Hololens 2 would be extremely limited. Indeed, a complete toolset requires not only the visualization of the model but also the interaction with the spectral data. This means, for example, integrating a window to analyze the spectral signal, as well as additional solutions to shift between different spectral maps for the same sample and even putting multiple samples side by side for integrated analysis.

For that purpose, we opted to develop a tailored software solution, building a Unity project within a 3D Universal Windows Platform adapted to the Microsoft Hololens 2 headset. Unity is a game engine that can be used to create 3D and 2D interactive content. Unity excels in real-time rendering, which together with its cross-platform capabilities make it a natural choice for a versatile approach to addressing the major challenge of the manuscript [32,33]. In addition, to better explore AR capabilities, the Mixed Reality Toolkit (MRKT) package was utilized in the project. The MRKT adds a unique collection of components and advanced functionalities such as spatial mapping, hand tracking, and eye tracking, which are essential for creating immersive and interactive AR experiences.

In Figure 1 (right), we present the designed interface, composed of a text box, a dropdown menu, a 2D graph, and the sample itself. Each one of these components has a particular function in the interface, with the information updated according to that contained in the loaded model. The text box displays the sample name and the dropdown menu, the relevant features for which we previously generated the spectral maps (e.g., element lines for LIBS). By interacting with this icon, the user can switch between spectral maps. The 2D graph displays the image previously generated in the second step, which contains the average spectroscopy signal for the sample. Finally, in the center of the interface, the model of the sample appears, displaying the selected spectral map seamlessly overlaid on the sample surface while preserving the relevant object texture information. We added collision properties and object manipulator functions to the objects alongside the GameObjects interactions tool to provide the elements with all the necessary gesture-based interactions.

Finally, a menu window can be utilized to access additional functions. Among these functions, the first button allows us to hide all elements of the interface, except for the sample and menu to achieve an environment with fewer distractions. The second allows us to access a list of samples, enabling also the simultaneous display of samples to facilitate comparative examination. Finally, a third function activates a QR code reading mode to allow users to move the sample to their desired location, which can be utilized to load models directly from a remote server in the future. All of these buttons have the particularity of being voice-activated, allowing the user to have hands-free operation for other tasks.

3. Results

As previously said, for the purpose of this work, we focused on the LIBS technique for obtaining the spectral imaging datasets. In short, LIBS imaging allows for the creation of two-dimensional chemical maps of samples by identifying elemental and molecular emission lines (Figure 2) through the analysis of spectral signatures generated during the decay of laser-induced plasma on the sample surface. This method was applied to two distinct samples for illustrative purposes as follows (Figure 3): (i) an oxidized double open-end wrench composed of chrome–vanadium steel, and (ii) a traditional Portuguese tile fragment.

For the case of the wrench sample, the presence of a spatial distribution of iron (Fe) (I) and (II) emission lines was detected during the analytical phase in step 2, as well as vanadium (V), chromium (Cr), aluminum (Al), and oxygen (O). All of these are consistent with the surface oxidation observed on the sample. On one hand, the presence of iron and vanadium is consistent with the chrome–vanadium steel composition. On the other hand, Cr and oxygen lines were detected with a higher intensity as they are part of the protective coating, revealing the less damaged part.

For the tile fragment, the distribution of cobalt (Co) was particularly noteworthy as a part of the paint used in the tile. The expected spatial match underscores the capability of LIBS to identify and analyze heritage artifacts, offering significant potential for the study and preservation of cultural heritage artifacts.

Figure 4 presents the user perspective of the developed interactive tool, illustrating the user-friendly and informative interface that integrates spectral imaging with augmented reality (AR). This interface allows users to observe all relevant characteristics of the spectral map projected on a digital model of the sample, offering a degree of contextualized interpretation and gesture-based interaction of the acquired data not achievable with traditional visualization methods. In particular, from user experience (which can be seen in the video in the Supplementary Materials), we highlight the following interaction capabilities:

• Moving and rotating the sample: It allows us to position and move the sample freely, analyzing the composition in detail from multiple perspectives;
• Scaling: Allowing us to scale up small samples or reducing larger ones, the solution offers unique perspectives of the samples in integration with additional spectral features;
• Texture-based interpretation: Including the texture of the sample surface and possible roughness, it is possible to relate common signal variations (e.g., plasma signal in LIBS) due to the surface properties and better contextualize them.
• Multiple sample comparison: Adding multiple samples at the same time to the environment eases visual comparisons, enabling the deduction of clear connections or distinctions.

4. Discussion

Leveraging these capabilities, the potential applications of this approach are vast and will be discussed in relation to the following two major directions, in close connection with the samples presented: industrial applications and academic ones, with a focus on heritage science.

From an industrial perspective, the applications of this technology essentially focus on quality control and monitoring. As seen for the tool in Figure 2, LIBS imaging can identify and monitor material degradation, providing critical information for maintenance and conservation efforts. Integrated with AR, areas of corrosion or material fatigue may become more evident, allowing us to establish a deeper connection with the object volume. In addition, and in the context of quality control, AR can guide maintenance personnel through complex inspection processes, overlaying spectral data onto physical components to highlight areas requiring attention, thus improving worker efficiency. This holds potential for upstream interventions, including the detection of production defects and ensuring product quality by detecting elemental inconsistencies or contaminations.

From an academic perspective, the applications are also compelling. Focusing on heritage science for illustrative purposes, the study of objects and artifacts strongly benefits from using spectral imaging techniques like LIBS as they offer sensitive and rapid spatial maps of chemical compositions with minimal sample preparation. With such a technique, the researcher can analyze paintings, metals, ceramics, and stones, providing detailed information about the composition of the object, which may aid in their characterization and inform the planning of preservation interventions. AR is also rather popular in this research domain, as it provides a means for users to visualize and interact with accurate digitized models of objects, allowing free gesture-based interaction without harming asset preservation [34]. In addition, it can transform cultural and educational experiences into engaging experiences, facilitating collaborative and inclusive environments among scholars using shared digital platforms regardless of their physical location [34,35,36].

In this context, the potential to add novel layers of information to digital models of heritage artifacts coming from spectral imaging technologies may enable deeper analysis and yield insights without physical interaction, contributing to a better yet harmless understanding of the materials and techniques used for example, as well as the formation of connections to the historical period [34]. The 3D interactivity may also be utilized to inform the planning of preservation strategies and to establish collaborative playgrounds with multiple layers of information and multiple samples (e.g., fragments [37,38]).

All in all, besides the natural exploration of the tool in multiple domains of science and technology such as heritage research [34], biology and medicine [19,39,40], geology [41], and industrial process control [42], we anticipate that promising future research lines may also include the exploration of a few other ramifications, including the following: the exploration of other spectral imaging techniques with multimodality and sensor fusion; the integration of machine learning tools and solutions for analysis and sample classification; and the exploration of the tool in classroom settings for teaching and demonstration purposes.