Vibrational spectroscopic techniques have become standard in a wide variety of scientific fields, some of the most common being near-infrared (NIR), mid-infrared (MIR) and Raman spectrometry. These techniques have been used to great effect, however traditionally researchers relied on average spectroscopic data from a single sample point or, often in the case of materials, from the average spectrum of a material after pulverisation. Chemical imaging takes spectroscopy beyond this limitation. Where standard spectroscopy acquires chemical information from a single region, chemical imaging collects chemical information over many spatial regions. These regions can be used to form a high resolution matrix image over an area, where each element is an image pixel comprising a spectrum. Chemical variation across the surface of a material can therefore be assessed. Such information is used to provide an insight into the properties of foods [1
], survey the geological and vegetal make up of landscapes [3
] or improve synthetic processes through a better understanding of their resulting products [4
While chemical imaging provides a marked advantage over standard spectroscopy, limitations associated with the various spectroscopy modalities also apply to corresponding chemical imaging techniques. For example, certain molecular features, such as asymmetric molecular vibrations are invisible or near invisible to Raman spectroscopy. Equivalently, symmetric molecular vibrations are invisible or near invisible to MIR spectroscopy. Additionally, the spatial resolution achievable by all spectral modalities is restricted by the diffraction limit which differs by modality.
One method adopted for the purposes of overcoming these limitations is the fusion of different techniques. Fusion can refer to the combination of two or more chemical image cubes to arrive at a greater insight than the sum of their parts. Alternatively, fusion can refer to the combination of a high spatial resolution image comprising little or no spectral information with an image with relatively low spatial resolution but higher spectral resolution [3
]. Where a common area is imaged by different imaging techniques, one fusion approach has been the overlaying of two or more images at a specific absorption band such that the physical detail of the high spatial resolution image is visible with a colour scheme denoting different chemical regions elucidated from the imaging technique with greater spectral information [5
]. The above examples are often called multi-model fusion techniques, where images from different modalities are fused, however mono-modal fusion is also possible. For example, mono-modal fusion, where images are taken using the same modality, but at different spatial resolutions, or at different time points, can also be performed.
Clarke et al. [5
] were amongst the first to consider the combination of chemical imaging techniques, opting to experiment with FT-NIR imaging and complementary Raman point mapping. The study was performed in the context of visualising pharmaceutical formulations. The approach involved the imaging of a specific sample area using both techniques. The sample area was delineated with reference markers and with careful consideration of these it was possible to obtain Raman and NIR images of the same sample area with the same number of spatial pixels in each cube. Chemical image fusion, a term coined by the authors, was performed by overlaying several NIR and Raman images at wavenumbers specific to different components within the pharmaceutical blend. This study was very effective as the techniques proved to be complementary and each technique aided in removing ambiguities which would have been present if only one technique where used.
Multi-way chemometric approaches have also been explored for coupling chemical imaging data from different microspectroscopic modalities. For example, co-inertia (also known as Tucker) analysis is a multiway method that can be used to relate two data matrices with rows representing the same image pixels. In one paper [7
], Allouche et al. used multiple co-inertia analysis to combine three hyperspectral images of maize sections with different spatial resolutions, obtained using IR, fluorescence and Raman microspectroscopy. Prior to co-inertia analysis, the three hyperspectral images were co-registered to brightfield images and differences in spatial resolution were removed by pixel averaging. The initial resolution of the Raman and fluorescence images was recovered by projecting the original images along the block or global loadings obtained. In a related paper [8
], Allouche et al coupled fluorescence and IR images of a maize cross section using an extended co-inertia approach. Each 1 × 1 pixel in the IR data was related to a 7 × 7 pixel in the higher resolution fluorescence data by registration to brightfield images. The co-inertia approach was extended to simultaneously analyse a two and a three way data table, where the two way table comprised the unfolded IR image and the three way table comprised the unfolded fluorescence image (the third way representing the 49 pixels of fluorescence data for each IR pixel). The authors demonstrated that the extended analysis enabled preservation of the higher resolution of the fluorescence data while not affecting the spectral interpretation of co-inertia loadings.
In a more recent study, Ewing et al. [4
] obtained ATR-IR and Raman chemical image cubes of pharmaceutical tablet samples of different sizes. The tablets comprised an API trapped within a polymer matrix which swelled when exposed to an appropriate liquid, exposing the API particles to the liquid. The team argued that while the solubility was known for the API, if disproportionation (i.e., the API ion reverting to the free acid form) occurred, the dissolution rate would change markedly as the free acid is less soluble in a polar solvent. The experiments were conducted in a flow cell and various pH solutions were used. The two modalities were used to image both tablets and investigate for spatial variation, particularly searching for the presence of the API ion and free acid forms. Though the imaging modalities were performed on different samples and no multivariate data fusion was conducted, the results complemented one another and confirmed that different pH levels, different amounts of disproportionation occur.
Data fusion is also useful in the field of remote sensing. For example, Licciardi et al. [3
] described the combination of hyperspectral (such as a Compact High Resolution Imaging Spectrometer (CHRIS) sensor acquisition of 63 spectral bands in range of 400–1050 nm using) and panchromatic images (such as a Quickbird-PAN acquisition of a black and white image generated over the wavelength range of 405–1050 nm). The approach incorporates dimensionality reduction and indusion (induction and fusion). The team’s main aim was the fusion of both modalities to arrive at an image with the high spatial resolution of panchromatic imaging and the chemical information of chemical imaging. The proposed fusion approach involved performing non-linear principle component analysis (NLPCA) on the chemical image cube for dimensionality reduction purposes, followed by down-scaling of the panchromatic image using a filter to fit the size of the NLPCs. Histogram-matching was then performed between the NLPCs and the down-sized panchromatic image. The NLPCs were up-scaled as was the histogram-matched panchromatic image. Histogram-matching was then performed between the up-scaled NLPCs and the original panchromatic image. The difference between the histogram-matched up-scaled panchromatic image and the histogram-matched original panchromatic image was found and this difference was added to the up-scaled NLPCs. The original spectral bands were reconstructed through decoding. Visual quality assessments of the resultant image were then conducted. The spectra were also altered in the indusion process and the root mean square error (RMSE) was calculated between the produced spectra and the original spectra and while some issues were reported for features that were visible in the pan chromatic image but not in the chemical image cube, the results appear to be quite favourable.
Fusion has also been used in non-vibrational spectroscopic imaging modalities for material characterisation. For example, Artyushkova et al. [6
] describes a method of fusion of X-ray photoelectron spectroscopy (XPS) and atomic force microscopy (AFM) images of polymer blends. AFM has nanometer spatial resolution while XPS provides spectral information but low spatial information. The fusion technique requires the de-resolution of the AFM image such that the resulting image has the same spatial resolution as the XPS. The images were registered to each other using automatic image registration (AIR). This translates, rotates and scales the pixels appropriately. The mapping can then be checked and confirmed. When the necessary translation, rotation and scaling is known, the original AFM image can then be used and the chemical detail of the XPS image overlaid and presented as colour changes. This results in an image with the high spatial resolution of AFM and the chemical information of XPS. Van de Plas et al. [9
] describes the use of data fusion through the combination of imaging mass spectrometry (IMS) and high resolution light microscopy. The imaging techniques were combined to take advantage of the high spatial resolution of light microscopy and the chemical information of lower resolution IMS. The team aimed to combine ion intensity readings with photon based variables by modelling the distribution of each throughout the area of the images. This required the use of partial least squares regression (PLSR), which predicted ion concentrations at different regions of brain tissue. It is reported that the microscopy image (the spatially distributed photon variables) could predict ion distribution with a reasonable degree of accuracy for several ion mass/charge ratios. In addition, the team reported that looking at modalities in tandem was useful in confirming signals that would otherwise have been considered instrument noise.
Challenges associated with data fusion methods should also be noted. A first such challenge may be any destructive effect that one imaging technique has on a sample, particularly as this could affect subsequent readings on other modalities. For example, mass spectrometry chemical imaging necessarily removes small amounts of sample in order to conduct analysis, which will affect a sample to some extent, however small. Further, Raman spectroscopy, while not generally considered a destructive technique, does require the focusing of a laser on a sample and can therefore lead to small amounts of sample evaporation/sublimation or provide the activation energy required for a new reaction to take place, including but not limited to sample combustion in oxygen. One solution may be to order imaging modalities with increasing destructiveness so as to reduce sampling artefacts. In addition, when an experiment involves imaging over a time series for a dynamic sample, the experimenter must decide on a compromise, choosing to image of same the sample at different times, or image different but similar samples at the same time. Both approaches unfortunately reduce experimental certainty. Further challenges include: potential difficulties with imaging the same sample area across microscopic modalities, lack of prior knowledge of sample composition and or availability of pure component spectra.
Our literature survey indicates that the multivariate nature of chemical image data is often neglected in data fusion (e.g., [4
]). Multivariate chemical fusion techniques differ from standard fusion in that the imaging techniques used are not considered in mathematic isolation. Multivariate chemical fusion techniques therefore have at least two very significant advantages over standard fusion techniques. The first is the reduction of experimenter bias in the designation of signal correlations between features of different imaging techniques. This is because a sole reliance on visual interpretation is no longer required. Further, correlations between imaging techniques which are not intuitively obvious to an experimenter visually may be found with multivariate chemical fusion. Therefore, multivariate chemical fusion approaches increase certainty by serving to reduce both false positives and false negatives in relation to inter-modality correlations.
In this paper we provide a framework for multivariate chemical image (CI) fusion to enable: cross modality image registration; improved classification performance; investigation of cross modality correlations; prediction of one modality from another and resolution enhancement.