Interesting Features Finder: A New Approach to Multispectral Image Analysis

Abstract

In this paper, we discuss a new approach to the analysis of multi/hyper-spectral data sets, based on the Interesting Features Finder (IFF) method. The IFF is a simple algorithm recently proposed in the framework of Laser-Induced Breakdown Spectroscopy (LIBS) spectral analysis for detecting ‘interesting’ spectral features independently of the variance they represent in a set of spectra. To test the usefulness of this method to multispectral analysis, we show in this paper the results of its application on the recovery of a ‘lost’ painting from the Etruscan hypogeal tomb of the Volumni (3rd century BCE—1st century CE) in Perugia, Italy. The results obtained applying the IFF algorithm are compared with the results obtained by applying Blind Source Separation (BSS) techniques and Self-Organized Maps (SOM) to a multispectral set of 17 fluorescence and reflection images. From this comparison emerges the possibility of using the IFF algorithm to obtain rapidly and simultaneously, by varying a single parameter in a range from 0 to 1, several sets of elaborated images all containing the ‘interesting’ features and carrying information comparable to what could have been obtained by BSS and SOM, respectively.

1. Introduction

The human species has used colors since the remote past for decorating objects and representing the surrounding environment, in a process which is difficult not to call ‘artistic’ [1]. Some of these expressions of human creativity have been, unfortunately, severely damaged by the combined effects of time and environment but also, in many cases, by the alteration of their closed environment, at the time of their discovery, caused by human presence.

The examples of such degradation are innumerable. To remain in the temporal and geographical framework of the example that will be discussed in this paper, the mural paintings of the Etruscan hypogeal Tomb of the Monkey in Chiusi, Siena (5th century BCE) have been seriously damaged by anthropic impact, after the discovery of the tomb in 1846 by the Florentine archaeologist Alessandro François [2]. A similar fate was suffered by the paintings of the Etruscan tombs of the Blue Demons, which have been recently studied by Adinolfi et al. [3].

In some cases, the remaining traces of the original paintings can only be interpreted thanks to the drawings realized by the professional designers which were often called to document the paintings after the discovery of the tombs [4,5]. However, in some cases, either we don’t have such documentation [3] or the correspondence to the original of the drawings can be questioned—for various reasons—the principal one being the difficulty in accessing the archaeological site [6] and/or the limited legibility of the paintings since the moment of their discovery.

The paramount importance of developing scientific tools able to recover the legibility of ancient ‘lost’ paintings is clear. A common (and powerful) approach to the problem is the use of multi/hyper-spectral imaging techniques [7,8], which have been developed in the last decades following technological advances in digital photography and landscape surveying [9]. However, in most cases, multi/hyper-spectral methods are used for the identification of the pigments in the painting [10,11,12,13] and/or studying the underdrawings through infrared imaging [14,15,16,17,18,19]; less frequent are the studies aimed at the recovery of the legibility of the images. Important works have been done in the past decade towards the recovery of the visibility of degraded manuscripts [20,21,22,23] but some effort has also been devoted to ancient mural paintings, one noteworthy example being the development of the DStretch^® software [24] and its application to rock [25] and wall [26] paintings. However, the software uses only three spectral bands (usually RGB) for its analysis, and this is an important limitation. Other methods have been proposed which exploit the full multi/hyper-spectral set for revealing hidden details in paintings [27]. Among them, a particularly efficient procedure is MHX (Multi-illumination Hyperspectral eXtraction), developed by the National Research Council of Pisa together with Art-Test of Luciano Marras as a combination of advanced illumination sources (high purity UV sources plus visible and infrared illuminators), with a dedicated multispectral camera for the acquisition of low noise, high resolution spectrally resolved images coupled to a series of different Blind Source Separation (BSS) algorithms, described in detail in ref. [28], for the extraction of the hidden or invisible features of the painting.

The MHX method has been improved over the years from its initial applications for manuscripts [23,29,30,31,32,33,34,35,36,37] to the analysis of ‘lost’ paintings [3,28,38,39,40].

2. The Volumni Hypogeum

One of the most successfully application of the MHX procedure has been the recent study of the painted urn of Arnth Velimna in the Etruscan Volumni hypogeal tomb in Perugia, Italy. The site was frequented from the 3rd century BCE to the 1st century CE and is part of a vast necropolis that can be dated back to the 5th century BCE.

The complete description of the experimental procedure, the analysis of the multispectral set and its interpretation are described in a paper in press in the prestigious specialized journal ‘Studi Etruschi’ [41]. However, for the purposes of this paper (the discussion of a new approach for the analysis of the multispectral set), it would be sufficient here to summarize the main results of the study.

The urn of Arnth Velimna represents, on its front, the Door of the Hades, a symbolic element which is present in many Etruscan paintings to signify the passage from the world of the living to the world of the dead. The painting is strongly degraded, especially in its upper part, which results in its being completely illegible to the naked eye (see Figure 1a).

The Volumni hypogeum was casually discovered in 1840 during the repairing of a nearby road. The archaeologist Giovanni Battista Vermiglioli supervised the opening of the hypogeum and, in the same year, published a drawing of the painting [42] showing two figures in the upper part which are no longer visible (see Figure 1b).

However, the style of the drawing and the clothing of the two individuals in the foreground is not fully convincing, a demonstration that the legibility of the painting was not optimal even in the immediacy of the opening of the tomb.

3. Acquisition of the Multispectral Image Set

In an experimental campaign started in April 2021 we acquired with a specialized multispectral camera (Chroma from DTA, Cascina (Pisa)) a set of 17 low-noise, high-resolution spectrally resolved images, 9 in UV/Vis reflection (from 350 nm to 750 nm in 50 nm steps), 3 in the infrared band (from 850 nm to 1050 nm in steps of 100 nm) and 5 in UV-Vis/IR fluorescence (from 450 to 750 nm in steps of 100 nm). For selecting the spectral bands, we used a set of interferential filters with passbands of ±25 nm around the central wavelength that were placed in front of the camera objective. The corresponding images are shown in Figure S1 in the Supplementary Information.

From Figure S1 it can be seen that a few details of the painting reappear, especially in the UV-IR fluorescence images (750 nm and 850 nm). Moreover, the infrared reflection images at 950 nm and 1050 nm evidence the presence of a recent writing drawn probably with a pencil over the heads of the two individuals in the foreground, presumably in a moment in which the visibility of this part of the painting was already compromised. No further information can be obtained from a direct observation of the single images of the hyperspectral set. By applying the BSS algorithms which are part of the MHX procedure, however, a much better recovery of the painting’s legibility can be achieved.

Two of the BSS images are shown in Figure 2.

The position of the individuals and of the text in the painting is shown in Figure 3.

The MHX BSS techniques recover very efficiently the legibility of the painting (Figure 2a) and also improve the details of the writing, which reveals the presence of two sets of initials and two dates, probably both referring to the year (19) 51 (Figure 2b). Thanks to the MHX method, it is now possible to compare the recovered image in Figure 2a with the Vermiglioli 1840 drawing (Figure 1b) evidencing the differences in the attire and posture of the couple in the forefront, with the hand of the woman resting on the shoulder of the man in an intimate act that was lost in the Vermiglioli drawing. However, despite the exceptional results of the MHX BSS algorithms, some doubts remain on the representation of the wings of the individual (probably a demon) in the background at the left, which can be, on the other hand, guessed by the fluorescence images at 750 nm and 850 nm (see Figure S1 in the Supplementary Materials).

In fact, the wings do not appear clearly in any of the BSS recovered images (see Figure 4a,b). In an attempt to solve this last issue, we tried the application of an alternative method for the analysis of multi/hyper-spectral sets, called Self-Organizing Maps (SOM) [43]. This method allows for the clustering of the pixels in the multispectral set according to their spectral components (hypercolors). With respect to other clustering methods, the SOM algorithm does not require an a priori knowledge of the number of clusters expected; moreover, contrarily to the BSS algorithms, the SOM clusters are associated to a corresponding prototype hypercolor, a feature that may be important for recovering information about the spectral response of the original colors of the painting.

In the case of the Door of the Hades, the SOM algorithm clusters the pixels of the hyperspectral set in 16 main groups, shown in Figure S2 of the Supplementary Materials. Also in this case, the outline of the wings does not appear clearly (see Figure 4c), but essentially confirms the information extracted by the MHX BSS algorithms, as shown in Figure 4c (the images correspond to the region evidenced in blue in Figure 1).

The average spectral reflectivity of the pixels in the SOM cluster graphically represented in Figure 4c is shown in Figure 5.

From the reflectivity curve, we can infer that the color of the wings corresponds to a yellowish white, which is coherent with the few remains still visible in the RGB image (Figure 6).

The difficulty of both MHX BSS and SOM algorithms to recover the visibility of the demon wings leads to the consideration that both algorithms do not perform well in the presence of details which do not contribute too much to the variance in the hyper/multi-spectral set. This is particularly evident for SOM, which does not recover the two signatures in Figure 3, for example, while the MHX BSS algorithm can extract their spectral information, instead.

In the case of the wings, not only do they correspond to a small fraction of the pixels in the image, but also their (hyper)color is similar to the background, hence the difficulties of both methods in extracting the corresponding information.

Given this situation, it is thus worth exploring the possibility of using a different feature extraction method, that would recover the ‘interesting features’ independently of the variance they represent in the multispectral set.

4. Discussion

4.1. The Interesting Features Finder (IFF) Method

The Interesting Features Finder method has been recently proposed by Wu et al. [44] for the analysis of spectral sets acquired for multi-elemental Laser-Induced Breakdown Spectroscopy (LIBS) imaging applications. The progress in LIBS analysis currently allows the acquisition of large spectral datasets characterized by high spatial (>1 Megapixel) and spectral (>20,000 spectral points) resolutions [45]; such a large amount of data should be processed using chemometric methods for determining the chemical nature of the different components of the sample [46,47]. However, the Principal Component Analysis approach [48], which is one of the simplest and more used methods for this task, fails to recover features that express a low variance in the hyperspectral set. In general, all the techniques based on decorrelation or independence of the sources, as the BSS algorithms used in the MHX protocol, might potentially suffer from the same problem.

The proposal of the authors of ref. [44] was probably inspired by the Pixel Purity Index (PPI) method, which determines the extreme points in the hypercolor space by projecting the hypervectors corresponding to each pixel on many random vectors (>10,000).

In the PPI method, a single image is then built, in which the intensity of each pixel is proportional to the number of times in which that pixel has been marked as an extreme. The authors of [44] consider, in fact, the extreme pixels, but are interested in their spectral signature, instead, because the vectors defined in the hypercolor space by the purest pixels can be considered as a basis of independent components, and all the other pixels of the multispectral image can be expressed as linear combinations of these components. A simple Matlab code for the selection of the extreme points in the hypercolor space is provided by the authors of [44] as supplementary material in their paper.

4.2. Application of the IFF Method

To exploit the IFF method, it is necessary to reduce the number of the independent components, removing all the pure pixels that have similar spectral signatures of the purest one. In the case of the Door of the Hades, imposing an upper threshold of 0.95 on the correlation between the pure pixels for considering the corresponding spectral components as independent, we obtain a set of 23 components, whose linear combination would represent the hypercolors of all the pixels of the hyperspectral set.

Having so defined the basis of our hyperspectral set, it is now possible to reproject the pixels on the new basis to obtain a series of images where the intensity of each pixel is given by the scalar product of the corresponding hypervector with each vector of the basis.

The resulting 23 images are shown in Supplementary Information Figure S3.

The images obtained using the IFF method are visually similar to the ones that can be obtained using the MHX BSS algorithms (see Figure 7).

An important difference is, however, that each image is associated to the hypercolor of the corresponding independent component (pure pixel); this association can be better exploited by imposing an intensity threshold so that only the pixels very close to the corresponding pure pixel hypercolor would be displayed in the image. The definition of this threshold is arbitrary; it may range from 0 (Figure S3) to a value close to 1 (100% of the maximum intensity); in Supplementary Information Figure S4 we show the same IFF images of Figure S3 after the application of an intensity threshold of 85% of the maximum.

By applying an intensity threshold, the IFF images become very similar to the SOM images shown in Figure S2, with some important differences. In this case, the pixels corresponding to the signatures are, although partially, separated in a single image (see Figure S4); moreover, there are several images containing very few pixels, which is exactly what we were expecting given the characteristic of the IFF method of selecting pure pixels, independent of their impact on the variance of the hyperspectral set.

As for the demon wings, their visibility in the IFF image set at the zero and 0.85 thresholds (see Figure 8) is similar to what was obtained using the MHX BSS and the SOM algorithms, respectively.

Therefore, in this specific application, the application of the IFF method does not improve the legibility of the painting with respect to the BSS and SOM algorithms used in the MHX protocol. However, the IFF is considerably faster than the combination of the BSS + SOM algorithms. On a medium-power laptop computer, the non-optimized IFF algorithm produced the 17 images of the multispectral set considered here from 10,000 random vectors in 28 s, compared to 23 s of the BST algorithm and 164 for the SOM (4 × 4 grid, 1000 epochs).

5. Conclusions

The IFF method, although in the case here discussed does not provide better results than the BSS methods used in the MHX protocol, is interesting for several reasons. The method is normally very fast in its application to the typical hyperspectral sets that can be acquired with a variable filter CCD camera; moreover, each image that can be obtained is associated with a defined hypercolor and, with the application of a proper intensity threshold, it is possible to pass continuously from a visualization comparable to the one obtained with the most advanced BSS techniques to spectral information substantially equivalent to the one obtained through the SOM algorithm, with the additional potential capability of extracting details associated to few pixels and/or characterized by hypercolors poorly contrasted against the background.