The CaliPhoto Method

Abstract

We propose an innovative method based on photography and image processing of interdisciplinary relevance, permitting the uncomplicated and inexpensive evaluation of material properties. This method—CaliPhoto—consists of using a dedicated colour plate with a specific design, placed in the field of view of a photograph of the material to be characterized. A specific image processing workflow is then applied to obtain colour vectors independent of illumination conditions. The method works using commercial colour cameras (e.g., smartphone cameras), and the colour plate can be printed on any colour printer. Herein, we describe the principle of the method and demonstrate that it can be used to describe and compare samples, identify materials or make relatively precise concentration measurements. The CaliPhoto method is highly complementary to any scientific research and may find applications across a range of domains, from planetary science to oceanography. The method may also be widely used in industry.

1. Introduction

Colour is the primary parameter used in many domains to identify and characterize a material. It is used for quality control in such industries as food [1], building [2,3] or textiles [4], amongst others. For scientific purposes, colour observation is a rapid method to estimate the composition of a material, as in the geosciences (Munsell soil colour charts [5]) or in biology (e.g., dermatology [6]), to monitor changes in physical properties (e.g., rock ageing [7,8]), or to obtain specific physical properties (e.g., temperature or pH). Nevertheless, the apparent colour of material is directly dependant on light conditions and photon detectors; the human eye is limited and person-dependant (e.g., colour blindness), and camera detectors have different colour responses. Precise colour measurement is thus made using UV-Vis spectroscopy or, if based on photography, requires calibrated cameras and the use of specific light sources (CIE Standard Illuminants D50 or D65 for example) [9]. Thus, despite its ubiquitous utility for material characterization, colour is rarely used to evaluate sample properties in scientific studies.

Here, we describe a new method for material characterisation, named CaliPhoto (patent n° FR3042058), which uses colour as acquired by any commercially available camera (e.g., smartphone, tablet, webcam, compact camera, etc.) and is possible with an uncalibrated light source (e.g., the Sun, halogen lamp, LED, etc.). Cameras are probably the most common system of observation, and most scientists use photography. In addition to a camera, the method simply requires the use of a specific reference colour plate, which can be printed on any colour printer and placed in the same field of view as the material of interest. We have developed an image processing methodology, using the RGB parameters of the reference colour plate (8 bit encoding, meaning that R, G and B may vary between 0 and 255), which can be used to obtain RGB values independent of both the camera and the light source for the material of interest. This method can be used to identify or to compare materials, and also to determine specific physical properties and follow their changes, in different light conditions. Due to its very low cost, portable nature (requiring only a sheet of paper) and the facility and rapidity of its implementation, the method is particularly relevant for field investigations. It permits the comparison of materials located in different places without the need for sampling and field instrumentation and is thus highly applicable for use in harsh environmental conditions or isolated areas such as mountains, cliffs, deserts, volcanoes, deep seas, caves, archaeological sites or extraterrestrial bodies. Indeed, the method was originally developed for space exploration purposes, to aid in the identification of rocks during in situ exploration where instrumentation is generally limited for technical reasons. It is also a good companion for laboratory investigations, enabling the precharacterisation and selection of the samples best suited to be analysed in detail using potentially slow and expensive methods, or to rapidly monitor changes in material properties.

This paper explains the CaliPhoto protocol and is applicable to a wide range of scientific studies. The main part of this paper is thus voluntarily short. Nevertheless, the detailed protocol and different validation tests are given in the Appendices so that any scientist can take advantage of its use. The algorithm (encoded in MATLAB) is available online in the Supplementary Materials of this article and may be used freely for any noncommercial purpose.

2. Materials and Methods

The principle of the CaliPhoto method comprises several steps:

2.1. Printing the CaliPhoto Colour Plate

Since they cover a wide and relevant range of colours, the RGB values of the colour squares of the CaliPhoto colour plate used in this study are similar to those of the GretagMacBeth ColorChecker typically used for colour calibration (Figure 1) [9,10]. It is expensive and, moreover, quite impossible to print a colour plate with the exact theoretical colour values displayed in Figure 1. Nevertheless, contrary to colorimetry methods, the aim of the CaliPhoto method is not to obtain true colours, as for calibrating a camera or a printer, nor to obtain colours representative of human vision. The CaliPhoto colour plate (Figure A1) can thus be printed using any commercial printer and paper. The size of the colour plate, ranging from a few cm to several tens of cm, is chosen according to the material of interest and the imaging system used (see Appendix A for details). Note that the colour palette used in Figure 1 has been chosen only as an example and can be changed to be more relevant to the colour chart of the studied materials. Similarly, their number can be increased or decreased if required (and the code given in Supplementary Materials will need to be modified accordingly).

2.2. Make a Reference Image of the Printed CaliPhoto Colour Plate

The processing required to obtain the reference image of the colour plate is explained in Figure 2. Since the colour plate has been printed using a noncalibrated printer, it is necessary to make a reference image of that colour plate taking into accounts its specific colours. The CaliPhoto method requires neither high-quality colour printing nor a specific light source or camera. The reference image is obtained by simply acquiring a photograph of the printed colour plate (Figure 2a). The relatively unrestricted recommendations for the acquisition of this photograph are detailed in Appendix B. The image is then rescaled and resized in order to remove perspective distortion and to obtain an image with a width/height ratio similar to that of the numerical model (i.e., a 4:3 ratio rectangle without the logo band). The resolution of the images may also be fixed; here we used a resolution of 1200 pixels × 1020 pixels including the logo band (Figure 2b). This can easily be made using image processing software, such as GIMP (see Appendix B, Figure A3 for details). Since it is difficult to control the orientation and homogeneity of the light source, the external and internal white rectangles of the colour plate (see Figure 1) are used to evaluate and correct the inhomogeneity of the illumination. This approach is inspired by the standard “plane fitting” data processing used in Scanning Probe Microscopy in order to remove the slope induced by the nonparallelism between the sample surface and the scanning plane (e.g., [11]). Indeed, in the absence of shadow, the light orientation will form a brightness continuum over the colour plate that can be fitted using a polynomial surface (see details in Appendix B, Figure A4). The equation of this plane can then be used in order to remove the illumination inhomogeneity over the whole image. In order to avoid values higher than 255, the average R, G and B values of the white rectangles are reset to 230 (Figure 2c). The fourth step consists of setting the brightness and contrast of the image. This is done by using a polynomial regression of order 3, linking the R, G and B values measured on the previously corrected image to the theoretical values displayed in Figure 1 (Figure 2d). These two steps also compensate the chromaticity of the light source (i.e., after correction, R ≈ G ≈ B in the greyscale squares). After correction, the colours of the squares in the reference image (Figure 2d) are different from the theoretical values (Figure 1).

2.3. Making Comparable Images of the Materials

The aim of the CaliPhoto method is to compare different materials based on their colours when observed in different light conditions. The idea is thus to process the images of the materials using the colour plate in order to obtain images where all the colour squares have similar RGB values to those of the reference image (Figure 2d). The image of a material is obtained by placing the sample below the window of the same printed colour plate as used previously. It is important that the colour plate is as close as possible to, and ideally in contact with, the surface of the material of interest. This is because the light conditions must be the same on both the colour plate and the sample. Image processing is then similar to that used in order to obtain the reference image of the colour plate (Figure 2). Following this, a complementary step is applied to correct the colours of these new images, resetting the RGB values of the colour squares of the colour plate to those of the reference image using a polynomial transform similar to those explained in [6,12] (Figure 3). We have demonstrated that a polynomial regression of order 11 yields the best correction (see Appendix C).

After correction, the RGB values of the colour squares in the image of the sample must be similar to those on the reference image of the colour plate; however, the correction can never be perfect. The accuracy of the image processing has been tested under different illumination conditions, using different imaging systems and for different image resolutions (see Appendix C and Appendix D). In order to evaluate the colour correction of the images using the CaliPhoto method, the final RGB values of the colour squares on the final images of the samples are compared to those of the reference image of the colour plate using Δ, defined as:

$$\Delta =\frac{1}{24}{\displaystyle \sum }_{i=1}^{i=24}\frac{\sqrt{{\left(R_i^{ref}-R_i^{im}\right)}^2+{\left(G_i^{ref}-G_i^{im}\right)}^2+{\left(B_i^{ref}-B_i^{im}\right)}^2}}{\sqrt{{255}^2+{255}^2+{255}^2}}$$

(1)

where R_i^ref, G_i^ref and B_i^ref are the average RGB values of the square i in the reference image of the colour plate and R_i^im, G_i^im and B_i^im are the average RGB values of the corresponding square in the corrected image.

The detailed results of the tests are reported in Appendix C and Appendix D; however, one can conclude that:

• The best correction is obtained when the size of the colour plate in pixels on the image is close to that of the colour plate on the reference image before resizing.
• The correction of an image with respect to the reference image remains good, even in low-light environments. It is shown that Δ < 3% when the luminance µ = (R + G + B)/3 of the white square in the sample image is higher than approximately 100, and when the image dynamic (i.e., µ of white square minus µ of black square) is higher than approximately 100.
• The “colour” of the light does not play a role in the final correction; for a difference of 54 between the highest and the lowest of the three RGB values of the white square, Δ = 1%.
• It is better to use the same camera throughout the whole procedure, even if Δ < 2% when using different imaging devices.

Finally, the tests show that in “normal” conditions, the method allows the user to obtain a dataset of images in which the difference between the RGB values of colour squares in the colour plate (between two images) is always approximately Δ ≈ 1%. In other words, this means that the percentage of uncertainty in the CaliPhoto colour of a material (i.e., its colour plate-dependent colour) is only about 1%.

2.4. Comparison of Materials from the Colours Given by CaliPhoto

For samples that are relatively homogeneous in colour, materials can be compared using their average CaliPhoto RGB values in the following equation:

$${\Delta }_{ij}=\frac{\sqrt{{\left(R_i-R_j\right)}^2+{\left(G_i-G_j\right)}^2+{\left(B_i-B_j\right)}^2}}{\sqrt{{255}^2+{255}^2+{255}^2}}$$

(2)

where R_i, G_i, B_i and R_j, G_j, B_j are the average CaliPhoto RGB values of the sample i and j respectively. Nevertheless, depending on the materials of interest, more complex vectors may be defined (see Appendix E). In particular, we propose to use the histograms of the rg chromaticity (r,g,b) and luminance (µ) parameters and to merge them into a unique colour vector Vec, which is defined for material i as:

$$Ve{c}_i=\frac{Hist\left(r_i\right){\displaystyle \cup }\left[1+Hist\left(g_i\right)\right]{\displaystyle \cup }\left[2+Hist\left(b_i\right)\right]{\displaystyle \cup }\left[3+Hist\left({\mu }_i/256\right)\right]}{Max\left\{r_i,g_i,b_i,\left(\frac{{\mu }_i}{256}\right)\right\}}$$

(3)

where Hist(r_i) corresponds to the histogram of r_i, rounded to 0.05, over the sample area, [1 + Hist(g_i)] to the histogram of g_i, rounded to 0.05, displayed between 1 and 2, [2 + Hist(b_i)] to the histogram of b_i, rounded to 0.05, displayed between 2 and 3, and [1 + Hist(µ_i/256)] to the histogram of µ_i/256, rounded to 0.05, displayed between 3 and 4, and Max {r_i, g_i, b_i, (µ_i/256)} to the maximal values of the different histograms. This simple method allows the inhomogeneity of the colour of a material to be easily taken into account while avoiding complex textural image processing. Finally, for samples that are relatively homogeneous in colour, materials 1 and 2 can be compared based on their colour vector using the following equation:

$$\Delta Ve{c}_{12}=\frac{\sqrt{{\left(Ve{c}_1-Ve{c}_2\right)}^2}}{Ve{c}_1+Ve{c}_2}$$

(4)

3. Results

The CaliPhoto method permits the comparison of materials from images taken under different light conditions. The applications of this approach are thus very wide and numerous. For instance, in the field, it can be used to measure compositional variations across a rock outcrop or to compare the different sides of an object, from a hand sample to a geological structure or a building, where the shadows and illumination necessarily vary between the different snapshots (change of illumination conditions with time and space) (Figure 4).

In addition to allowing the comparison of different materials, this method is also relevant for monitoring physical processes associated with colour change, such as alteration or chemical reactions (oxidation, weathering, irradiation, water content, etc.).

Furthermore, by creating a database of CaliPhoto colour vectors (or RGB values) the method can be used to perform preliminary characterisation of an “unknown” material based on analyses of comparable materials using laboratory-based instrumentation. The philosophy is then to link properties obtained using expensive and/or nonportable instrumentation to measurements made by low-cost, highly portable imaging. The principle consists of characterising the materials expected to be found in the field, using laboratory instruments if needed, to make their CaliPhoto image following the process previously described, then to store their RGB CaliPhoto vectors in a database linking these vectors to the different materials or to any other relevant physical parameter(s). By comparison with the database, it would then be possible to identify or determine the physical properties of an unknown material using only a photograph. The method is resilient: a database can be made and completed at any time and, thus, the characterisation of “unknown” materials is continuously improved and refined with time. Figure 5 illustrates the use of the CaliPhoto method for sample identification and for compositional measurement (detailed results can be found in Appendix E and Appendix F).

Finally, the only restriction of the method is the fact that it is necessary to associate a database to a specific colour plate. If this colour plate were to be lost or altered, the database would become obsolete. We thus propose to maintain a Master Colour plate in a secure place so that the reference image of a new colour plate can be recalibrated for use with a previously established database (see Appendix G).

4. Discussion

The rationale of the CaliPhoto method is not to provide extremely precise measurements, but to rapidly and easily give preliminary results using a very versatile and uncomplicated, low-cost, portable system. It permits the evaluation of properties that would normally need to be studied using expensive, time-consuming, high-tech methodologies or instrumentation. It can be used as a rapid monitoring and precharacterization method in the field or in the laboratory and it may, for instance, help in identifying the most appropriate samples for further in-depth analysis. The CaliPhoto method thus has applications in various domains:

• Space exploration: As mentioned above, instrumentation used during in situ space exploration is generally limited for technical reasons (limited mass, volume, energy and computational power). On the other hand, all space probes have on board a camera. Integrating a CaliPhoto colour plate with imaging systems on future space probes would be simple since it easily complies with the imposed technical limitations and does not require the development of new instrumentation. It could be particularly relevant for geological and astrobiological investigations, permitting the comparison of images of rocks taken in different places or evaluation of lithologies using images of rocks taken in the laboratory, for example.
• Construction: The method may find different applications in this domain, for example, to check the homogeneity of materials over large areas or on the different faces of a structure, to monitor the ageing of materials over space and time, or to control the water content of cement.
• Geology: As for space exploration, the method allows easy comparison of samples observed in different places in the field. When associated with a database, it can also facilitate the use of soil colour charts.
• Biology: Colour is used as a parameter to pre-identify microbial colonies or to evaluate their reaction to external stresses (sunlight, dehydration, pH, salinity, etc.). This applies more generally for any living system (the yellowing of leaves, for example). The CaliPhoto may thus be used as a noninvasive and rapid method of monitoring such types of samples.
• Materials science: Changes in the physico-chemical characteristics of materials may be associated with a change in colour. The possibility to monitor such changes during specific experiments may be particularly useful in determining reaction kinetics. Association with a database linking specific properties to CaliPhoto colours may circumvent or limit the systematic use of sophisticated instrumentation. It may help to refine measurements based on colour, such as those obtained using pH paper.
• Submarine investigation: Underwater imaging is generally affected by water colour and inherently limited control of the light source. Submarine instrumentation is also limited compared with laboratory instrumentation. The CaliPhoto method may thus also be particularly useful in this setting. By linking physical properties obtained by laboratory analysis to a CaliPhoto database, it could be possible to estimate these properties from images made underwater, without sampling.
• Cosmetics and dermatology: The CaliPhoto method may assist in the selection of makeup adapted to specific types of skin. It may also provide a control for Sun exposure to prevent sunburn. In the latter case, the idea would be to take pictures regularly of the same part of the body to monitor any change in colour.

Several studies on various applications of the CaliPhoto method are in progress in field geology and space exploration. In particular, the method will be adapted for the upcoming European Space Agency (ESA)/Roscosmos ExoMars 2020 mission to Mars to assist in the identification of rocks using the Close-Up Imager (CLUPI) camera [13].

While the CaliPhoto method has many applications for scientific purposes, it could also be used by the general public, for which an app for smartphones and tablets is under development. This implies automation of the method since manual image processing and MATLAB code cannot be used. Significant work has thus been carried out to integrate the automatic detection of the colour plate and of the sample area, and for automated comparison with databases. Finally, prototypes of apps and specific colour plates are currently under development for cosmetics, paints and decoration, for example, to identify complementary colours of textiles (N.B. in aesthetic cases, the matching percentage uses the Lab colour system to take into account the higher sensitivity of the human eye to green). The method could also be particularly relevant for assisting the colour-blind.

Finally, it is important to note that the CaliPhoto method is in line with the long-term vision for future developments in scientific technologies that favour low-cost solutions prior to methods that consume large amounts of resources. The philosophy of this approach is now echoed in many domains concerned by environmental considerations and future limitations in resources.

5. Patents

A patent has been deposited for this method under the number FR3042058 (A1).