Reflectance Transformation Imaging Visual Saliency: Local and Global Approaches for Visual Inspection of Engineered Surfaces

Abstract

Reflectance Transformation Imaging (RTI) is a non-contact technique which consists in acquiring a set of multi-light images by varying the direction of the illumination source on a scene or a surface. This technique provides access to a wide variety of local surface attributes which describe the angular reflectance of surfaces as well as their local microgeometry (stereo photometric approach). In the context of the inspection of the visual quality of surfaces, an essential issue is to be able to estimate the local visual saliency of the inspected surfaces from the often-voluminous acquired RTI data in order to quantitatively evaluate the local appearance properties of a surface. In this work, a multi-scale and multi-level methodology is proposed and the approach is extended to allow for the global comparison of different surface roughnesses in terms of their visual properties. The methodology is applied on different industrial surfaces, and the results show that the visual saliency maps thus obtained allow an objective quantitative evaluation of the local and global visual properties on the inspected surfaces.

1. Introduction

Mastering the functional properties of manufactured surfaces is a major scientific and industrial challenge for a wide variety of applications [1]. The functional properties of surfaces cover a large spectrum. A non-exhaustive list includes tribological properties [2,3], properties associated with material mechanics [4], wettability [5], biological properties, and functional parameters linked to light–surface interaction. This approach induces many issues and scientific challenges, in particular, the choice of measurement technology, the choice of the scale for both measurement and analysis tasks, and the choice of filtering and post-treatment methods. In this context, the RTI imaging method is currently experiencing significant growth [4,6,7] and is positioned as a complement to conventional techniques in industry, which are often based on 3D microscopy for the measurement of surface roughness.

During the RTI acquisition, the orientation of the light source varies around the surface and passes through a hemispherical space (constant distance). A fixed camera positioned orthogonally to the surface captures the response of the surface (local reflectance) for different lighting angles (angular reflectance). From the acquisition, N stereo-photometric type images are obtained, each pixel of which corresponds to a discrete measurement of the reflectance of the corresponding point on the surface, as shown in Figure 1, where $lu$ and $lv$ represent the components associated with the directions of illumination projected in the horizontal plane.

RTI was originally developed at HP labs by Malzbender et al. [8,9] under a name, Polynomials Texture Mappings (PTM), that refers to the implemented modeling method. Other modeling methods have been developed, such as the HSH (Hemispherical Harmonics) model [10,11,12,13], the DMD (Discrete Modal Decomposition) method [14,15,16,17], and the approach based on RBF (Radial Basis Function) [18,19]. Particularly important development has occurred in the field of the digitization of historical and cultural heritage objects [20,21,22]. More recently, this technique has been implemented in the context of a wide variety of industrial applications. These applications have often been related to the challenge of mastering of surface appearance perception and the automation of inspection processes [23,24], although other applications have been found in biomedicine, forensics [25], materials mechanics [4,26], and the analysis of tribological properties [17]. The richness of the contribution of the RTI technique is linked to the quality and variety of surface information that it can estimate. Indeed, this type of acquisition makes it possible to characterize the local angular reflectance of surfaces, i.e., the way light is re-emitted at each point as a function of the angle of illumination. Certain descriptors can statistically describe the angular reflectance response of a surface from RTI data (Figure 2). This type of information is particularly relevant for issues related to surface appearance quality [27], where the challenge lies in increasing the robustness of the results obtained from conventionally implemented sensorial analysis [28,29,30]. The RTI technique makes it possible to estimate characteristics linked to the local micro-geometry of surfaces, such as local descriptors of slopes, curvatures, and even topographic descriptors, through integration following the stereophotometric technique. In addition, the specularity and diffusion components provide access to sub-pixel light behaviors, as responses from multiple surface points can be integrated into a single pixel. Thus, we can observe the response of a point on the surface despite it being geometrically of a smaller size than the resolution of the sensor. RTI acquisition and characterization makes it possible to more directly measure and describe aspect function, i.e., the effect that is not geometry intrinsic and is instead created by the geometry of the surface [7,31].

Therefore, depending on the density of the acquisition angles and the methods chosen, RTI acquisition can lead to obtaining a voluminous and complex set of data [18,32,33]. Estimation of the descriptors constitutes a first step in data reduction, which allows a better understanding of the local surface characteristics and facilitates the analysis and post-processing carried out based on this type of acquisition. However, these descriptors vary in terms of unit, amplitude, and even dispersion; analyzed independently, they generally do not make it possible to match the various stakes of analysis in the context of controlling the appearance of manufactured surfaces. In this article, we propose a methodological contribution to answer this problem. The proposed method consists in locally (intra-surface) or globally (inter-surface) estimating the visual saliency from the data and descriptors extracted from RTI acquisitions.

Indeed, saliency maps aim to respond to an important issue for manufacturers, mentioned above, which is to allow better detection while assessing the criticality of aspect anomalies. In Section 2, we show how a multivariate and multi-scale analysis of the descriptors previously extracted from the RTI acquisitions can be used to determine the local saliency of the points/pixels of the inspected surface in an efficient way. This approach is then extended in Section 3 to the analysis of global anomalies, i.e., the mapping of the difference in appearance between one or more surface states and a reference surface state. These maps make it possible to functionally quantify the distance between two surfaces, i.e., in terms of overall appearance thereby meeting the challenge of managing the manufacturing and surface finishing processes, which require that differences in terms of appearance between surfaces be discriminated and evaluated.

2. Visual Saliency Estimation

The visual saliency of a pixel of an image corresponds to its ability to attract attention in relation to other points on the surface, in particular in relation to its neighbors. When observing a set, a human concentrates most of their perceptual and cognitive resources on the most salient subset. Therefore, estimation of visual saliency is an important issue in the analysis and characterization of the appearance of surfaces. The mapping of points with appearance attributes that are different from those of their environment and the quantification of these deviations are essentials that provide objective support for many surface inspection and analysis tasks. In the literature, an approach for calculating saliency from RTI data has already been proposed by Pitard et al. [16] based on the analysis of the Discrete Modal Decomposition coefficients, an experimental model for relighting. Here, we propose to extend this approach using the set of descriptors extracted from RTI data and to improve the performance of visual saliency estimation on multiple levels by taking multi-scale aspects into account in the calculation.

In order to illustrate the proposed methodology, we apply it to an application surface associated with the watch industry, specifically, a watch dial with a precious material deposit and a polished finish. Manufactured micro-scratches in known locations and with different geometric characteristics were made on the surface with a scratch test machine (Anton Paar $MC{T}^3$). The characteristics of this sample are shown in Figure 3. The acquisitions were carried out with a custom device consisting of a monochromatic two-thirds active pixel-type CMOS sensor with a resolution of 12.4 MPix ($4112\times 3008$) and a white uniform light (high power LED). In terms of RTI acquisition modalities, 149 angular positions distributed homogeneously in the angular space ($\varphi ,\theta$) were acquired, with an exposure time of 125 ms.

2.1. Proposed Methodology

In order to identify and evaluate surface anomalies in the context of appearance quality control, it is necessary to be able to compare the points of the surface with each other using an objective criterion. A point is an anomaly when its behavior is different from its neighborhood or from the sought function. As the comparison cannot be absolute, only relative, this amounts to calculating distances between surface points. The distance, called the saliency in the case of intra-surface distances, is estimated from a multivariate analysis of the descriptors of the observed pixel descriptors Y relative to the average values of descriptors of the reference pixels X associated with the neighborhood under consideration. The method chosen here is based on the Mahalanobis distance [34], as illustrated in Figure 4, and is described in Equation (1) below:

$$Maha{l}^{*}(Y,X)=\sqrt{{[Y-\mu \left(X\right)]}^T\Sigma {\left(X\right)}^{-1}[Y-\mu \left(X\right)]},$$

(1)

where the matrix $Y=[Y_1,Y_2,\dots ,Y_n]$ corresponds to the descriptor vectors of length k of n observation pixels and the matrix $X=[X_1,X_2,\dots ,X_m]$ corresponds to the descriptor vectors of length k of m reference pixels used to estimate the distance. Here, $\Sigma$ and $\mu$ are respectively associated with the covariance matrix and the average of X over the dimension of the descriptors. During computation of the distance, the centering $Y-\mu \left(X\right)$ is performed on the dimension of the descriptors.

The Mahalanobis distance has the advantage of being a unitless and data scale-invariant multivariate method of analysis, and takes into account both the scatter and correlation of the data. Thus, several different unit and scale descriptors can be used when calculating the Mahalanobis distance, and the different descriptors can be weighted according to their respective dispersions. The choice of descriptors used when estimating saliency is important, as each allows the discrimination of different behaviors, which may potentially be antagonistic. Therefore, it is necessary to estimate which descriptors are the most relevant for characterizing a desired behavior, for example, a type of aspect abnormality, or to choose them based on experience or prior knowledge. An example is presented in Figure 5 highlighting the the significant difference in the saliency maps (monovaried in this case) depending on the choice of descriptor(s), here, the geometrics and the statistics descriptors, respectively, of the angular reflectance response.

2.2. Multi-Level Optimization Criterion

When estimating saliency, the most salient pixels sought are included among those used for reference. However, including salient pixels among the reference pixels can significantly modify the value of the descriptors. Depending on the case, the saliency estimated at each point can be over- or underestimated by the number and intensity of the salient points in the zone considered. For example, the salient points of the surface may have their saliency levels underestimated when are compared, in terms of distance, with points of the surface that have the same salience as themselves. If these points are salient, they probably have similar characteristics, and therefore similar descriptors. On the other hand, the non-salient points of the surface have their saliency levels increased because, althoough their characteristics are far from the salient ones, they are used as a reference. To overcome this problem, we propose the level factor saliency optimization method described in Equation (2). This method allows both the standard deviation and the mean of the saliency of the reference pixels used for the saliency computation to be reduced by removing the outliers (high saliency values). The method, which is illustrated in Figure 6, begins with an initialization step in which the saliency of the pixels studied must be estimated for the first time. A percentage of pixels is then removed based on the deviation from the mean of a distribution of points considered to be normal, corresponding to the most salient points of the reference pixels, before iterating the calculation of the distance with the remaining reference pixels.

$$\begin{array}{cc}\hfill F(X,y,z)& \equiv X(y\le P_z\left(y\right)),\hfill \\ \hfill X^{S{P}_z}& \equiv F(X,Mahal(X,X),z),\hfill \\ \hfill Maha{l}^{S{P}_z}(Y,X)& \equiv Mahal(Y,X^{S{P}_z}),\hfill \end{array}$$

(2)

$\mathrm{where}\left\{\begin{array}{c}{P}_z\left(y\right)\mathrm{is}\mathrm{the}\mathrm{z}-\mathrm{th}\mathrm{percentile}\mathrm{of}y\hfill \\ X\left(\mathit{condition}\right)\mathrm{correspond}\mathrm{to}\mathrm{the}\mathrm{value}\mathrm{of}X\mathrm{for}\mathrm{which}\mathrm{the}\mathrm{condition}\mathrm{is}\mathrm{met}\hfill \\ F\mathrm{is}\mathrm{the}\mathrm{filter}\mathrm{function}\hfill \\ z\mathrm{is}\mathrm{the}\mathrm{percentage}\mathrm{of}\mathrm{value}\mathrm{to}\mathrm{filter}\hfill \end{array}\right.$

The percentage of pixels to remove is a parameter of the method, and is defined by the user beforehand. This parameter can be defined automatically to ensure that the standard deviation or the mean of the saliency of the reference pixels is close to a desired value. Figure 7 shows the effect of the choice of the percentage when using the level factor saliency estimation method; the higher the percentage, the greater the dynamics of the saliency. This increase in dynamics allows better visual observation of surface anomalies on the saliency maps, and helps with the segmentation and classification of surface anomalies.

Level factor saliency filters out salient pixels with a certain percentage. Moreover, it is possible to compress the saliency information from several level factors into a single saliency called the multi-level saliency. Multi-level saliency is computed from the weighted sum of differents factor levels of saliency, as described in the Equation (3). In our case, the weight of a level factor saliency corresponds to its entropy in order to take into account the disparity of the saliency information [35]. The results of multi-level saliency can be seen in Figure 8.

$$Maha{l}^{S{P}_Z}(Y,X)\equiv {\displaystyle \frac{1}{{\sum }_{i=0}^N{w}_i}}\sum _{i=0}^{N}Maha{l}^{S{P}_{Z_i}}(Y,X)\times w_i$$

(3)

$\mathrm{where}\left\{\begin{array}{c}Z\mathrm{is}\mathrm{the}\mathrm{vector}\mathrm{of}\mathrm{factors}\mathrm{level}\hfill \\ w\mathrm{is}\mathrm{the}\mathrm{vector}\mathrm{of}\mathrm{weight}\mathrm{associate}\mathrm{to}Z\hfill \\ N\mathrm{is}\mathrm{the}\mathrm{length}\mathrm{of}Z\mathrm{and}w\hfill \end{array}\right.$

2.3. Multi-Scale Saliency

The estimation of the visual saliency of a point on the surface depends on the points used as reference, and in particular on the possible presence of salient points in the reference considered, as mentioned in the previous section. Another important parameter is the scale of observation, which is even more the case when analysing the appearance of surfaces, where it is well known that scale effects are particularly important in perception. Thus, an anomaly may appear more or less salient when the surface is observed locally or as a whole, and the saliency must therefore integrate these scale factors. For example, many small and weak surface anomalies can be attenuated by the presence of large anomalies in the neighbors or their presence among the reference surface points. Conversely, changing the scale of observation by varying the size of the neighborhood around the calculation point can increase the saliency value of the weakest anomalies and facilitate their detection/evaluation. The calculation of the scale factor saliency makes it possible to solve this problem using a sliding window, as described in Equation (4) and illustrated in Figure 9.

$$\begin{array}{cc}\hfill x_{(t,i,j)}& =x_{(i^{\prime },j^{\prime })},i^{\prime }\in [i-t/2;i+t/2],j^{\prime }\in [j-t/2;j+t/2],\hfill \\ \hfill Maha{l}_{(t,i,j)}^{*}(Y,X)& \equiv Maha{l}^{*}(Y_{(t,i,j)},X_{(t,i,j)}),\hfill \end{array}$$

(4)

where $X_{(r,i,j)}$ corresponds to the sliding window centered on pixel $(i,j)$ of size $t\times t$ pixels and $(i^{\prime },j^{\prime })$ are the coordinates of the pixels in the sliding window.

At each sliding window position, the saliency of the points contained in the window is calculated by taking these same points as a reference. However, in order to take into account that the salience varies with the distance from the considered neighborhood, the calculation is weighted using a 2D Gaussian function, thus allocating more weight in the estimation of the salience to the points close to the calculation point. At each saliency calculation, a weight matrix is filled with the Gaussian function when moving the sliding window. The different parameters of the algorithm are then the size of the window $Si$, the translation step of the window $Tr$, and the standard deviation of the Gaussian function $St$.

The result of the scale factor salience can be seen in Figure 10. It can be seen that the number of small anomalies increases when the size of the sliding window decreases. Moreover, we observe that the area and the level of salience of anomalies detected with a small window seems to decrease. This reduction is due to the proportionality (or not) of salient points inside the sliding window. Indeed, when taking the whole of the surface, the surface anomalies are proportionally less numerous than the other points of the surface. However, by decreasing the observation scale, the surface anomalies become proportionally more numerous when they are inside the observation window.

Similar to multi-level saliency, it is possible to calculate multi-scale saliency from several scale factor saliency measurements, as described in Equation (5) and shown in Figure 11.

$$\begin{array}{cc}\hfill Maha{l}_{(T,i,j)}^{*}(Y,X)& \equiv {\displaystyle \frac{1}{{\sum }_{k=0}^N{w}_k}}\sum _{k=0}^{N}Maha{l}^{*}(Y_{(T_k,i,j)},X_{(T_k,i,j)})\times w_k,\hfill \end{array}$$

(5)

$\mathrm{where}\left\{\begin{array}{c}T\mathrm{is}\mathrm{the}\mathrm{vector}\mathrm{of}\mathrm{factors}\mathrm{scale}\hfill \\ w\mathrm{is}\mathrm{the}\mathrm{vector}\mathrm{of}\mathrm{weight}\mathrm{associate}\mathrm{to}T\hfill \\ N\mathrm{is}\mathrm{the}\mathrm{length}\mathrm{of}T\mathrm{and}w\hfill \end{array}\right.$

2.4. Synthesis of Multi-Level and Multi-Scale Salience

In Section 2.2, we noted that with multi-level saliency the level of salience of the points decreases when similar points are taken as references. However, by decreasing the size of the sliding window when estimating multi-scale salience, in Section 2.3 we increased the proportionality of the observed and reference salient points when inside the sliding window, thus decreasing the level of salience. We then avoided this drop in the level of salience by coupling the multi-scale method (Equation (4)) to the multi-level method (Equation (3)), as described in Equation (6). It can be observed in Figure 12 that small surfaces anomalies are always visible thanks to the multi-scale approach, while the salience dynamics are increased thanks to the multi-level approach. Thus, the coupling of the two approaches makes it possible to reconcile the discrimination of smaller defects while keeping a large dynamic, which, for example, allows them to be classified and synthetic values of salience to be extracted, thereby integrating the aspects of scale.

$$Maha{l}_{(T,i,j)}^{S{P}_Z}(Y,X)\equiv Mahal(Y_{(T,i,j)},X_{(T,i,j)}^{S{P}_Z}),$$

(6)

where $S{P}_Z$ corresponds to the multi-level saliency parameter and $T,i,$ and j are the multi-scale saliency parameters.

3. Global Approach: Inter-Surface Saliency from RTI Data

The Mahalanobis distance is generally used with the RTI to estimate the visual saliency at each point on the surface. However, the Mahalanobis distance compares observations with a reference sample. Therefore, the observations and reference samples can correspond to other data obtained by the RTI method. In Section 3.1, a method is proposed for estimating distances between several surfaces. This method allows temporal monitoring of the appearance of a surface in order to understand and/or prevent an alteration of the surface state or to explore the surface state manufacturing parameters in order to approach a reference appearance. We then show how this approach can be generalized for analysis of the reconstruction quality of Reflectance Transformation Imaging models in terms of appearance descriptors.

3.1. Distance between Different Surface Roughnesses

In this part, we propose to extend the use of the Mahalanobis distance to both the pixels and to the RTI acquisitions themselves. Thus, we no longer compare the appearance attributes of the points of a surface, only the surface states between them. The comparison of surfaces allows detection of changes in the surface state, helps to observe and prevent anomalies, and enables exploration of the surface state manufacturing parameters in order to approach a reference surface appearance. To compare several surfaces with each other, a distance must be estimated between the local descriptors of the reference surface and the surfaces studied by the Mahalanobis method. When using Equation (1), X then corresponds to the local descriptors of the reference surface and Y corresponds to the local descriptors of the surface for which we want to know the distance. If we have several surfaces to compare with a reference, we obtain several salience maps which discriminate the local changes with an intensity correlated to the degree of change with respect to the reference. Thus, as illustrated in Figure 13, several nickel surfaces can be observed. These surface states have an average roughness ($Rz$) and a maximum roughness ($Ra$), provided by the manufacturer, that are different for each sample. Then, it can be seen in Figure 14 that the Mahalanobis distance between the surfaces is correlated with the difference of $Ra$ and $Rz$ between the surface reference and the studied surface. To validate these results, an additional study should be conducted to determine the reflectance measurement error and the appropriate normal estimation method [36,37].

4. Conclusions

A methodology for the evaluation of the visual saliency of surfaces from RTI acquisitions is proposed. This method is based on the implementation of a multi-scale and multi-level approach which integrates the contribution of the local saliency estimated at different scales around the computation point and takes into account the dispersion of the points in the neighborhood for the calculation. The methodology is extended for inter-surface evaluation applications, i.e., for the quantification of the distance in terms of visual properties between different surface roughnesses. The obtained results show that the proposed method can robustly quantify the local appearance properties of the inspected surfaces in the case of local or inter-surface saliency assesment.