According to the dichromatic reflection model proposed by Shafer [
13], the reflected light can be regarded as a combination of diffuse reflection and specular reflection, which can be described mathematically as:
where
is diffuse reflection component and
is specular highlight component;
and
represent weighting factors of diffuse reflection and specular reflection, respectively; and
,
, where
is wavelength,
is the range of wavebands.
is the camera induction function,
H represents spectral reflectance, and
E denotes the spectral power distribution function of light source.
In a conclusion, the specular highlight has different polarization states in different wavebands. Therefore, the diffuse reflection component and highlight component can be distinguished effectively through comprehensive utilization of the polarization information in different wavebands.
3.2.1. Highlight Detection Based on Multiband Polarization
We conduct a statistical analysis to further analyze the differences in spectral and polarization characteristics between specular highlight and diffuse reflection areas intuitively. Here 100 images affected by specular highlights are collected as the data for statistical analysis. Firstly, we manually segment these image data to obtain the two independent areas: for each image data, two small patches that consist of
pixels are extracted from the diffuse reflection area and specular highlight areas, respectively. Thus, we obtain two sets of images, one is specular highlight patches and the other is diffuse reflection patches. Then we compute the DoLP (Degree of Linear Polarization) and grayscale values of each pixel in the R, G, and B bands, and the average values are shown in
Figure 4. It can be clearly seen that the DoLP values of pixels in the diffuse reflection area are quite low while the specular highlight pixels have higher DoLP. Besides, for different wavebands, the distributions of DoLP are different either. Similarly, the grayscale value of the diffuse reflection area is much lower than that of the specular highlight area and the distributions are also different in each waveband. Therefore, the specular highlight and diffuse reflection components can be distinguished accurately by analyzing the differences in intensity, spectral and polarization characteristics, so that effective specular highlight detection can be achieved.
Based on the above analysis, we define the joint multi-band-polarization characteristic vector as follows:
where
is the degree of linear polarization,
and
represents intensity values of three visible bands, respectively. Additionally, it should be noted here that the ratios of
are fixed weight coefficients (0.299 for
r rand, 0.587 for
g rand and 0.114 for
b rand), since we use the function “rgb2gray” in MATLAB to transform the color image into a gray-scale image. The values of
range from 0 to 1, and both
and
are in the same vector
, so
is normalized for consistency. Namely, the possible values of
range from 0 to 1 as well.
The specular highlight area and the diffuse reflection area can be detected by effective constraints based on the joint multi-band-polarization characteristic vector. For the target image under the influence of specular highlight, its pixel
x meets the following constraints:
where
and
represent diffuse reflection area and specular highlight area, respectively,
denotes the global threshold vector,
is a tiny positive constant.
To facilitate subsequent highlight removal, a binary mask image is generated for the detection results of pixel
x, and its expression is:
Based on the joint multi-band-polarization characteristic vector constraint, for the target image shown in
Figure 3, The values of
and
are determined by the following analysis.
Figure 5a is the target image, for the marked area of the target image in (a), the distribution statistical characteristics of Dolp, grayscale and luminance in R, G and B bands are shown in
Figure 5b–e, where the X-axis is the pixel position, and the Y-axis is the values of each parameter. We can easily find the approximate pixel positions of specular area (100–170) from
Figure 5b since specular has much higher grayscale values. So in
Figure 5b, we can obtain the threshold of
is 0.1; in
Figure 5c, the threshold of
is 0.2; in
Figure 5d, the threshold of
is 0.25. For the luminance
, we can see the threshold is about 130 in
Figure 5e, so the final threshold of
is 0.5 after normalization.In a conclusion, we set
. The value of
is obtained by comparing the experimental results, and the best result of specular highlight area detection can be obtained when
. The specular highlight detection result is shown in
Figure 6.
3.2.2. Highlight Removal Based on Multiband Polarization
Compared with the unpolarized diffuse reflection component, the specular highlight component has different polarization states in different wavebands. Therefore, in this section, a Max-Min multi-band-polarization differencing scheme is proposed to generate a single specular-free image (SSF) by utilizing the multi-band polarization characteristic differences between diffuse reflection component and specular highlight component. Meanwhile, for the detected highlight area acquired in the previous section, an ergodic least-squares coefficients decomposition strategy is proposed to obtain the reflection coefficients of diffuse reflection and highlight and thus the effective separation of specular highlight can be achieved.
- a.
Max-Min multi-band-polarization differencing scheme
Combining multi-band-polarization imaging model (7) and dichromatic reflection model (2), we have:
It is assumed that the light source has a uniform energy distribution in the visible bands, so for
three visible wavebands, we have:
For the visible wavebands
, we can define the Max-Min multi-band-polarization differencing image
as:
Combining the three equations above, we have:
It can be derived that the Max-Min multi-band-polarization differencing image is only related to the diffuse reflection component rather than the specular highlight. Therefore, spectral differential images in three polarization angles—0°, 45°, and 90°—are merged with the linear weighting to generate a single specular-free (SSF) image, which can be expressed as:
where
represents rotation angle of the polarizer, and
denotes the adaptive weight coefficient, which is associated with the intensities of polarization images with different angles,
represents the normalization factor, namely the weighting sum.
As shown in
Figure 3, the SSF image can effectively avoid the influence of specular highlight, and meanwhile retaining the original information of the target, which can effectively guide the subsequent removal of the specular highlight.
- b.
Ergodic least-squares separation algorithm
For the pixels in the detected highlight area in
Figure 6, we can find a diffuse pixel with the closest intensity value in the SSF image, and the intensity of the diffuse pixel is taken as the diffuse intensity of the highlighted pixel. Based on this idea, we propose an ergodic least-squares separation algorithm to perform a global ergodic search in the SSF image, to realize effective guidance of reflection coefficients separation in the detected specular highlight area. The process is as follows:
Step 1: Highlight separation of pixels in highlight area.
For pixel
p marked
in specular highlight area, searching for a diffuse reflection pixel
q which has the nearest intensity value to
p in SSF image, so we have:
Take the original intensity value of
q as the diffuse intensity of the highlighted pixel
p, and combining with (1), we have:
Using the least-squares coefficient decomposition method, the reflection coefficient matrix is:
where “
” represents the pseudo-inverse of the matrix. For the diffuse reflection factor
, its corresponding diffuse reflection intensity can be expressed as:
Furthermore, the intensity of corresponding highlight component is:
After separation, pixel p is marked as a diffuse pixel, that is . The algorithm then proceeds to the next highlight pixel.
Step 2: Highlight separation of pixels in diffuse reflection area.
For the diffuse reflection area where , some diffuse reflection pixels may contain a small amount of highlight component. To further realize the separation and removal of specular highlight, we use the same least-squares strategy to obtain the diffuse reflection and specular highlight coefficients.
For the diffuse reflection area where
, searching for a diffuse reflection pixel
r whose intensity value is the highest in the SSF image, namely:
Then seeking a diffuse reflection pixel
z with the nearest intensity value to pixel
r in the SSF image, which satisfies:
Next taking the original intensity value of
r as diffuse reflection intensity of pixel
z, the reflection coefficient matrix can be calculated by using the least-squares coefficient decomposition method:
Since the reflection coefficient of highlight is non-negative, if the calculated
, (24) can be converted to:
Similar to (20) and (21), its corresponding diffuse reflection and specular highlight components can be calculated. Finally, pixel z is marked as a processed pixel, and then the same procedure can be adopted to the next diffuse reflection pixel which has the nearest intensity value to pixel r according to (23).
In summary, our proposed method considers not only the reflection separation of pixels in the highlight area, but also the possibility of the existence of a small amount of highlight in the diffuse reflection area, so that the removal result is much more thorough and accurate.
- c.
The compensation of missing information based on local chromaticity consistency regularization constraint
The proposed highlight removal algorithm in Section
ignores the local detail information of the original target and leads to partial color distortion and edge discontinuity after specular highlight removal. So we use our previous work, local chromaticity consistency [
46], to realize weighting regularized constraint of highlight removal result and meanwhile variable splitting [
47] is combined to achieve fast optimization solving. In this way, the missing information after highlight removal can be effectively compensated in time and the visual effect is further improved. For the highlight suppression result
, the weighting regularization term can be denoted as
. The pixels with a similar grey level neighborhood to
have larger weights in the average, thus like that in nonlocal means methods [
48],
can be set to be inversely proportional to the distance between pixel x and y and can be expressed as:
where
I is the original target image, and
represents standard deviation. Therefore, for two adjacent pixels
x and
y in a local area, when this two pixels belong to diffuse and highlight areas, respectively, significant difference exists between
and
, namely
; while if this two pixels belong to the same reflection area, the weighted value
is bigger.
To simplify, the matrix forms of the weighting regularization term and weighting function (26) can be expressed as
and
, where
D is first-order forward difference operator, ∘ represents matrix product, ⊗ represents convolution and
i is color channel. Therefore, the problem of weighting regularization information compensation can be converted into the following energy function minimization problem:
where
is weighting factor,
is the initial highlight removal result. For the above energy function optimization problem, we can quickly solve it by the variable splitting method. Here we introduce an intermediate variable
u, so (27) can be converted into:
where
is a weighting factor. Obviously, (28) converges to the optimal solution of (27) when
.
For (28), when is fixed, the minimization problem can be treated as alternating optimization solution of u and . That is, first fixing to calculate optimal u; then fixing u to obtain optimal . The process continues until convergence. The detailed optimization process is as follows:
- (1)
Fixing and optimizing u
In (28), for a given
, its minimum energy function is:
It can be converted into:
where are
,
,
all known and can be obtained directly, namely:
here
is the symbol function.
- (2)
Fixing u and optimizing
We regard the calculated
u as a fixed value to obtain the optimal
. The objective function is:
Notice that (35) is a quadratic term function of the variable
, so we can obtain the optimal value of
by taking the partial derivative of (35) and we have:
where
is the transpose matrix of
D. Then we set (36) to be zero to obtain the optimal value of
:
Since convolution calculation is complicated in the time domain, so we use 2D FFT (Fast Fourier Transform) to calculate the optimal value of
in the frequency domain. Thus, we have:
where
represents FFT and
denotes complex conjugate. Finally the optimal value of
, denoted as
, can be obtained by Inverse Fast Fourier Transform (IFFT) transformation:
where
represents IFFT.