Next Article in Journal
UAV Data as an Alternative to Field Sampling to Monitor Vineyards Using Machine Learning Based on UAV/Sentinel-2 Data Fusion
Next Article in Special Issue
ABOships—An Inshore and Offshore Maritime Vessel Detection Dataset with Precise Annotations
Previous Article in Journal
A Novel Deeplabv3+ Network for SAR Imagery Semantic Segmentation Based on the Potential Energy Loss Function of Gibbs Distribution
Previous Article in Special Issue
Towards Semantic SLAM: 3D Position and Velocity Estimation by Fusing Image Semantic Information with Camera Motion Parameters for Traffic Scene Analysis
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Specular Reflection Detection and Inpainting in Transparent Object through MSPLFI

1
School of Engineering and Information Technology, The University of New South Wales (UNSW@ADFA), Canberra, ACT 2610, Australia
2
Department of Computer Science and Engineering, Dhaka University of Engineering & Technology (DUET), Gazipur 1700, Bangladesh
*
Author to whom correspondence should be addressed.
Remote Sens. 2021, 13(3), 455; https://doi.org/10.3390/rs13030455
Submission received: 24 November 2020 / Revised: 20 January 2021 / Accepted: 26 January 2021 / Published: 28 January 2021

Abstract

:
Multispectral polarimetric light field imagery (MSPLFI) contains significant information about a transparent object’s distribution over spectra, the inherent properties of its surface and its directional movement, as well as intensity, which all together can distinguish its specular reflection. Due to multispectral polarimetric signatures being limited to an object’s properties, specular pixel detection of a transparent object is a difficult task because the object lacks its own texture. In this work, we propose a two-fold approach for determining the specular reflection detection (SRD) and the specular reflection inpainting (SRI) in a transparent object. Firstly, we capture and decode 18 different transparent objects with specularity signatures obtained using a light field (LF) camera. In addition to our image acquisition system, we place different multispectral filters from visible bands and polarimetric filters at different orientations to capture images from multisensory cues containing MSPLFI features. Then, we propose a change detection algorithm for detecting specular reflected pixels from different spectra. A Mahalanobis distance is calculated based on the mean and the covariance of both polarized and unpolarized images of an object in this connection. Secondly, an inpainting algorithm that captures pixel movements among sub-aperture images of the LF is proposed. In this regard, a distance matrix for all the four connected neighboring pixels is computed from the common pixel intensities of each color channel of both the polarized and the unpolarized images. The most correlated pixel pattern is selected for the task of inpainting for each sub-aperture image. This process is repeated for all the sub-aperture images to calculate the final SRI task. The experimental results demonstrate that the proposed two-fold approach significantly improves the accuracy of detection and the quality of inpainting. Furthermore, the proposed approach also improves the SRD metrics (with mean F1-score, G-mean, and accuracy as 0.643, 0.656, and 0.981, respectively) and SRI metrics (with mean structural similarity index (SSIM), peak signal-to-noise ratio (PSNR), mean squared error (IMMSE), and mean absolute deviation (MAD) as 0.966, 0.735, 0.073, and 0.226, respectively) for all the sub-apertures of the 18 transparent objects in MSPLFI dataset as compared with those obtained from the methods in the literature considered in this paper. Future work will exploit the integration of machine learning for better SRD accuracy and SRI quality.

Graphical Abstract

1. Introduction

The emerging significance of specular reflection detection and inpainting (SRDI) has been actively pursued in the computer vision community over the last few decades. The presence of specular reflection creates potential difficulties for tasks such as detection, segmentation, and matching, as it captures significant information about an object’s distribution, shape, texture, and roughness features that cause discontinuity in its omnipresent, object-determined diffuse part [1]. Once specular reflection is detected, it may be used to synthesize a scene [2] or to estimate lighting direction and surface roughness [3,4]. While passing through the surface of a transparent object, some incoming lights are immediately reflected back into the space and are called surface or specular reflections, and others penetrate the surface and then reflect back into the air body or diffuse reflections [5]. Due to a transparent object lacking its own texture, it is always a difficult and challenging task to detect its specular reflections and inpainting [6]. The potential application of specular reflection detection and inpainting in transparent objects through multispectral polarimetric light field imagery (MSPLFI) includes 3D shape reconstruction, detection and segmentation, surface normal generation, and defect analysis.
By integrating advanced communication tools and techniques, multispectral polarimetric imagery (MSPI) can extract an object’s meaningful information, such as surface features, shapes, and roughness, in optical sensing images [7]. Potential applications of it could investigate acquiring an imaging system that performs image denoising [8], image dehazing [9], and semantic segmentation [10]. Multispectral imaging is a mode commonly reported in the literature for enhancing color reproduction [11], illuminant estimation [12], vegetation phenology [13,14], shadow detection [15], and background segmentation [16,17]. Additionally, although a multispectral cue is capable of generating information through penetrating deeper into an object, it is sometimes infeasible for extracting the object’s inherent features. Together with a polarimetric cue, where specific photoreceptors are used for polarized light vision, MSPI is applied in applications such as specular and diffuse separation [18], material classification [19], shape estimation [20], target detection [21,22,23], anomaly detection [24], man-made object separation [25], and camouflaged object separation [26]. Recently, a light field (LF) cue has gained popularity in the graphics community for detecting and segmenting some complex tasks, such as transparent object recognition [27], classification [28], and segmentation [29] from a background, by analyzing the distortion features of a single shot captured by an LF camera. Each pixel in an LF image is capable of having six degrees of freedom to extract the hidden information unable to be captured by MSPI cues. The aim of the proposed research is to use the multisensory cues of MSPLFI, which can effectively detect the specular reflection and the corresponding suppression in a transparent object.
Firstly, it is necessary to separate specular reflection from diffuse reflection. Each pixel in MSPLFI can be defined as the sum of specular and diffuse reflections following the dichromatic reflection model [30] as
L ( λ ,   ρ ,   ,   θ i ,   θ r ,   g ) =   L S p e c ( λ ,   ρ ,   ,   θ i ,   θ r ,   g )   +   L D i f f ( λ ,   ρ ,   ,   θ i ,   θ r ,   g ) ,
where L s ( λ ,   ρ ,   ,   θ i ,   θ r ,   g ) is the specular reflection, L s ( λ ,   ρ ,   ,   θ i ,   θ r ,   g ) the diffuse reflection, λ the wavelength in the multispectral visible band (400 nm–700 nm), ρ the orientation of the polarimetric filter (rotating at 0°, 45°, 90°, 135°), the LF direction in which the light rays are traveling in space, and θ i ,   θ r ,   g the geometric parameters indicating incidence, viewing, and phase angles, respectively.
The individual components in Equation (1) can be further decomposed into two parts, composition and magnitude, as in Equation (2). Composition is a relative spectral power distribution ( c S p e c (surface reflection) or c D i f f (body reflection)) that depends on only wavelength, polarization, and LF but is independent of geometry. Magnitude is a geometric scale factor ( ω S p e c or ω D i f f ) which depends on only geometry and is independent of the wavelength, polarization, and LF.
L ( λ ,   ρ ,   ,   θ i ,   θ r ,   g ) =   ω S p e c ( θ i ,   θ r ,   g ) c S p e c ( λ ,   ρ ,   )   +   ω D i f f ( θ i ,   θ r ,   g ) c D i f f ( λ ,   ρ ,   ) ,
As the appearance of a transparent object is highly biased by its background’s texture and color, it is a challenging task to detect, segment, and suppress the specular reflections on it. Through predicting multispectral changes per sub-aperture image in the LF, the proposed research detects specular reflected pixels. In terms of inpainting, as it can be predicted that a pixel in a LF image has six degrees of freedom and can appear within any surrounding four-connected pixels in a sub-aperture image, a pixel pattern with maximum acceptability is selected to suppress an SRD pixel. Briefly, the proposed system firstly describes the significance of the joint utilization of multisensory cues, then captures an MSPLFI object dataset, proposes a two-fold algorithm for detecting and suppressing specular reflections, evaluates both detection accuracy and suppression quality in terms of statistical distinct metrics and, finally, compares performance with those of some other methods in the existing literature.
The main contribution of this research is two-fold. Firstly, an SRD algorithm that predicts changes in MSPLFI by calculating mean ( μ ) and covariance ( Σ ) of each sub-aperture index of the LF to predict specular reflections through applying the Mahalanobis distance is proposed. Then, the predicted changes in unpolarized and polarized images are averaged, and a threshold is applied to obtain a final SRD pixel mask (SRD-PM). However, due to the absence of publicly available multisensory 6D datasets to evaluate the performance of the proposed research, we firstly built an image acquisition system to capture an MSPLFI object dataset. Secondly, an SRI algorithm which extends the final SRD-PM in an immediately neighboring pixel using the RGB channels of both polarized and unpolarized sub-apertures in the LF is proposed. For a pixel in the SRD-PM, all the four-connected neighboring pixel patterns per sub-apertures of the LF, excluding those already in the SRD-PM, are carefully selected and a distance matrix is computed based on their intensities. Finally, the pixel pattern with the minimum distance is chosen for the task of inpainting. The performances of these approaches are evaluated and compared using a private MSPLFI object dataset to demonstrate the significance of this research.
This paper is organized as follows. In Section 2, the background to SRD and SRI is fully described. In Section 3, the details of the private MSPLFI dataset, including image acquisition setup, multisensory cues, and pixels’ degrees of freedom, are analyzed. In Section 4, a complete two-fold SRDI framework and corresponding algorithms are presented with proper mathematical and logical explanations. In Section 5, the performances of the proposed SRD and SRI algorithms are evaluated by distinct statistical metrics. Additionally, detection accuracy and suppression quality of the proposed SRDI are visualized and compared with those of existing approaches. Finally, concluding remarks and suggested future directions are provided in Section 6.

2. Related Works

SRD techniques usually assume that the intensities of specular pixels vary from those of diffuse ones in multiple spectra as
P ( x ,   y ,   c ,   λ ,   ρ |   i ) =   { 1 i f   d ( I ( x ,   y ,   c ,   λ ,   ρ |   i ) ,   S ( x ,   y ,   c ,   λ ,   ρ |   i ) ) > τ G 0                  o t h e r w i s e ,
where τ G is a global threshold, P ( x ,   y ,   c ,   λ ,   ρ |   i ) the final SRD-PM at pixel ( x , y ) of a fused spectrum ( λ ) at a polarimetric orientation ( ρ ) in sub-aperture index i of the LF (), d the distance between the pixel of the predicted specular pixel ( S ) and that of the fused image in spectrum λ ( I ) at orientation ρ . In this section, a brief review of the literature related to SRDI techniques for multisensory cues of MSPLFI is provided.

2.1. Specular Reflection Detection (SRD)

Recent works on SRD are categorized in two major ways, single and multiple image-based, where the latter depends on specific conditions such as lighting direction and viewpoint. Based on a single-textured color image, Tan [31] iteratively shifts the maximum chromaticity of each pixel between two neighboring ones. An iteration stops when the chromaticity difference satisfies a certain threshold value and generates a specular-free (SF) image. The final SF image ensures a similar geometrical distribution even though it contains only diffuse reflections. However, for a large image with more specularity, this techique may lead to erroneous diffuse reflections with excessive and inaccurate removal as well as higher computational complexity. Subtracting the minimum color channel value from each channel, Yoon [32] obtains an SF two-band image. Capturing images from a dynamic light source, Sato [33] integrates the dichromatic reflection model for separation by analyzing color signatures in many images captured by a moving light source. A series of linear basis functions are introduced by Lin [34], and the lighting direction is changed to decompose the reflection components.
The modified SF (MSF) technique introduced by Shen [35] ensures robustness to the influence of noise on chromaticity. It subtracts the minimum RGB value from an input image and works in an iterative manner by selecting a predefiend offset value using the least-squares criterion. Nguyen [36] proposes an MSF method that integrates tensor voting to obtain the dominant color and distribution of diffuse reflections in a region. To improve the separation performance, Yamamoto [37] applies a high-emphasis filter on individual reflection components to separate them [35]. However, all these methods suffer from artifacts and inaccuracy if the brightness of the input image is high.
Recent literature on SRD reveals that the specular reflection of an object’s area has a stronger polarization signature than its diffuse reflection. Placing a polarization filter in front of an imaging sensor, Nayar [18] proposes separating the specular reflection components from an object’s surface with heavy textures. Considering the textures and the surface colors of neighboring pixels, many authors [31,38,39] could separate specular reflections through neighboring pixel patterns. Applying a bilateral filter with coefficients, Yang [39] proposes an extension of Tan’s [31] method in which the diffuse chromaticity is maximized. Although it provides faster separation and better accuracy, it still suffers from some problems for separating specular reflections in a transparent object. Akashi [40] also employs the dichromatic reflection model to separate specular reflections in single images based on sparse non-negative matrix factorization (NMF) composed of only non-negative values regulated by parameters such as sparse regularization, pixel color, and convergence. Although this method demonstrates better separation accuracy than those of Tan [31] and Yang [39], inaccurate parameter settings may lead to artifacts in the separation of specular reflections.
An SUV color space for separating specular and diffuse reflections from S and UV channels, respectively, of a single image or image sequence in an iterative manner is proposed by Mallick [38]. However, discontinuities in the surface color may lead to erroneous detection of secular reflections. In [41], Arnold applies image segmentation based on non-linear filtering and thresholding to separate specular and diffuse reflections in medical imaging. Saint [42] proposes increasing the gap between two reflection components and then applying a non-linear filter to isolate spike components in an image histogram. In [43], Meslouhi integrates the dichromatic reflection model to detect specular reflections. In our research, we use multisensory cues to detect specular reflections by predicting changes among multiband data.

2.2. Specular Reflection Inpainting (SRI)

SRI refers to restoring an SRD pixel pattern with semantically and visually believable content through analyzing neighboring pixel patterns. Recent works in the literature on SRI depend mainly on patch-based similarity, with similar patch- or diffusion-based inpainting proposed to fill an SRD pixel pattern by spreading color intensities from its background to its holes [8,9,44,45]. Traditional inpainting approaches apply an interpolation technique on the surrounding pixels to restore an SRD pixel pattern [46,47]. Based on temporal information in an endoscopic video image sequence, Vogt [48] proposes a well-inpainting method. Cao [49] develops an inpainting technique for averaging the pixels in a sliding rectangular window and later replacing it with an SRD pixel. Although this method is simple and relatively fast to compute, it lacks robustness due to varying window sizes based on the SRD’s connected pixels. In [50], an average intensity of a contour is calculated to replace the SRD pixels by author Oh but may lead to strong gradients.
In [41], Arnold proposes a two-level inpainting technique which replaces SRD pixels with the centroid color within a certain distance and applies a Gaussian kernel for smoothing using a binary weight mask. Although the inpainting quality is better than those of other methods, it may produce some artifacts and blur for large spectral areas by integrating a partial differential equation with gradient thresholding. In [51], Yang proposes a convex model for suppressing the reflection from a single input image. In [52], Criminisi describes an image inpainting method in which an affected region is filled by some exemplars. As these techniques may produce artifacts and fail to suppress large reflection areas, our proposed method reconstructs the specular reflected pixels through analyzing their four-connected neighbors in the sub-apertures of the 4D-LF.

3. Analysis of MSPLFI Transparent Object Dataset

Regarding SRD and SRI, the proposed research uses multisensory cues through capturing different objects in MSPLFI, each of which is defined as a function of 6D as
L 6 D = L ( u ,   v ,   s ,   t ,   λ , ρ ) ,
where ( u ,   v ) is the image plane referring to an image’s spatial dimensions, ( s ,   t ) the viewpoint plane referring to the direction in which the light rays are traveling in space, λ the wavelength in the multispectral visible band (400 nm–700 nm), and ρ the orientation of the polarimetric filter (rotating at 0°, 45°, 90°, 135°).
In this section, acquisition of the MSPLFI object dataset and then its use for detecting and suppressing specular reflections in a transparent object are described.

3.1. Experimental Setup

As there is no dataset available for the evaluation of SRDI in a transparent object that integrates multiple cues of MSPLFI, Figure 1 illustrates our setup for image acquisition to generate a problem-specific object dataset in a constrained environment with a plenoptic camera, Lytro Illum, used to capture all the LF images. We place different band filters in front of the camera to capture multispectral images and a linear polarization filter rotating at 0°, 45°, 90°, and 135° to manually obtain different polarimetric images with two light sources used to obtain accurate spectral reflections. The lighting is similar for different objects, and we retain the same background for them, which completely matches most of the objects in most of the area with the purpose of creating a complex environment from which to segment a whole object. One of the light sources is located beside the camera lens at 45° angle and another is located on the top object’s location. The energy levels of multiple spectra are not similar; however, individual cues contain a useable amount of information when capturing MSPLFI.

3.2. MSPLFI Transparent Object Dataset

In Figure 2, the median specular reflections of the sub-aperture images of 18 transparent objects (O#1–O#18) captured through MSPLFI are presented with their corresponding labels. To evaluate the performance of the image inpainting technique, some balls are placed inside object O#1.
We consider five different shots for each spectrum of each object. Of them, one corresponds to the unpolarized version of the image captured without using a polarization filter and the other four to four different polarization filter orientations (0°, 45°, 90°, and 135°) using a linear polarizer. We consider multiple spectra in the visible range (400 nm–700 nm) to obtain images in the multispectral environment. Figure 3 shows the center sub-aperture images of object O#8 in multiple color bands of violet, blue, green, yellow, orange, red, pink, and RGB in polarized and unpolarized versions. As can be seen, due to the nature of polarization, on average, 50% of the photons get blocked while passing through a lossless polarizer at different orientations.
The LF images are 4D data obtained from different viewpoints, with each image presented as a sub-aperture plane ( s ,   t ) with its tangent direction ( u ,   v ). In our experiments, we consider 11 × 11 sub-aperture images, including their center viewpoints, with their spatial representations denoted by ( u ,   v ). Figure 4 shows the 4D-LF images of object O#8 in the violet color band, with the center viewpoint image at the cross-section of the S and the T lines denoted as the (6,6) position in the hyperplane ( s ,   t ,   u ,   v ).

3.3. Degrees of Freedom

Figure 5 presents an example of object O#1’s scene flow among its sub-aperture images and their relative directions. In Figure 5a, the arrow indicates that all the viewpoint images’ motion flows to the center viewpoint image and, in Figure 5b, each pixel has six degrees of freedom in the LF images, with the region of interest (ROI) regarding the scene flow indicated by a yellow rectangle. In Figure 5c, the pixel displacements are shown with their corresponding intensity flow plots, which confirm that the intensity of the ROI varies in different viewpoints.

4. Proposed Two-fold SRDI Framework

In this section, the proposed two-fold SRDI framework based on the distinctive features of MSPLFI cues is discussed and presented in Figure 6. Firstly, a 6D dataset of different transparent objects is captured, and then Reed-Xiaoli (RX) detector [53] is applied to obtain the actual specular reflection of an object through predicting changes among multiband. Secondly, a pixel neighborhood-based inpainting method for suppressing this reflection is proposed.

4.1. Specular Reflection Detection (SRD)

The proposed system detects specular reflected pixels in transparent objects through predictions of multiband changes. Firstly, a raw lenslet (.LFR) image is decoded into a 4D ( s ,   t ,   u ,   v ) LF one, where ( s ,   t ) denotes the image’s position in the hyperplane and ( u ,   v ) its spatial region. The MSPLF imagery was captured by the Lytro Illum camera, which can capture 15 × 15 sub-apertures per shot. However, due to the main lens of the camera being circular, vignetting occurs at its edge. Hence, only the inner 11 × 11 sub-apertures are retained. It could be argued that few more sub-apertures at the top, the bottom, the left, and the right could be as good—if not better—than the corner sub-apertures kept in the 11 × 11 array, but excluding them keeps them in a square array for simplicity. As our main purpose is to detect and suppress specularity in a transparent object, we maximize an object’s area with a minimum surrounding background. In order to compute the specular reflections in unpolarized images, we convert all the multiband unpolarized 4D LF ones into their corresponding grayscale ones. For each sub-aperture index, we store the individual band images in a column vector, with their mean ( μ ) and covariance ( Σ ) calculated for the Mahalanobis distance as
( x μ ) T   Σ 1   ( x μ ) ,
The 2D distance matrix represents the changes among the multiband images per sub-aperture index, which is also observed as specular reflection. We also predict the maximum specularity in unpolarized 4D images. In order to draw specular reflections in polarized images, we firstly calculate the Stokes parameters ( S 0 S 2 ) [54], which describe the linear polarization characteristics using a three-element vector ( S ), as shown in Equation (6), where S 0 represents the total intensity of light, S 1 the difference between the horizontal and vertical polarizations, and S 2 the difference between the linear +45° and –45° ones. The I 0 0 , I 45 0 , I 90 0 , and I 135 0 are the different input images for the system at polarized angles of 0 0 ,   45 0 ,   90 0 , and 135 0 , respectively.
S = [   S 0 S 1 S 2   ] =   [ I 0 0 +   I 90 0 I 0 0   I 90 0 I 45 0   I 135 0 ] ,
The degree of linear polarization ( D o L P ) is a measure of the proportion of the linear polarized light relative to the light’s total intensity, and the angle of linear polarization ( A o L P ) is the orientation of the major axis of the polarization ellipse, which represents the polarizing angle where the intensity should be the strongest. They are derived from the Stokes vector according to Equations (7) and (8), respectively. To calculate the linear polarized image, firstly, the polarimetric components are concatenated, as shown in Equation (9). Then, a concatenated image is generated in the hue, saturation, value (HSV) color space and converted to the RGB color space, as in Equation (10), where L P stands for linear polarization.
D o L P =   I p o l I t o t =   S 1 2 +   S 2 2 S 0 ,
A o L P =   1 2   t a n 1 ( S 2 S 1 ) ,
h s v = ( ( A o L P +   π / 2 ) / π )     ( D o L P × 2 )     S 0 ,
L P = R G B   ( h s v ) ,
For each sub-aperture index of D o L P and L P , we store individual band images in a separate column vector. Then, a similar procedure (unpolarized specular detection) is followed to calculate the maximum specularity in the LP and the DoLP 4D imagery. The average of three specularities ( R X N P ,   R X L P ,   R X D o L P ) shows the overall predicted specularity in an object of MSPLFI, with a threshold (Otsu’s method and in the range (0–1)) applied to obtain the SRD pixels in binary form. The complete process for detecting specular pixels in a transparent object is described in Algorithm 1.
Algorithm 1. SRD in Transparent Object
Input: MSPLFI Object Dataset
Output: SRD Pixel in Binary
1: for all lenslet (.LFR) image do
2:  Decode raw lenslet (.LFR) multiband polarized and unpolarized images into 4D ( s ,   t ,   u ,   v ) LF images
3:  Remove and clip unwanted images and pixels
4: end for
5: for all sub-aperture image do
6:  for all multiband do
7:    Calculate D o L P ,   L P as in Equations (7)–(10)
8:    if type ( L ( u ,   v ,   s ,   t ,   λ , ρ ) = “unpolarized” then
9:     Convert multiband image into corresponding grayscale
      Store multiband grayscale image as column vector
10:    else if type ( L ( u ,   v ,   s ,   t ,   λ , ρ ) = “polarized” then
11:      Store multiband image as column vector
12:    end if
13:  end for
14:  Calculate mean ( μ ) and covariance ( Σ ) per sub-aperture index of LF
15:  Calculate Mahalanobis distance as in Equation (5)
16:  Reshape distance vector as 2D image which represents SRD per sub-aperture image
17: end for
18: Calculate maximum changes/specularities observed in all sub-aperture indexes for object type “ R X N P
19: repeat steps 5–18 for object type = “ R X D o L P ” and object type= “ R X L P
20: Calculate mean ( μ ) specularity of object type: R X N P , R X D o L P and R X L P
21: Apply threshold ( τ ) to binarize SRD pixels

4.2. Specular Reflection Inpainting (SRI)

In this research, the SRD pixels are suppressed through analyzing the distances among four connected neighboring pixels. Firstly, four different regions in an image are identified, as shown in Figure 7. Algorithm 1 predicts region A as an SRD pixel but, for better inpainting quality, both regions A and B are considered specular reflected pixels. It is to be noted that region B contains the pixel patterns (color channels) that are the immediate neighbors of region A. Then, all the connected regions are identified and labeled for the task of inpainting. The complete process for inpainting the detected specular pixels in transparent object is described in Algorithm 2.
Algorithm 2. SRI in Transparent Object
Input: MSPLFI Object Dataset, SRD-PM
Output: SRD Pixel Inpainting in RGB
1: Strengthen SRD-PM (output from Algorithm 1) by labeling all neighboring pixels as SRD ones
2: Compute connected components and label them
3: Calculate baseline image per sub-aperture index by taking minimum pixel intensities of both polarized and unpolarized images in RGB channels
4: for all common sub-aperture images do
5:  for all labels do
6:    for all pixel patterns ( P ( x , y , c   |   i ) ) in SRD-PM do
7:      if labels (SRD-PMs) exist then
8:        Compute distances ( d ( j , k   |   x ,   y   ) ) among 4-connected neighbors not in SRD-PM in each channel, as in Equation (11), and store them in 2D-matrix ( d M ( n r o w , n c o l ) ), as in Equation (12)
9:        Winning pixel pattern is index ( I D X ) of pixel pattern corresponding to column-wise minimum sum of d M ( n r o w , n c o l ) , as in Equations (13) and (14) for inpainting of specular reflections
10:      end if
11:    end for
12:  end for
13: end for
14: repeat steps 4 to 13 to calculate maximum specular reflection in suppressed image of transparent object from already suppressed sub-apertures
A baseline image per sub-aperture index is computed by taking the minimum pixel intensities in both polarized and unpolarized RGB channels. The aim is to suppress the specular reflected areas in the image, with the distance between two pixel-patterns calculated by
d ( j , k   |   x , y ) =   c = R , G , B ( P ( x ,   y ,   c ,   j |   i )   P ( x ,   y ,   c ,   k |   i ) ) 2 ,
where P ( x ,   y ,   c ,   j   |   i ) and P ( x ,   y ,   c ,   k   |   i ) are the two four-connected neighbors of the pixel pattern ( P ( x ,   y ,   c   |   i ) ) in sub-aperture index i and d ( j , k   |   x , y ) the distance between the two pixel patterns corresponding to P ( x ,   y ,   c   |   i ) in sub-aperture index i . A 2D matrix [55] of the distances among the pixel patterns is calculated by Equation (12). The pattern corresponding to the lowest column-wise sum of the distances is selected as the winning one ( P ( x ,   y ,   c ,   I D X |   i ) ) for the task of SRI in Equations (13) and (14).
d M ( n r o w , n c o l ) = ( d ( j 4 , k 4   |   x , y ) d ( j + 4 , k 4   |   x , y ) d ( j , k   |   x , y ) d ( j 4 , k + 4   |   x , y ) d ( j + 4 , k + 4   |   x , y ) )
I D X = a r g m i n k   d M ( n r o w ,   k )
P ( x ,   y ,   c |   i )   =   P ( x ,   y ,   c ,   I D X |   i )

5. Experimental Results

In this section, performance evaluations and comparisons of the proposed two-fold SRDI and other approaches using different metrics for specular pixel detection and inpainting are discussed. Additionally, analyses of their computational times are conducted.

5.1. Selection of Performance Evaluation Metric

Both SRD and SRI are evaluated by commonly used statistical evaluation metrics for quantifying detection accuracy and inpainting quality.

5.1.1. Selection of SRD Metric

The SRD method is evaluated at the pixel level of a binarized scene in which the pixels related to the specular and the diffuse reflections are white and black, respectively. Its performance can be divided into four pixel-wise classification results: true positive ( T p ), which means a correctly detected diffuse pixel; false positive ( F p ), that is, a specular reflected pixel incorrectly detected as a diffuse reflected one; true negative ( T n ), which indicates a correctly detected pixel with specularity; and false negative ( F n ), that is, a diffuse reflected pixel incorrectly detected as a specular reflected one. The binary classification metrics used are precision, recall or sensitivity, F1-score, specificity, geometric-mean (G-mean), and accuracy. Precision is the number of diffuse reflected pixels detected that are actually diffuse reflected ones, while recall is the number of diffuse reflected pixels detected from the actual diffuse reflected ones (recall and sensitivity are similar). The F1-score (a boundary F1 measure) is the harmonic mean of precision and recall values, which measures how closely the predicted boundary of an object matches its ground-truth and is an overall indicator of the performance of binary segmentation. Specificity (a T n fraction) is the proportion of actual negatives predicted as negatives, sensitivity (a T p fraction) the proportion of actual positives predicted as positives, G-mean the root of the product of specificity and sensitivity, and accuracy the proportion of true results obtained, either T n or T p . The mathematical evaluation measures of the aforementioned metrics are shown in Equations (15) to (20) [17,56].
P r e c i s i o n   ( P R ) = T p T p + F p ,
R e c a l l   ( R C )   o r   S e n s i t i v i t y   ( S N ) = T p T p + F n ,
F 1 S c o r e   ( F 1 S ) = 2 × P r e c i s i o n × R e c a l l P r e c i s i o n + R e c a l l ,
S p e c i f i c i t y   ( S P ) = T n T n + F p ,
G e o m e t r i c M e a n   ( G M ) = S p e c i f i c i t y × S e n s i t i v i t y ,
A c c u r a c y   ( A C ) = T p + T n T p + F n + T n + F p ,

5.1.2. Selection of Inpainting Quality Metric

Currently, the quality of a fused image can be quantitively evaluated using the metrics [57] structural similarity index (SSIM), peak signal-to-noise ratio (PSNR), mean squared error (IMMSE), and mean absolute deviation (MAD). The SSIM is an assessment index of the image quality based on computations of luminance, contrast, and structural components of the reference and the reconstructed images, with the overall index a multiplicative combination of these three components. The PSNR block computes the PSNR between the reference and the suppressed images in decibels (dB), with higher values of SSIM and PSNR indicating better quality of the reconstructed or the suppressed image. The IMMSE computes the average squared error between the reference and the reconstructed images, while MAD indicates the sum of the absolute differences between the pixel values of these images divided by the total number of pixels, which is used to measure the standard error of the reconstructed image. Lower values of IMMSE and MAD indicate better quality of the reconstructed image. Considering two images ( x and y ), the aforementioned mathematical evaluation metrics are shown in Equations (21) to (24).
S S I M ( x , y ) = [ l ( x , y ) α ] · [ c ( x , y ) β ] · [ s ( x , y ) γ ] ,
where,
l ( x , y ) = 2 μ x μ y +   C 1 μ x 2 + μ y 2 + C 1 c ( x , y ) =   2 σ x σ y +   C 2 σ x 2 +   σ y 2 +   C 2 s ( x , y ) = σ x y +   C 3 σ x σ y + C 3
where μ x , μ y , σ x , σ y and σ x y are local means, standard deviations, and cross-covariances of images x and y .
P S N R ( x , y ) = 10 . l o g 10   ( M A X I 2 I M M S E ( x , y ) ) ,
where M A X denotes the range of the image ( x   o r   y ) datatype
I M M S E ( x , y ) = 1 n i = 1 n ( x i y i ) 2 ,
M A D   ( x , y ) = 1 n   i = 1 n | ( x i   y i ) | ,

5.2. Generation of Ground Truth

To evaluate the performance of the proposed two-fold SRDI, we generate two different ground truths for each object, as shown in Figure 8. The SRD and the SRI ones are created manually by an expert, with the maximum possible specular reflected area in the MSPLFI object dataset covered. Figure 8 shows the two-way SRD ground truth generation, where a pixel with an intensity above a threshold (Otsu’s method and in the range (0–1)) level is considered a specular reflected pixel. The final column in Figure 13 presents the objects’ SRD binary ground truths, with black and white pixels indicating their diffuse and specular reflected pixels, respectively. The final column in Figure 18 shows the objects’ SRI ground truths. Due to the real scene in the MSPLFI object dataset, some pixels in an object may exhibit amounts of both specular and diffuse reflections but, to measure the performance in terms of quantity and enable further comparisons, each pixel is classified manually as either specular or diffuse reflected, and the ground truth is re-named as the quasi-ground truth.

5.3. Performance Evaluation of SRD

5.3.1. Analysis of SRD Rate

Figure 9 shows the SRD rates in terms of the SRD metrics of precision, recall, F1-score, G-mean, and accuracy for nine sample objects both separately (Figure 9) and together for all objects (O#1–O#18) (Figure 10) using the proposed method. For each object, a total of 121 sub-aperture images are used to measure its specularity and box plots to statistically analyze our experiments. Figure 9 exhibits the SRD metric values obtained for nine sample objects separately. Remaining objects are presented in Appendix A (Figure A1). Accuracy has a higher median value than the F1-score and the G-mean for all the objects, with O#9 and O#3 having superior median values of 0.804, 0.832, and 0.996, and 0.874, 0.882, and 0.991 for F1-score, G-mean, and accuracy, respectively, compared with those of the other objects.
Similarly, Figure 10 shows the combined SRD rates for 121 sub-aperture + 1 maximum images × 18 objects = 2196 images. Accuracy has a better overall median and 75th percentile values for all the objects combined (0.981 and 0.992, respectively) compared to the F1-score (0.643 and 0.770, respectively) and the G-mean (0.656 and 0.752, respectively).

5.3.2. Comparison of SRD Rates of Proposed Method and Those in Literature

It is worth mentioning that the performances of the existing SRD methods considered are not exactly comparable, as each reports its accuracy for a specific image set using different contexts. Moreover, the accuracy values obtained from them and the color-mapping techniques used for segmentation may vary.
In Table 1, the performances of SRD in terms of different evaluation metrics for the proposed and other methods are compared for the 18 individual objects. For visualization purposes, short forms of the authors’ names are written in the first column, that is, Ak., Sn., Yn., Ym., Ar., St., and Ms. refer to Akashi, Shen, Yang, Yamamoto, Arnold, Saint, and Meslouhi, respectively. The SRD metric values in the object index columns correspond to the maiden specular image among the sub-aperture ones. The final column (overall mean ?SA)) corresponds to the mean ± SD values of the 121 sub-aperture + 1 maximum images × 18 objects = 2196 images together.
As can be seen, the overall mean SRD different metric values are higher for the proposed method than the studies discussed in this paper, as shown in the final column in Table 1. Additionally, considering all the sub-aperture images of the 18 distinct objects, mean F1-score, G-mean, and accuracy values for the proposed method are 0.546 ± 0.13, 0.654 ± 0.11 and 0.974 ± 0.01, respectively. In Figure 11, the SRD metric values for the 18 individual objects (O#1–O#18) and their maximum specular reflections obtained from different methods are compared. As can be seen, the proposed method achieves superior median values for the F1-score, G-mean and accuracy of 0.662, 0.816 and 0.971, respectively.
In Figure 12, the SRD metric values for 121 sub-aperture + 1 maximum images × 18 objects = 2196 images with their specular reflections obtained by different methods are presented. As can be seen, the proposed method has superior median values for F1-score, G-mean, and accuracy of 0.643, 0.676, and 0.981, respectively, to those of the others.

5.3.3. Visualization of SRD Rates of Different Methods

In Figure 13, the SRD accuracies obtained by different methods for the maximum specular reflected images of sample objects in the MSPLFI object dataset are presented. As can be seen, the proposed approach reports fewer SRD errors than the others. Remaining objects are presented in Appendix A (Figure A2).

5.4. Performance Evaluation of SRI

5.4.1. Analysis of SRI Quality

The SRI qualities in terms of the normalized SRI metrics SSIM, PSNR, IMMSE, and MAD for the nine sample objects using the proposed method are presented separately in Figure 14 and then together for all objects (O#1–O#18) in Figure 15. For each object, a total of 121 sub-aperture + 1 maximum images are considered to measure its SRI and box plots used to statistically analyze our experiments. It is to be noted that a suppressed image with high SSIM and PSNR values and low IMMSE and MAD ones is close to the quasi-ground truth. Figure 14 shows that the SSIM has a higher median value than the PSNR but the IMMSE a lower one than the MAD for all the objects while object O#1 has superior median values of 0.966, 0.820, 0.038, and 0.131 for SSIM, PSNR, IMMSE, and MAD, respectively, to those of the other objects. Remaining objects are presented in Appendix B (Figure A3). Similarly, Figure 15 shows the normalized SRI qualities of (121 Sub-aperture + 1 maximum) × 18 Objects = 2196 images together. The SSIM has better overall median and 75th percentile values for all the objects combined (0.966 and 0.980, respectively) than the PSNR (0.735 and 0.778, respectively) and the IMMSE better overall median and 75th percentile values for all the objects (0.073 and 0.118, respectively) than the MAD (0.226 and 0.273, respectively).

5.4.2. Comparison of SRI Rates of Proposed Method and Those in Literature

It is worth mentioning that the performances of the existing SRI methods are not exactly comparable, as each reports its accuracy for a specific image set in a different context. Additionally, the quality obtained by the methods and the color-mapping techniques used for inpainting may vary.
In Table 2, the performances of SRI in the proposed and other methods for the 18 individual objects are compared using different evaluation metrics. For visualization, short forms of the authors’ names written in the first column as Ar., Yg., Cr., St., Ak., Sn., and Ym. refer to Arnold, Yang, Criminisi, Saint, Akashi, Shen, and Yamamoto, respectively. The SRI metric values in the object index columns correspond to the maiden image of the 121 sub-aperture specular reflected suppressed ones. The final column (overall mean (SA)) corresponds to the mean ± SD values of the 121 sub-aperture + 1 maximum images × 18 objects = 2196 images together. As can be seen, the SRI metric values are significantly better for the proposed method than for the others considered, as shown in the final column in Table 2. For all the sub-aperture images of the 18 distinct objects, the mean SSIM, PSNR, IMMSE, and MAD values obtained from the proposed method are 0.956 ± 0.02, 24.51 ± 2.11, 257.6 ± 119, and 8.427 ± 2.51, respectively.
In Figure 16, comparisons of the SRI metric values of individual methods in terms of SSIM, PSNR, IMMSE, and MAD of 18 individual objects (O#1–O#18) with their maiden specular inpainting is presented. It can be seen that the proposed method has superior median values for SSIM and PSNR of 0.985 and 0.754 and the lowest median values for IMMSE and MAD of 0.063 and 0.217, respectively.
Figure 17 shows the SRI metric values of individual methods in terms of SSIM, PSNR, IMMSE, and MAD of 121 sub-aperture + 1 maiden images × 18 objects = 2196 images. As can be seen, the proposed method has superior median values for SSIM and PSNR of 0.966 and 0.735, respectively, and the lowest median values for IMMSE and MAD of 0.073 and 0.226, respectively, compared with those of the other methods.

5.4.3. Visualization of SRI Quality Assessment

Figure 18 presents the SRI qualities obtained by different methods for the maiden specular reflected images of sample scenes in the MSPLFI object dataset. Remaining objects are presented in Appendix B (Figure A4). As can be seen, the proposed approach demonstrates better SRI quality than the others.

6. Conclusions

In this paper, a two-fold SRDI framework is proposed. As transparent objects lack their own textures, combining multisensory imagery cues improves their levels of specular detection and inpainting. Based on the private MSPLFI object dataset, the proposed SRD and SRI algorithms demonstrate better detection accuracy and suppression quality, respectively, than other techniques. In SRD, predictions of multiband changes in the sub-apertures in both polarized and unpolarized images are calculated and combined to obtain the overall specularity in transparent objects. In SRI, firstly, a distance matrix based on four-connected neighboring pixel patterns is calculated, and then the most similar one is selected to replace the specular pixel. The proposed algorithms predict better detection accuracy and inpainting quality in terms of F1-score, G-mean, accuracy, SSIM, PSNR, IMMSE, and MAD than other techniques reported in this paper. The experimental results illustrate the validity and the efficiency of the proposed method based on diverse performance evaluation metrics. They also demonstrate that it significantly improves the SRD metrics (with mean F1-score, G-mean, and accuracy 0.643, 0.656, and 0.981, respectively) and SRI ones (with the mean SSIM, PSNR, IMMSE, and MAD 0.966, 0.735, 0.073, and 0.226, respectively) for 18 transparent objects, each with 121 sub-apertures, in MSPLFI compared with those in the existing literature referenced in this paper.
As an extension of this work, we will investigate a machine learning technique for feature extraction and learning and testing of SRD and SRI performances on the MSPLFI object dataset. As it is known that a transparent object contains the same texture as its background, developing an automatic algorithm for segmenting it from its background in multisensory imagery will also be explored.

Author Contributions

Conceptualization, M.N.I. and M.T.; methodology, M.N.I. and M.T.; software, M.N.I.; validation, M.N.I. and M.T.; investigation, M.T. and M.P.; data curation, M.N.I.; writing—original draft preparation, M.N.I.; writing—review and editing, M.N.I., M.T. and M.P.; supervision, M.T. and M.P.; funding acquisition, M.P. All authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

Not applicable.

Acknowledgments

The authors would like to thank Denise Russell for her assistance with English expression.

Conflicts of Interest

The authors declare no conflict of interest.

Appendix A

Visualizations of SRD Methods.
Figure A1. Evaluation results for SRD performances of proposed method for 122 specular reflected images (121 sub-apertures + 1 maximum) of nine sample objects separately using different SRD metrics.
Figure A1. Evaluation results for SRD performances of proposed method for 122 specular reflected images (121 sub-apertures + 1 maximum) of nine sample objects separately using different SRD metrics.
Remotesensing 13 00455 g0a1
Evaluation results for SRD performances of proposed method for 122 specular reflected images (121 sub-apertures + 1 maximum) of 9 sample objects separately using different SRD metrics.
Figure A2. Comparison of SRD accuracies of different methods for sample objects in MSPLFI dataset.
Figure A2. Comparison of SRD accuracies of different methods for sample objects in MSPLFI dataset.
Remotesensing 13 00455 g0a2aRemotesensing 13 00455 g0a2b

Appendix B

Visualizations of SRI Methods.
Figure A3. Comparison of SRI accuracies of different methods for sample objects in MSPLFI dataset.
Figure A3. Comparison of SRI accuracies of different methods for sample objects in MSPLFI dataset.
Remotesensing 13 00455 g0a3aRemotesensing 13 00455 g0a3b
Evaluation results for SRI performances of proposed method for 122 specular reflection suppressed images (121 sub-aperture + 1 maximum ones) of 9 sample objects separately using different SRI metrics.
Figure A4. Evaluation results for SRI performances of proposed method for 122 specular reflection suppressed images (121 sub-aperture + 1 maximum ones) of nine sample objects separately using different SRI metrics.
Figure A4. Evaluation results for SRI performances of proposed method for 122 specular reflection suppressed images (121 sub-aperture + 1 maximum ones) of nine sample objects separately using different SRI metrics.
Remotesensing 13 00455 g0a4

References

  1. Yang, Q.; Wang, S.; Ahuja, N. Real-time specular highlight removal using bilateral filtering. In Proceedings of the European Conference on Computer Vision, Crete, Greece, 6–9 September 2010. [Google Scholar]
  2. Xin, J.H.; Shen, H.L. Accurate color synthesis of three-dimensional objects in an image. JOSA A 2004, 21, 713–723. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  3. Lin, S.; Lee, S.W. Estimation of diffuse and specular appearance. In Proceedings of the Seventh IEEE International Conference on Computer Vision, Kerkyra, Greece, 20–27 September 1999. [Google Scholar]
  4. Hara, K.; Nishino, K.; Ikeuchi, K. Determining reflectance and light position from a single image without distant illumination assumption. In Proceedings of the Ninth IEEE International Conference on Computer Vision, Nice, France, 3 April 2008. [Google Scholar]
  5. Tan, R.T.; Ikeuchi, K. Separating reflection components of textured surfaces using a single image. In Proceedings of the Digitally Archiving Cultural Objects, Nice, France, 14–17 October 2003. [Google Scholar]
  6. Kalra, A.; Taamazyan, V.; Rao, S.K.; Venkataraman, K.; Raskar, R.; Kadambi, A. Deep polarization cues for transparent object segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA, 13–19 June 2020. [Google Scholar]
  7. Tyo, J.S.; Goldstein, D.L.; Chenault, D.B.; Shaw, J.A. Review of passive imaging polarimetry for remote sensing applications. Appl. Opt. 2006, 45, 5453–5469. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  8. Yan, Q.; Shen, X.; Xu, L.; Zhuo, S.; Zhang, X.; Shen, L.; Jia, J. Crossfield joint image restoration via scale map. In Proceedings of the IEEE International Conference on Computer Vision, Sydney, NSW, Australia, 1–8 December 2013. [Google Scholar]
  9. Schaul, L.; Fredembach, C.; Susstrunk, S. Color image dehazing using the near-infrared. In Proceedings of the 16th IEEE International Conference on Image Processing (ICIP), Chiang Mai, Thailand, 7 November 2009. [Google Scholar]
  10. Salamati, N.; Larlus, D.; Csurka, G.; Süsstrunk, S. Semantic image segmentation using visible and near-infrared channels. In Proceedings of the European Conference on Computer Vision, Florence, Italy, 7–13 October 2012. [Google Scholar]
  11. Berns, R.S.; Imai, F.H.; Burns, P.D.; Tzeng, D.Y. Multispectral-based color reproduction research at the Munsell Color Science Laboratory. In Proceedings of the Electronic Imaging: Processing, Printing, and Publishing in Color, Proceedings of the SPIE, Zurich, Switzerland, 7 September 1998. [Google Scholar]
  12. Thomas, J.B. Illuminant estimation from uncalibrated multispectral images. In Proceedings of the 2015 Colour and Visual Computing Symposium (CVCS), Gjovik, Norway, 25–26 August 2015. [Google Scholar]
  13. Motohka, T.; Nasahara, K.N.; Oguma, H.; Tsuchida, S. Applicability of green-red vegetation index for remote sensing of vegetation phenology. Remote Sens. 2010, 2, 2369–2387. [Google Scholar] [CrossRef] [Green Version]
  14. Dandois, J.P.; Ellis, E.C. Remote sensing of vegetation structure using computer vision. Remote. Sens. 2010, 2, 1157–1176. [Google Scholar] [CrossRef] [Green Version]
  15. Rfenacht, D.; Fredembach, C.; Süsstrunk, S. Automatic and accurate shadow detection using near-infrared information. IEEE Trans. Pattern Anal. Mach. Intell. 2014, 36, 1672–1678. [Google Scholar] [CrossRef] [PubMed]
  16. Sobral, A.; Javed, S.; Ki Jung, S.; Bouwmans, T.; Zahzah, E.H. Online stochastic tensor decomposition for background subtraction in multispectral video sequences. In Proceedings of the 2015 IEEE International Conference on Computer Vision Workshop (ICCVW), Santiago, Chile, 7–13 December 2015. [Google Scholar]
  17. Islam, M.N.; Tahtali, M.; Pickering, M. Hybrid Fusion-Based Background Segmentation in Multispectral Polarimetric Imagery. Remote Sens. 2020, 12, 1776. [Google Scholar] [CrossRef]
  18. Nayar, S.K.; Fang, X.-S.; Boult, T. Separation of reflection components using color and polarization. Int. J. Comput. Vis. 1997, 21, 163–186. [Google Scholar] [CrossRef]
  19. Wolff, L.B. Polarization-based material classification from specular reflection. IEEE Trans. Pattern Anal. Mach. Intell. 1990, 12, 1059–1071. [Google Scholar] [CrossRef]
  20. Atkinson, G.A.; Hancock, E.R. Shape estimation using polarization and shading from two views. IEEE Trans. Pattern Anal. Mach. Intell. 2007, 29, 2001–2017. [Google Scholar] [CrossRef]
  21. Tan, J.; Zhang, J.; Zhang, Y. Target detection for polarized hyperspectral images based on tensor decomposition. IEEE Geosci. Remote Sens. Lett. 2017, 14, 674–678. [Google Scholar] [CrossRef]
  22. Goudail, F.; Terrier, P.; Takakura, Y.; Bigue, L.; Galland, F.; DeVlaminck, V. Target detection with a liquid-crystal-based passive stokes polarimeter. Appl. Opt. 2004, 43, 274–282. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  23. Denes, L.J.; Gottlieb, M.S.; Kaminsky, B.; Huber, D.F. Spectropolarimetric imaging for object recognition. In Proceedings of the 26th AIPR Workshop: Exploiting New Image Sources and Sensors, Washington, DC, USA, 1 March 1998. [Google Scholar]
  24. Romano, J.M.; Rosario, D.; McCarthy, J. Day/night polarimetric anomaly detection using SPICE imagery. IEEE Trans. Geosci. Remote Sens. 2012, 50, 5014–5023. [Google Scholar] [CrossRef]
  25. Islam, M.N.; Tahtali, M.; Pickering, M. Man-made object separation using polarimetric imagery. In Proceedings of the SPIE Future Sensing Technologies, Tokyo, Japan, 12–14 November 2019. [Google Scholar]
  26. Zhou, P.C.; Liu, C.C. Camouflaged target separation by spectral-polarimetric imagery fusion with shearlet transform and clustering segmentation. In Proceedings of the International Symposium on Photoelectronic Detection and Imaging 2013: Imaging Sensors and Applications, Beijing, China, 21 August 2013. [Google Scholar]
  27. Maeno, K.; Nagahara, H.; Shimada, A.; Taniguchi, R.I. Light field distortion feature for transparent object recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Portland, OR, USA, 23–28 June 2013. [Google Scholar]
  28. Xu, Y.; Maeno, K.; Nagahara, H.; Shimada, A.; Taniguchi, R.I. Light field distortion feature for transparent object classification. Comput. Vision Image Underst. 2015, 139, 122–135. [Google Scholar] [CrossRef]
  29. Xu, Y.; Nagahara, H.; Shimada, A.; Taniguchi, R.I. Transcut: Transparent object segmentation from a light-field image. In Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile, 7–13 December 2015. [Google Scholar]
  30. Shafer, S.A. Using color to separate reflection components. Color Res. Appl. 1985, 10, 210–218. [Google Scholar] [CrossRef] [Green Version]
  31. Tan, R.T.; Ikeuchi, K. Reflection components decomposition of textured surfaces using linear basis functions. In Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), San Diego, CA, USA, 20–25 June 2005. [Google Scholar]
  32. Yoon, K.J.; Choi, Y.; Kweon, I.S. Fast separation of reflection components using a specularity-invariant image representation. In Proceedings of the 2006 International Conference on Image Processing, Atlanta, GA, USA, 8–11 October 2006. [Google Scholar]
  33. Sato, Y.; Ikeuchi, K. Temporal-color space analysis of reflection. JOSA A 1994, 11, 2990–3002. [Google Scholar] [CrossRef]
  34. Lin, S.; Shum, H.Y. Separation of diffuse and specular reflection in color images. In Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001, Kauai, HI, USA, 8–14 December 2001. [Google Scholar]
  35. Shen, H.L.; Cai, Q.Y. Simple and efficient method for specularity removal in an image. Appl. Opt. 2009, 48, 2711–2719. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  36. Nguyen, T.; Vo, Q.N.; Yang, H.J.; Kim, S.H.; Lee, G.S. Separation of specular and diffuse components using tensor voting in color images. Appl. Opt. 2014, 53, 7924–7936. [Google Scholar] [CrossRef]
  37. Yamamoto, T.; Nakazawa, A. General improvement method of specular component separation using high-emphasis filter and similarity function. ITE Trans. Media Technol. Appl. 2019, 7, 92–102. [Google Scholar] [CrossRef] [Green Version]
  38. Mallick, S.P.; Zickler, T.; Belhumeur, P.N.; Kriegman, D.J. Specularity removal in images and videos: A PDE approach. In Proceedings of the European Conference on Computer Vision, Graz, Austria, 7–13 May 2006. [Google Scholar]
  39. Quan, L.; Shum, H.Y. Highlight removal by illumination-constrained inpainting. In Proceedings of the Ninth IEEE International Conference on Computer Vision, Nice, France, 13–16 October 2003. [Google Scholar]
  40. Akashi, Y.; Okatani, T. Separation of reflection components by sparse non-negative matrix factorization. Comput. Vis. Image Underst. 2016, 146, 77–85. [Google Scholar] [CrossRef] [Green Version]
  41. Arnold, M.; Ghosh, A.; Ameling, S.; Lacey, G. Automatic segmentation and inpainting of specular highlights for endoscopic imaging. EURASIP J. Image Video Process. 2010, 2010, 1–12. [Google Scholar] [CrossRef] [Green Version]
  42. Saint-Pierre, C.A.; Boisvert, J.; Grimard, G.; Cheriet, F. Detection and correction of specular reflections for automatic surgical tool segmentation in thoracoscopic images. Mach. Vis. Appl. 2011, 22, 171–180. [Google Scholar] [CrossRef]
  43. Meslouhi, O.; Kardouchi, M.; Allali, H.; Gadi, T.; Benkaddour, Y. Automatic detection and inpainting of specular reflections for colposcopic images. Open Comput. Sci. 2011, 1, 341–354. [Google Scholar] [CrossRef]
  44. Fedorov, V.; Facciolo, G.; Arias, P. Variational framework for non-local inpainting. Image Process. Line 2015, 5, 362–386. [Google Scholar] [CrossRef] [Green Version]
  45. Newson, A.; Almansa, A.; Gousseau, Y.; Pérez, P. Non-local patch-based image inpainting. Image Process. Line 2017, 7, 373–385. [Google Scholar] [CrossRef]
  46. Shih, T.K.; Chang, R.C. Digital inpainting-survey and multilayer image inpainting algorithms. In Proceedings of the Third International Conference on Information Technology and Applications (ICITA’05), Sydney, NSW, Australia, 4–7 July 2005. [Google Scholar]
  47. Kokaram, A.C. On missing data treatment for degraded video and film archives: A survey and a new Bayesian approach. IEEE Trans. Image Process. 2004, 13, 397–415. [Google Scholar] [CrossRef]
  48. Vogt, F.; Paulus, D.; Heigl, B.; Vogelgsang, C.; Niemann, H.; Greiner, G.; Schick, C. Making the invisible visible: Highlight substitution by color light fields. In Proceedings of the Conference on Colour in Graphics, Imaging, and Vision, Poitiers, France, 2–5 April 2002. [Google Scholar]
  49. Cao, Y.; Liu, D.; Tavanapong, W.; Wong, J.; Oh, J.; De Groen, P.C. Computer-aided detection of diagnostic and therapeutic operations in colonoscopy videos. IEEE Trans. Biomed. Eng. 2007, 54, 1268–1279. [Google Scholar] [CrossRef]
  50. Oh, J.; Hwang, S.; Lee, J.; Tavanapong, W.; Wong, J.; de Groen, P.C. Informative frame classification for endoscopy video. Med Image Anal. 2007, 11, 110–127. [Google Scholar] [CrossRef]
  51. Yang, Y.; Ma, W.; Zheng, Y.; Cai, J.F.; Xu, W. Fast single image reflection suppression via convex optimization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, 15–20 June 2019. [Google Scholar]
  52. Criminisi, A.; Pérez, P.; Toyama, K. Region filling and object removal by exemplar-based image inpainting. IEEE Trans. Image Process. 2004, 13, 1200–1212. [Google Scholar] [CrossRef]
  53. Reed, I.S.; Yu, X. Adaptive multiple-band CFAR detection of an optical pattern with unknown spectral distribution. IEEE Trans. Acoust. Speech Signal. Process. 1990, 38, 1760–1770. [Google Scholar] [CrossRef]
  54. Stokes, G.G. On the composition and resolution of streams of polarized light from different sources. Trans. Camb. Philos. Soc. 1851, 9, 399. [Google Scholar]
  55. Dowson, N.D.; Bowden, R. Simultaneous modeling and tracking (smat) of feature sets. In Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), San Diego, CA, USA, 20–25 June 2005. [Google Scholar]
  56. Chiu, S.Y.; Chiu, C.C.; Xu, S.S.D. A Background Subtraction Algorithm in Complex Environments Based on Category Entropy Analysis. Appl. Sci. 2018, 8, 885. [Google Scholar] [CrossRef] [Green Version]
  57. Somvanshi, S.S.; Kunwar, P.; Tomar, S.; Singh, M. Comparative statistical analysis of the quality of image enhancement techniques. Int. J. Image Data Fusion 2017, 9, 131–151. [Google Scholar] [CrossRef]
Figure 1. Schematic diagram of the proposed image acquisition in multispectral polarimetric light field imagery (MSPLFI).
Figure 1. Schematic diagram of the proposed image acquisition in multispectral polarimetric light field imagery (MSPLFI).
Remotesensing 13 00455 g001
Figure 2. Median specular reflections of MSPLFI for different objects: (O#1) round container with ball; (O#2) classic jug; (O#3) empty round container; (O#4) jar with cork lid; (O#5) sauce container; (O#6) ice glass; (O#7) clear glass jar; (O#8) coffee cup; (O#9) cuvee tumbler; (O#10) glass tumbler; (O#11) teacup; (O#12) water glass; (O#13) Bordeaux wine glass; (O#14) red wine glass; (O#15) hi-ball glass; (O#16) food box; (O#17) jar with cork handle; (O#18) port wine glass.
Figure 2. Median specular reflections of MSPLFI for different objects: (O#1) round container with ball; (O#2) classic jug; (O#3) empty round container; (O#4) jar with cork lid; (O#5) sauce container; (O#6) ice glass; (O#7) clear glass jar; (O#8) coffee cup; (O#9) cuvee tumbler; (O#10) glass tumbler; (O#11) teacup; (O#12) water glass; (O#13) Bordeaux wine glass; (O#14) red wine glass; (O#15) hi-ball glass; (O#16) food box; (O#17) jar with cork handle; (O#18) port wine glass.
Remotesensing 13 00455 g002
Figure 3. Multiband polarimetric images of object O#8 (seven individual bands and RGB band in visible range (400 nm–700 nm) at four polarimetric orientations (0°, 45°, 90°, and 135°) with no polarization setting).
Figure 3. Multiband polarimetric images of object O#8 (seven individual bands and RGB band in visible range (400 nm–700 nm) at four polarimetric orientations (0°, 45°, 90°, and 135°) with no polarization setting).
Remotesensing 13 00455 g003
Figure 4. Captured 4D light filed images through Lytro camera (sample object O#8 with 121 sub-aperture images).
Figure 4. Captured 4D light filed images through Lytro camera (sample object O#8 with 121 sub-aperture images).
Remotesensing 13 00455 g004
Figure 5. Scene flows in light field (LF) imagery: (a) views of positional and directional movements corresponding to central viewpoint; (b) each pixel in LF imagery has six degrees of freedom, with region of interest (ROI) indicated by yellow rectangle; and (c) example of ROI displacement and corresponding intensity plot.
Figure 5. Scene flows in light field (LF) imagery: (a) views of positional and directional movements corresponding to central viewpoint; (b) each pixel in LF imagery has six degrees of freedom, with region of interest (ROI) indicated by yellow rectangle; and (c) example of ROI displacement and corresponding intensity plot.
Remotesensing 13 00455 g005
Figure 6. Proposed two-fold framework for specular reflection detection (SRD) and specular reflection inpainting (SRI).
Figure 6. Proposed two-fold framework for specular reflection detection (SRD) and specular reflection inpainting (SRI).
Remotesensing 13 00455 g006
Figure 7. Classification of regions of MSPLFI object imagery: (A) specular reflection; (B) mixed specular and diffuse reflections; (C) diffuse reflection; (D) background.
Figure 7. Classification of regions of MSPLFI object imagery: (A) specular reflection; (B) mixed specular and diffuse reflections; (C) diffuse reflection; (D) background.
Remotesensing 13 00455 g007
Figure 8. Quasi-ground truth of SRD.
Figure 8. Quasi-ground truth of SRD.
Remotesensing 13 00455 g008
Figure 9. Evaluation results for SRD performances of proposed method for 122 specular reflected images (121 sub-apertures + 1 maximum) of nine sample objects separately using different SRD metrics.
Figure 9. Evaluation results for SRD performances of proposed method for 122 specular reflected images (121 sub-apertures + 1 maximum) of nine sample objects separately using different SRD metrics.
Remotesensing 13 00455 g009
Figure 10. Evaluation results for SRD performances of proposed method for 122 specular reflected images (121 sub-aperture + 1 maximum) × 18 objects = 2196 images for all objects (O#1–O#18) combined using different SRD metrics.
Figure 10. Evaluation results for SRD performances of proposed method for 122 specular reflected images (121 sub-aperture + 1 maximum) × 18 objects = 2196 images for all objects (O#1–O#18) combined using different SRD metrics.
Remotesensing 13 00455 g010
Figure 11. Evaluation results for SRD performances of different methods for maximum specular reflected images of 18 objects in terms of precision, recall, F1-score, G-mean and accuracy.
Figure 11. Evaluation results for SRD performances of different methods for maximum specular reflected images of 18 objects in terms of precision, recall, F1-score, G-mean and accuracy.
Remotesensing 13 00455 g011
Figure 12. Evaluation results for SRD performances of different methods for 121 sub-aperture + 1 maximum images × 18 objects = 2196 images with specular reflections in terms of precision, recall, F1-score, G-mean, and accuracy.
Figure 12. Evaluation results for SRD performances of different methods for 121 sub-aperture + 1 maximum images × 18 objects = 2196 images with specular reflections in terms of precision, recall, F1-score, G-mean, and accuracy.
Remotesensing 13 00455 g012
Figure 13. Comparison of SRD accuracies of different methods for sample objects in MSPLFI dataset.
Figure 13. Comparison of SRD accuracies of different methods for sample objects in MSPLFI dataset.
Remotesensing 13 00455 g013
Figure 14. Evaluation results for SRI performances of proposed method for 122 specular reflection suppressed images (121 sub-aperture + 1 maximum ones) of nine sample objects separately using different SRI metrics.
Figure 14. Evaluation results for SRI performances of proposed method for 122 specular reflection suppressed images (121 sub-aperture + 1 maximum ones) of nine sample objects separately using different SRI metrics.
Remotesensing 13 00455 g014
Figure 15. Evaluation results for SRI performances of proposed method for 121 sub-aperture + 1 maximum images × 18 objects = 2196 images for all objects (O#1–O#18) combined using different SRI metrics.
Figure 15. Evaluation results for SRI performances of proposed method for 121 sub-aperture + 1 maximum images × 18 objects = 2196 images for all objects (O#1–O#18) combined using different SRI metrics.
Remotesensing 13 00455 g015
Figure 16. Evaluation results for SRI performances of individual methods for each maiden specular suppressed image of 18 objects in terms of SSIM, PSNR, IMMSE, and MAD.
Figure 16. Evaluation results for SRI performances of individual methods for each maiden specular suppressed image of 18 objects in terms of SSIM, PSNR, IMMSE, and MAD.
Remotesensing 13 00455 g016
Figure 17. Evaluation results for SRI performances of different methods for 121 sub-aperture + 1 maiden images × 18 objects = 2196 images in terms of SSIM, PSNR, IMMSE, and MAD.
Figure 17. Evaluation results for SRI performances of different methods for 121 sub-aperture + 1 maiden images × 18 objects = 2196 images in terms of SSIM, PSNR, IMMSE, and MAD.
Remotesensing 13 00455 g017
Figure 18. Comparison of SRI accuracies of different methods for sample objects in MSPLFI dataset.
Figure 18. Comparison of SRI accuracies of different methods for sample objects in MSPLFI dataset.
Remotesensing 13 00455 g018aRemotesensing 13 00455 g018b
Table 1. Performance evaluation of different methods in terms of various SRD metrics for 18 objects (O#1–O#18) in MSPLFI object dataset and overall means (all sub-aperture images in 4D LF).
Table 1. Performance evaluation of different methods in terms of various SRD metrics for 18 objects (O#1–O#18) in MSPLFI object dataset and overall means (all sub-aperture images in 4D LF).
MethodsMetricsObject Index (Maximum SRD)Overall Mean (SA)
O#1O#2O#3O#4O#5O#6O#7O#8O#9O#10O#11O#12O#13O#14O#15O#16O#17O#18
Ak. [40]Precision0.1780.3480.6860.4450.6000.3540.4600.3820.6550.5190.2400.3110.3360.1240.5220.5420.5040.1230.362 ± 0.24
Recall0.6280.6290.6620.4270.5140.3450.4260.4170.7710.5360.5980.8660.6580.6220.4660.7270.3280.7470.512 ± 0.14
F1-Score0.2770.4480.6730.4360.5540.3500.4430.3980.7080.5280.3420.4570.4450.2070.4930.6210.3980.2110.377 ± 0.16
G-Mean0.7690.7810.8100.6440.7100.5780.6440.6340.8740.7220.7490.9170.7950.7540.6760.8350.5670.8340.689 ± 0.10
Accuracy0.9350.9620.9810.9430.9570.9390.9460.9390.9860.9480.9280.9700.9510.9100.9600.9440.9400.9290.926 ± 0.05
Sn. [35]Precision0.2200.6100.7590.5090.6130.4370.5270.4770.6020.5790.4620.4470.5740.3880.5900.6420.6220.5050.655 ± 0.15
Recall0.6670.5900.6390.3920.4930.3010.4110.3350.8310.5460.5130.8480.4570.4740.4760.6470.2750.5990.483 ± 0.15
F1-Score0.3300.6000.6940.4430.5460.3570.4620.3930.6980.5620.4860.5860.5090.4260.5270.6440.3810.5480.527 ± 0.13
G-Mean0.7970.7640.7970.6200.6960.5430.6350.5730.9060.7300.7090.9130.6720.6830.6850.7940.5220.7710.681 ± 0.11
Accuracy0.9460.9810.9830.9490.9580.9480.9520.9500.9840.9540.9660.9830.9740.9760.9640.9550.9460.9880.969 ± 0.01
Yn. [1]Precision0.2200.3960.6030.4020.4760.2690.3820.3640.5950.4380.2740.2240.2880.1660.4160.4940.4480.1560.433 ± 0.19
Recall0.8170.6380.6730.4570.5620.4300.4420.4750.8310.5710.6300.8840.6710.6520.4840.7580.3830.7540.529 ± 0.16
F1-Score0.3460.4880.6360.4280.5150.3310.4100.4130.6940.4960.3820.3580.4030.2650.4470.5980.4130.2580.446 ± 0.14
G-Mean0.8770.7890.8150.6640.7370.6360.6520.6750.9060.7390.7720.9190.7980.7820.6860.8480.6090.8450.707 ± 0.11
Accuracy0.9390.9680.9770.9370.9460.9170.9360.9350.9840.9370.9360.9540.9410.9310.9500.9360.9340.9450.953 ± 0.02
Ym. [37]Precision0.1990.4090.6570.4350.5310.2820.3020.3570.6310.4060.2430.2220.2960.1220.4030.3640.5130.1430.307 ± 0.23
Recall0.6450.6340.6650.4350.5470.3840.4560.4580.7780.5650.6460.8750.6800.6470.4920.7910.3280.7550.559 ± 0.15
F1-Score0.3040.4970.6610.4350.5390.3250.3630.4010.6970.4720.3530.3550.4120.2050.4430.4990.4000.2400.346 ± 0.17
G-Mean0.7820.7870.8110.6490.7300.6040.6560.6630.8770.7340.7770.9140.8040.7670.6910.8470.5670.8430.709 ± 0.10
Accuracy0.9410.9690.9800.9420.9520.9240.9200.9340.9850.9320.9250.9540.9420.9050.9480.9000.9400.9390.908 ± 0.06
Ar. [41]Precision0.1890.5200.4630.4710.5290.2580.4360.3830.4100.4680.3080.1910.2870.1780.3660.4960.4130.2550.561 ± 0.12
Recall0.5940.5870.6680.3940.3910.3510.4220.4490.7630.5260.6090.8770.3530.2810.4670.7270.3710.4470.434 ± 0.16
F1-Score0.2870.5520.5470.4290.4500.2980.4280.4140.5340.4950.4090.3140.3170.2180.4100.5900.3910.3250.466 ± 0.10
G-Mean0.7500.7610.8080.6200.6190.5770.6400.6580.8630.7130.7630.9100.5860.5240.6710.8310.5980.6630.644 ± 0.12
Accuracy0.9410.9770.9670.9460.9510.9210.9430.9390.9710.9420.9450.9440.9550.9620.9440.9360.9300.9760.966 ± 0.01
St. [42]Precision0.4610.6790.6800.5970.6920.3440.6090.3920.5860.6160.3400.2370.4910.3600.4210.6310.4870.1930.702 ± 0.12
Recall0.5920.5350.6370.3570.5020.3210.4000.3810.7710.4620.5580.8760.4570.3940.4950.5670.3150.7240.422 ± 0.15
F1-Score0.5180.5980.6580.4470.5820.3320.4830.3870.6660.5280.4230.3730.4730.3760.4550.5970.3830.3050.507 ± 0.11
G-Mean0.7640.7290.7950.5930.7040.5580.6280.6080.8730.6740.7340.9160.6710.6240.6930.7440.5550.8340.637 ± 0.12
Accuracy0.9780.9830.9800.9540.9630.9390.9570.9420.9830.9550.9520.9570.9700.9750.9500.9520.9380.9580.971 ± 0.01
Ms. [43]Precision0.6460.8780.9140.8760.7650.5920.7540.5850.8470.7020.5570.5570.5560.3480.6920.6570.7290.6600.868 ± 0.09
Recall0.5800.3670.5020.2480.4850.2120.3930.3070.5680.4450.5070.8310.4890.5720.3660.6270.2400.3380.283 ± 0.11
F1-Score0.6110.5180.6480.3870.5930.3120.5170.4030.6800.5450.5300.6670.5200.4330.4790.6420.3610.4470.412 ± 0.13
G-Mean0.7590.6060.7080.4980.6940.4590.6250.5510.7530.6640.7070.9070.6950.7480.6030.7830.4890.5810.520 ± 0.11
Accuracy0.9850.9830.9840.9590.9660.9560.9630.9560.9880.9600.9720.9880.9730.9710.9670.9560.9490.9890.971 ± 0.01
ProposedPrecision0.6300.6660.7280.6220.6680.6430.7980.5630.7560.6780.4850.6240.4700.4220.6650.6580.7190.6140.776 ± 0.10
Recall0.6300.5850.7370.7980.9460.2810.7670.4520.8080.6130.5260.7200.5540.7180.5530.7840.3200.5780.444 ± 0.15
F1-Score0.6300.6230.7320.6990.7830.3910.7820.5010.7810.6440.5040.6680.5090.5310.6040.7150.4420.5960.546 ± 0.13
G-Mean0.7910.7620.8550.8810.9600.5280.8710.6660.8960.7770.7180.8460.7370.8390.7390.8730.5630.7590.654 ± 0.11
Accuracy0.9850.9830.9840.9650.9730.9580.9780.9570.9900.9630.9670.9900.9680.9760.9700.9610.9510.9900.974 ± 0.01
Table 2. Performance evaluations of different methods using different SRI metrics for 18 objects (O#1–O#18) and overall mean (all sub-aperture images in 4D LF) in MSPLFI object dataset.
Table 2. Performance evaluations of different methods using different SRI metrics for 18 objects (O#1–O#18) and overall mean (all sub-aperture images in 4D LF) in MSPLFI object dataset.
MethodsMetricsObject Index (Maximum SRI)Overall Mean (SA)
O#1O#2O#3O#4O#5O#6O#7O#8O#9O#10O#11O#12O#13O#14O#15O#16O#17O#18
Ar. [41]SSIM0.9420.9670.9660.9650.9400.9610.9400.9590.9650.9290.9400.9460.9680.9580.9250.9430.9630.9550.941 ± 0.02
PSNR21.2520.4221.2620.9619.9920.9519.2220.2520.7419.0318.3318.5320.8318.5818.4219.5620.9819.6519.80 ± 0.99
IMMSE487.6590.1486.2520.9651.4522.7778.0613.9548.7813.4954.8911.5537.6901.9935.8720.2519.3705.4698.9 ± 162
MAD12.5316.2016.2615.2613.4914.8919.9415.1215.7918.5118.8719.8012.7417.9719.6318.2713.5515.5216.46 ± 2.36
Yg. [51]SSIM0.8870.9560.9430.9510.9260.9510.9100.9520.9540.9220.9440.9430.9600.9480.9110.9150.9580.9570.926 ± 0.02
PSNR18.3119.7420.1621.4218.4420.5319.2920.1220.4318.7218.9519.0621.3618.9817.6818.7822.0120.4519.53 ± 1.14
IMMSE958.6690.6626.5468.5931.3574.9766.7632.8589.2872.9828.4807.7475.5822.31110861.8408.9586.0749.8 ± 190
MAD18.0516.3617.1513.6816.0114.8819.0614.8815.5218.6817.1618.3611.2616.3420.9819.3611.5813.7716.48 ± 2.58
Cr. [52]SSIM0.9560.9680.9640.9480.9240.9630.9220.9610.9650.9270.9440.9470.9620.9560.9250.9400.9620.9550.935 ± 0.02
PSNR22.5020.6021.4020.4819.5221.3119.0620.6420.8419.1618.6818.9020.9718.6318.6019.6521.2319.7419.89 ± 1.04
IMMSE365.8566.9471.8582.8726.3480.6807.4561.7536.1789.5881.6838.4519.9891.1897.0704.5489.6690.5685.5 ± 161
MAD11.4115.9016.0916.0414.3614.3120.3314.5615.6918.2318.1219.0812.4517.7819.2518.0313.2415.3416.27 ± 2.36
St. [42]SSIM0.9560.9680.9670.9660.9450.9670.9430.9660.9660.9330.9480.9520.9700.9570.9290.9390.9670.9570.941 ± 0.02
PSNR22.4920.5921.4121.1120.0721.3019.5420.6120.8819.2318.6018.9021.1018.6618.5419.8321.4219.7020.01 ± 1.05
IMMSE366.4567.2469.7504.1639.7482.0722.3565.5531.5776.6896.9837.2505.1886.4910.4676.7469.1696.4667.6 ± 162
MAD11.5415.8916.0614.9113.3914.3519.0414.6815.5918.1318.3219.0012.2917.7319.4117.4812.9915.4016.04 ± 2.31
Ak. [40]SSIM0.9180.9790.9380.9410.9130.9280.9000.9290.9430.8990.9070.9120.9420.9330.8890.9140.9310.9280.899 ± 0.03
PSNR19.3624.3018.4918.9317.0017.8916.8217.1718.4116.2715.4916.0017.8016.0915.8416.4817.8917.2917.08 ± 1.12
IMMSE753.7241.5921.2831.61296105713511248936.81536183816311080159816941464105712151315 ± 334
MAD16.456.3621.4518.7719.4321.1925.7021.5119.9625.5426.1726.5217.8023.9026.3125.8719.2020.2822.23 ± 3.24
Sn. [35]SSIM0.9360.9610.9570.9520.9230.9590.9220.9560.9510.9170.9370.9410.9640.9520.9150.9340.9610.9540.929 ± 0.02
PSNR19.3219.9920.7819.9418.2320.7818.1719.9719.1317.7318.0918.4220.4118.2317.8019.0921.0119.5719.06 ± 1.05
IMMSE760.9652.2543.6659.6976.7543.1992.1654.8795.411011009934.8591.4976.51079802.5515.3717.4830.7 ± 197
MAD14.6016.8016.9316.3715.9515.0521.3515.5617.4920.5519.3120.0413.2018.5520.6618.9913.4815.6117.43 ± 2.43
Ym. [37]SSIM0.9060.9520.9450.9490.9170.9340.8970.9330.9500.8940.9110.9120.9380.9200.8900.8800.9440.9380.902 ± 0.03
PSNR18.3718.8319.1119.4617.7218.6916.3717.7219.1116.0115.9716.2317.8715.5715.8614.8419.3418.2017.27 ± 1.44
IMMSE946.5852.3798.1737.01100879.515001098797.61631164315501061180416862134756.5985.21289 ± 439
MAD17.8918.5719.2017.3617.3518.9225.9519.3817.9925.5423.9425.2416.7924.0425.5929.9015.8118.0321.27 ± 4.11
ProposedSSIM0.9920.9900.9890.9720.9410.9840.9610.9730.9910.9470.9640.9770.9780.9820.9500.9530.9830.9830.956 ± 0.02
PSNR33.4326.1629.2422.7922.2629.8525.6025.0629.2723.8422.7624.6226.5224.9121.7622.3727.6425.3624.51 ± 2.11
IMMSE29.54157.477.50341.9386.767.34179.2202.976.95268.6344.8224.2145.1209.9433.9376.7112.1189.2257.6 ± 119
MAD1.1727.9035.2057.6808.5364.5298.7238.2774.95910.0711.078.8885.2577.88013.2612.975.5457.6658.427 ± 2.51
SSIM: structural similarity index; PSNR: peak signal-to-noise ratio; IMMSE: mean squared error; MAD: mean absolute deviation.
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Islam, M.N.; Tahtali, M.; Pickering, M. Specular Reflection Detection and Inpainting in Transparent Object through MSPLFI. Remote Sens. 2021, 13, 455. https://doi.org/10.3390/rs13030455

AMA Style

Islam MN, Tahtali M, Pickering M. Specular Reflection Detection and Inpainting in Transparent Object through MSPLFI. Remote Sensing. 2021; 13(3):455. https://doi.org/10.3390/rs13030455

Chicago/Turabian Style

Islam, Md Nazrul, Murat Tahtali, and Mark Pickering. 2021. "Specular Reflection Detection and Inpainting in Transparent Object through MSPLFI" Remote Sensing 13, no. 3: 455. https://doi.org/10.3390/rs13030455

APA Style

Islam, M. N., Tahtali, M., & Pickering, M. (2021). Specular Reflection Detection and Inpainting in Transparent Object through MSPLFI. Remote Sensing, 13(3), 455. https://doi.org/10.3390/rs13030455

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop